What exactly is the high fee that Bitcoin Core is talking about? Does this client have exactly fixed number?
Bitcoin Core has a hard limit for what it considers to be an absurdly high fee, which is 0.1. Other node software may not impose this limit, or may have a different value for it.
Note that this is not part of the standard tx rules. It is simply a check to limit the fees for a single transaction.
Can I broadcast a transaction programmatically without any fee restrictions as intended if I use custom client?
You can remove the fee check from
If 2 is possible, can hackers make spamming attacks by setting fees lower than the current minimum limit(0.00001 BTC/kB)?
You can just change the
minrelayfee flag to go under the limit. There is no flag to override the high fee parameter, as far as I'm aware.
There are plenty of nodes on the network that will broadcast a high fee tx. Indeed, there have been a few cases of very high fees in the past, such as 13.65 BTC (https://www.blockchain.com/btc/tx/f1aee93304724cb8d1a0498f8c1fe9f2269710caa2b2dbc4bb2c758a5aaf9312), and even 200 BTC (https://www.blockchain.com/btc/tx/4ed20e0768124bc67dc684d57941be1482ccdaa45dadb64be12afba8c8554537) (this one was before the high fee check was added in 2015, though)!