add ignore_rpc_timeout option to allow suppressing rpc timeout errors

[gluckzhang]

Problem

Currently we run the watchtower to monitor validators on both mainnet and testnet. Though we have configured the instance to have a higher unhealthy threshold, ignore bad gateway errors, bear a longer connection time, and check the status less frequently (e.g., --unhealthy-threshold 2 --ignore-http-bad-gateway --rpc-timeout 60 --interval 65), we still receive a lot of operation timed out alerts. Such errors are more related to the availability of RPC endpoints and for now, we would like to suppress such errors.

Summary of Changes

This PR adds a new optional cli option --ignore-rpc-timeout to allow users to suppress rpc timeout errors. The default value of --ignore-rpc-timeout is false so merging this PR does not change the default behavior of watchtower. It is up to users to decide whether they would like to ignore rpc timeouts.

[gluckzhang]

Hello again, as there are changes merged to the master, I have rebased the branch of this PR. Please feel free to reject if you think --ignore-rpc-timeout is no longer needed since now the watchtower can take multiple RPC endpoints :)

[joncinque]

Hey @mircea-c 👋 can you take a look at this PR? It might not be needed after #4748, but I'll let you discuss with the author

[mircea-c]

Hi @gluckzhang. I'm going to rejecting this change as I don't think ignoring timeouts is a solution to the flaky endpoint problem.

As mentioned by Jon, #4748 is meant to add redundancy which reduces false positives while maintaining coverage and allowing for detection of bad RPC endpoints.

Additionally, there is the unhealthy-threshold flag available that can be used to achieve a result similar to ignoring RPC.

I would be much more willing to consider changes to unhealthy-threshold that make it "smarter"

[gluckzhang]

Hi @mircea-c, thanks for the feedback. Fully understand it and agree with the rejection. Cheers :)