sidecar: allow custom http con pool size fix #1953

[blockloop]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Allow custom connection pool size for Thanos sidecar to prevent excessive tcp connections on machines with very high load. More details are laid out in #1953

Verification

A build from my fork is currently running in production on our machines and handling ~1700 RPS with very little connection cycling.

$ netstat -a | awk '{ print $6 }' | grep -P '[A-Z]' | sort | uniq -c | sort -r
    951 ESTABLISHED
    166 TIME_WAIT
    146 CONNECTED
    133 CLOSE_WAIT
    129 FIN_WAIT2
     53 STREAM
     12 LISTEN
      4 FIN_WAIT1
      1 SEQPACKET
      1 I-Node
      1 Foreign

Other Info

HTTP connections

[squat]

🥳