Set keep-alive timeout higher than ALB idle timeout

[rbreslow]

Overview

Keep-alive, when enabled, enables the load balancer to reuse back-end connections until the keep-alive timeout expires. To ensure that the load balancer is responsible for closing the connections to your instance, make sure that the value you set for the HTTP keep-alive time is greater than the idle timeout setting configured for your load balancer.

The default keep-alive timeout for the Node http.Server is 5 seconds. The default idle timeout for the ALB is 60 seconds.

The ALB is opening connections for reuse, the Node http.Server closes that connection, and then the ALB tries to communicate over the closed connection which triggers a 502.

HTTP 502: Bad gateway

Possible causes:
. . .
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.

We can see more evidence of this by looking at the ALB access logs with Athena:

Details

SELECT elb_status_code,
         request_processing_time,
         target_processing_time,
         response_processing_time,
         request_url
FROM stg_alb_logs
WHERE elb_status_code LIKE '5%';

response_processing_time is set to -1 if the load balancer can't send the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request.

This PR sets the keep-alive timeout to be higher than the ALB timeout so that the ALB is responsible for prompting the closure of individual TCP connections to the server.

See:

• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-502-issues
• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-syntax
• https://adamcrowder.net/posts/node-express-api-and-aws-alb-502/ (Excellent deep dive into the problem)

Resolves #428

Checklist

• Description of PR is in an appropriate section of CHANGELOG.md and grouped with similar changes, if possible

Testing Instructions

I'm not sure how to replicate the failures we saw earlier. I've tried exercising the staging application and wasn't able to generate any 502s today before or after applying the fix (see the Jenkins deploy). Additionally, the prd_alb_logs table in Athena does not contain any 502s.

I think the evidence makes it clear we should apply this either way, and we can be on the look out for more 502s as we're able to gather more data.

[rbreslow]

This was resolved and back ported to Node v12 in nodejs/node#34131. However, it doesn't look like the fix will be accessible until Node v12.18.5 drops (https://github.com/nodejs/node/commits/v12.x. vs https://github.com/nodejs/node/commits/v12.x-staging).

[hectcastro]

The application server timeout changes appear to be working as an effective 502 deterrent. I added a simple k6 load test in 4827e9a and used it against an instance of the application with and without the changes.

When the timeout changes were not applied, 502s were present in the Athena ALB request logs. When the timeout changes were applied, 502s were not present in the logs. 👍

[coalkettler]

@rbreslow Chiming in to say that this was really useful to read and that Adam Crowder article offers a great explanation. Good stuff! 🕵️