reverseproxy: Add handle_response blocks to reverse_proxy (#3710)

[francislavoie]

Supersedes #3712, closes #2920, #3707, #3828

Adds a new handle_response subdirective to reverse_proxy, and also @<name> named response matchers to pair with it.

Example which might serve files that are otherwise not accessible based on some response header:

reverse_proxy localhost:8080 {
	@accel header X-Accel-Redirect *
	handle_response @accel {
		root * /path/to/private/files
		rewrite {http.response.header.X-Accel-Redirect}
		file_server
	}
}

The matcher is optional; having no matcher will match all responses. Any handle_response without a matcher will be sorted to the end just in case the user put them in a bad order.

This also supports an optional status code on the same line, to change the status code of the response. If using this syntax, then a block may not be defined, and the matcher is required. (Why would you ever want to

reverse_proxy localhost:8080 {
	@fooIsBad header Foo *
	handle_response @fooIsBad 500
}

We should also remember to document that a status code matcher like 4 matches 4xx, as per the caddyhttp.StatusCodeMatches() function. It's useful.

[frou]

Sorry for what's maybe an elementary question, but can this be used to add some nuance to what Caddy does when the server being reverse proxied to is completely down?

Caddy currently serves a blank page with status code 502 (Bad Gateway) in that scenario. I'd like to be able to serve a page with a readable message, rather than just a blank page.

[francislavoie]

@frou I'm pretty sure you can use https://caddyserver.com/docs/caddyfile/directives/handle_errors for that.

The difference is who emits the error. If it's Caddy itself that emits it, then you use handle_errors, and if the backend emits it as a response, then you use handle_response. A 502 is usually Caddy saying it couldn't connect to the upstream... But nothing stops an upstream from responding with a 502 if it feels like it.

[amyspark]

👋 I've been sorely trying to get a shadow of Gitlab Pages/nginx's error_page 404 /404.html with a proxy. Will this MR let me preserve the status code, while replacing only the response body with the result of handle_response?

[francislavoie]

Thanks for sparking the question @amyspark 😂

I realized that there was a piece lacking to make that work. The file_server handler would only write a 200 status code when writing the file. To make what you want work, I wrote this PR #4076 that should do it, i.e. something like this:

reverse_proxy localhost:8080 {
	@fourhundred status 4
	handle_response @fourhundred {
		root * /path/to/errors
		rewrite * /{http.reverse_proxy.status_code}.html
		file_server {
			status {http.reverse_proxy.status_code}
		}
	}
}

[amyspark]

Close enough! but what I need is something like this:

reverse_proxy localhost:8080 {
	@fourhundred status 4
	handle_response @fourhundred {
		rewrite * /{http.reverse_proxy.status_code}.html
		reverse_proxy localhost:8080 {http.reverse_proxy.status_code}
	}
}

(i.e. rewrite the petition, send it again upstream, but keep the returned status code from the original response's)

[francislavoie]

In that case you'll just need to make sure your upstream writes the correct status. I don't think it's a good idea to modify reverse_proxy for that.

[mholt]

Just little things, I think. Thanks for this contribution. Sorry it took me a while to devote the focus to review it.

I wonder if we could keep the UnmarshalCaddyfile() method if we keep the Caddyfile token parsing in there, but then leave any extra setup of the handler (e.g. the hanlde_response routes) to a caller, such as parseCaddyfile that takes the Helper. (I could elaborate in Slack...)

[mholt]

Excellent work! It turned out just about as I had hoped. Yes it's a bummer the complete Caddyfile unmarshaling can't take place inside UnmarshalCaddyfile(), but I think maintaining that interface is important, and the workaround is pretty reasonable.

It might be worth adding to the godoc comment for UnmarshalCaddyfile() that an additional call to Finalize... is needed when a Helper is available.

[mholt]

Great work. Thanks for your patience and persistence with this.

[francislavoie]

And thanks for @maxatome for doing the groundwork for this! 🎉