Compression breaks file_server browse template using httpInclude

[bones-was-here]

I have modified the default file_server browse template to integrate it with a website, by using httpInclude to load blocks of HTML containing the header, footer and theme. These files are hosted on the same Caddy instance, and it works well provided that compression is not enabled. When compression is enabled, the file_server browser becomes garbage: it appears that raw compressed data is included in the HTML served to the client.

Here is my Caddyfile workaround:

@not_integ {
        # FIXME: workaround for compression breaking httpInclude in templates
        not path /integ/*
}
encode @not_integ zstd gzip

[bones-was-here]

Caddy version is 2.4.5 from the official package for Debian.

[bones-was-here]

The included paths are all relative, eg {{ httpInclude "/integ/nav.html" }} so maybe Caddy could assume the file is on the same host and skip compression in this case.

[francislavoie]

Interesting. Should be an easy fix, the internal HTTP request should set the Accept-Encoding: identity header to tell Caddy not to encode it.

caddy/modules/caddyhttp/templates/tplcontext.go

Line 162 in 5fda961

virtReq.Header = c.Req.Header.Clone()

[u5surf]

@francislavoie Hi, Can I work this issue?
It seems quite easy following the patch,
I’m not sure that would be necessary anywhere else.

diff --git a/modules/caddyhttp/templates/tplcontext.go b/modules/caddyhttp/templates/tplcontext.go
index 71886c9a..570967a7 100644
--- a/modules/caddyhttp/templates/tplcontext.go
+++ b/modules/caddyhttp/templates/tplcontext.go
@@ -160,6 +160,7 @@ func (c TemplateContext) funcHTTPInclude(uri string) (string, error) {
        }
        virtReq.Host = c.Req.Host
        virtReq.Header = c.Req.Header.Clone()
+       virtReq.Header.Add("Accept-Encoding", "identity")
        virtReq.Trailer = c.Req.Trailer.Clone()
        virtReq.Header.Set(recursionPreventionHeader, strconv.Itoa(recursionCount))

[francislavoie]

@u5surf yes, go ahead! Please also add a test to cover this if it's not too difficult.

[u5surf]

@francislavoie
I wrote a unit test of the func funcHTTPInclude using handler structure with http.Handle interface, but I feel it's different from covering like this phenomenon.
Is there any generic tests or end to end tests about templates ?

--- a/modules/caddyhttp/templates/tplcontext_test.go
+++ b/modules/caddyhttp/templates/tplcontext_test.go
@@ -16,6 +16,7 @@ package templates

 import (
        "bytes"
+       "context"
        "fmt"
        "io/ioutil"
        "net/http"
@@ -26,10 +27,45 @@ import (
        "strings"
        "testing"
        "time"
+
+       "github.com/caddyserver/caddy/v2/modules/caddyhttp"
 )

+type handle struct{
+}
+
+func (h *handle) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+       w.Write([]byte("test contents"))
+}
+
+func TestHTTPInclude(t *testing.T) {
+       tplContext := getContextOrFail(t)
+       for i, test := range []struct {
+               uri     string
+               handler *handle
+               expect  string
+       }{
+               {
+                       uri:     "https://example.com/foo/bar",
+                       handler: &handle{},
+                       expect:  "test contents",
+               },
+       } {
+               ctx := context.WithValue(tplContext.Req.Context(), caddyhttp.ServerCtxKey, test.handler)
+               tplContext.Req = tplContext.Req.WithContext(ctx)
+               tplContext.Req.Header.Add("Accept-Encoding", "gzip")
+               result, err := tplContext.funcHTTPInclude(test.uri)
+               if result != test.expect {
+                       t.Errorf("Test %d: expected '%s' but got '%s'", i, test.expect, result)
+               }
+               if err != nil {
+                       t.Errorf("Test %d: got error: %v", i, result)
+               }
+       }
+}
+

[francislavoie]

Best to discuss this in a pull request! Go ahead and open one with your proposed changes!