dns: SRV regression vs KubeDNS 1.14 and below: Cannot unmarshal DNS message

[bwplotka]

--store dnssrv option does not work on master.

Version of querier from master is giving me this:

level=error name=querier-78b68cf9d4-q2wg2 ts=2019-04-09T15:01:03.916126258Z caller=provider.go:71 msg="dns resolution failed" addr=dnssrvnoa+store-apis.default.svc.cluster.local:19090 err="lookup SRV records \"store-apis.default.svc.cluster.local\": lookup store-apis.default.svc.cluster.local on 10.1.192.10:53: cannot unmarshal DNS message"

cc @mjd95 as we added dnssrvnoa ...

Debugging now.

0.3.2 works, just upgraded to master version to get recent fixes.

Is it some CGO/Non-CGO issue? 🤔

[bwplotka]

Potential matches:
golang/go#27546
golang/go#21160

[bwplotka]

Did version bi-search and looks like master-2019-03-06-5c899a0 is last working version and master-2019-03-12-910d438 has problems.

Looks like this introduced regression: #913

Which is exactly golang/go#27546 issue as this was introduced by Golang version change... :/

[bwplotka]

Ok, it's not Golang problem, Golang just started to implement proper DNS client RFC, it's just KubeDNS 1.14 and below issue.

[bwplotka]

golang/go#27546 is the issue.

We decided to add custom resolver that does not have this issue: #1016

[bwplotka]

The other fix is to disable net CGO, but miekg/dns might be useful for handling TTLs as well