[Ruler][DNS] Don't propagate no such host error if using default resolver

[OGKevin]

Signed-off-by: Kevin Hellemun 17928966+OGKevin@users.noreply.github.com

Fixes #3186

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Verification

[OGKevin]

Shall there be a debug log entry when this happens 🤔 ?

[bwplotka]

Thanks!

It's not the only a ruler who is using this, also Querier and other components and projects including Cortex.

I wonder if this makes sense, maybe putting it behind some option would do 🤔

No host found is when you cannot reach DNS host and this is definitely a configuration, server or just access failure. Imagine you are rolling to new cluster and somehow pod cannot each DNS server. By masking this error we will never know about this error right?

cc @pracucci @pstibrany

[bwplotka]

What exactly you want to achieve @OGKevin what's the use case? (:

[OGKevin]

AFAIK, Host not found is also returned when the DNS server is reachable. Have a look at this article for example.

➜  ~ host k3jhut933.messagebird.io
Host k3jhut933.messagebird.io not found: 3(NXDOMAIN)
➜  ~ host google.com  
google.com has address 172.217.17.78
google.com has IPv6 address 2a00:1450:4001:817::200e
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.

The DNS server is working fine, the DNS just does not exits, hence the NXDOMAIN which in go results in a Host not found error.

The issue linked in #3186 has more context of the use case that we want to avoid. Thought, this use case is ruler specific 🤔 not sure how other components handle this.

[bwplotka]

Ok, I checked a bit and it looks like indeed IsNotFound is returned when EAI_NONAME is given by Linux resolver (getaddrinfo) so you might be right.

Still I would consider this as an error, potentially a configuration error case. (imagine you made a typo in crafting a DNS target for alertmananger).

I think the main question regarding your issue is really, should a ruler just mention the error and continue running or not. This decision might orthogonal to this change and in fact depends what it resolves for (key functionality or not) 🤔

Some more opinions welcome, but I would rather stop crashing ruler on fail like this on start, but still log & instrument it as failure overall.

[OGKevin]

The problem might not be a config issue only. If Alertmanagers was up and running fine, and the DNS name becomes invalid because all pods are down. Ruler wil crashloop without making any config change to ruler.

So IMO Ruler should for sure not crash because Alertmanagers can't be resolved. Because ruler is not only responsible for evaluating alerting rules. I also agree that there should be a log entry saying that DNS resolution failed.

On the other issues, e.g. how it impacts the other components I can't make a clear assessment. What we could do indeed is, add a flag and set this to "true" by default for ruler, and false for the other components so that the behaviour only changes for Ruler.

[bwplotka]

. Ruler wil crashloop without making any config change to ruler.

Hm, crashlooop will actually apply all config changes right?

[OGKevin]

Let me try to rephrase: ruler will crashloop not because of a config change made by a human to ruler. Ruler will go in a crashloop because a dependency, Alertmanager, is down and k8s service FQDN resolution returns a NXDOMAIN (or equivalent) and dies. Subsequently, ruler won't start up because of the DNS resolution failure and eventually ends up in a crash loop.

thanos/cmd/thanos/rule.go

Line 418 in e4941a5

addDiscoveryGroups(g, amClient, alertmgrsDNSSDInterval)

thanos/cmd/thanos/rule.go

Lines 792 to 798 in e4941a5

	g.Add(func() error {
	return runutil.Repeat(interval, ctx.Done(), func() error {
	return c.Resolve(ctx)
	})
	}, func(error) {
	cancel()
	})

[OGKevin]

The link that is being reported as dead is not actually dead. I dont quite understand why e2e failed 🤔

[pstibrany]

Change to dns package looks good to me.

[bwplotka]

Awesome, thanks for offline chat, I think this makes sense, but let's make it a normal behaviour.

Some tests would be nice as well (:

[bwplotka]

Let's delete this and if only, let's have one for both miekg and Go DNS.

[OGKevin]

hmm I don't quite understand this comment 🤔. Can you elaborate a lillte bit more?

[bwplotka]

I think I am fine with this consistency check, thanks!

[bwplotka]

Can you rebase 🤗

[OGKevin]

@bwplotka I could not rebase as I merged master in this branch before. Rebasing now will cause a headache. Do you want me to squash this to 1 commit?

[bwplotka]

Nah no need, GH squashes this. Thanks and sorry for long discussion (: