Ubuntu 18.04 unable to resolve cognitiveservices DNS names

[mikeharder]

Describe the bug
Ubuntu 18.04 is unable to resolve *.cognitiveservices.azure.com DNS names by default. As a workaround, we are bypassing the local (stub) DNS server using the following command:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This may be related to https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1822416 and/or #397.

Area for Triage:
Servers

Question, Bug, or Feature?:
Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
Ubuntu 18.04 should be able to resolve *.cognitiveservices.azure.com DNS names by default.

Actual behavior
If we try to resolve a *.cognitiveservices.azure.com DNS name, it fails with SERVFAIL:

https://dev.azure.com/mharder/public/_build/results?buildId=634&view=logs&j=3dc411e8-b5bf-57f2-a8a7-b25d565c86b1&t=f636eda2-37c8-5cad-c3dc-807f9e9ed0bb&l=59

However, if we bypass the local (stub) DNS server using the following command:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Then the DNS name can be resolved successfully:

https://dev.azure.com/mharder/public/_build/results?buildId=634&view=logs&j=3dc411e8-b5bf-57f2-a8a7-b25d565c86b1&t=aef3c3f0-973b-547b-f96d-ab903995d1d8&l=59

This doesn't repro on Ubuntu 16.04:

https://dev.azure.com/mharder/public/_build/results?buildId=634&view=logs&j=88c4e28e-b89e-5514-cbb8-a3c153cbe716&t=a85e08da-6704-59a1-1759-4e62f4964eb3&l=59

Pipeline Sources: https://github.com/mikeharder/AzurePipelineTests/blob/f920bf50f72fe45c1d653a3bbaac9dcaf3df7682/azure-pipelines.yml

[mikeharder]

Also, this does not repro on a new Azure Ubuntu 18.04 VM. It only repros on a DevOps Ubuntu 18.04 Hosted Agent.

[Darleev]

@mikeharder Hello, Thank you for provided details and the investigation. I was able to reproduce the issue on Ubuntu 18.04 agent, and it seems something is configured incorrectly here, since file directly shows us that systemd-resolve service should be responsible for nameservers, but it is not.

systemd-resolve --status
.........
Link 2 (eth0)
          DNS Servers: 168.63.129.16
          DNS Domain: mqljooayxuiuncg2ys3siqhtpb.xx.internal.cloudapp.net

Locally, we still use 127.0.0.53 address, which is recorded in the file /etc/resolv.conf. It seems you are right, it is required to link local /etc/resolv.conf file with systemd-resolve file..
I will keep you posted.

[Darleev]

@mikeharder I have created Pull Request with suggested workaround. As soon as all verification processes are complete, workaround will be applied on Ubuntu 18 agents, until then your workaround is the best option here.
I will let you know additionally, when PR will be merged.

[Darleev]

@mikeharder We have added fix for the issue to the image and it will be rolled out next week.
In case of any questions, please let us know, we will be glad to assist you further.

[nerijusk]

@mikeharder I have created Pull Request with suggested workaround. As soon as all verification processes are complete, workaround will be applied on Ubuntu 18 agents, until then your workaround is the best option here.
I will let you know additionally, when PR will be merged.

The PR breaks agent build for self hosted agent pools:

Create resolv.conf link.
ln: failed to create symbolic link '/etc/resolv.conf': Device or resource busy