Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query corednse2e.com using AzureDNS as upstream server. Resp is truncated to 512 bytes, but TC flag is not set. #6370

Closed
SriHarsha001 opened this issue Oct 23, 2023 · 3 comments
Closed

Query corednse2e.com using AzureDNS as upstream server. Resp is truncated to 512 bytes, but TC flag is not set. #6370

SriHarsha001 opened this issue Oct 23, 2023 · 3 comments
Labels
bug

Comments

@SriHarsha001
Copy link
Contributor

What happened:
Use CoreDNS > v1.10.1 to query corednse2e.com using AzureDNS as upstream DNS server in an AKS cluster. Response is truncated to 512 bytes and TC flag is not set.

CoreDNS using AzureDNS as upstream server. There is no TC flag, but the response is truncated. 
---------------------------------------------------------------------------------------------------
kubectl exec dnsutils -- dig corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @10.96.0.10
; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44737
;; flags: qr rd ra; QUERY: 1, ANSWER: 30, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;corednse2e.com.                        IN      A

;; ANSWER SECTION:
corednse2e.com.         30      IN      A       10.96.0.247
corednse2e.com.         30      IN      A       10.96.0.55
corednse2e.com.         30      IN      A       10.96.0.97
corednse2e.com.         30      IN      A       10.96.0.242
corednse2e.com.         30      IN      A       10.96.0.254
corednse2e.com.         30      IN      A       10.96.0.248
corednse2e.com.         30      IN      A       10.96.0.96
corednse2e.com.         30      IN      A       10.96.0.95
corednse2e.com.         30      IN      A       10.96.0.252
corednse2e.com.         30      IN      A       10.96.0.255
corednse2e.com.         30      IN      A       10.96.0.50
corednse2e.com.         30      IN      A       10.96.0.99
corednse2e.com.         30      IN      A       10.96.0.51
corednse2e.com.         30      IN      A       76.223.105.230
corednse2e.com.         30      IN      A       10.96.0.52
corednse2e.com.         30      IN      A       10.96.0.246
corednse2e.com.         30      IN      A       10.96.0.94
corednse2e.com.         30      IN      A       10.96.0.93
corednse2e.com.         30      IN      A       10.96.0.54
corednse2e.com.         30      IN      A       10.96.0.92
corednse2e.com.         30      IN      A       10.96.0.91
corednse2e.com.         30      IN      A       10.96.0.249
corednse2e.com.         30      IN      A       13.248.243.5
corednse2e.com.         30      IN      A       10.96.0.244
corednse2e.com.         30      IN      A       10.96.0.251
corednse2e.com.         30      IN      A       10.96.0.243
corednse2e.com.         30      IN      A       10.96.0.253
corednse2e.com.         30      IN      A       10.96.0.250
corednse2e.com.         30      IN      A       10.96.0.98
corednse2e.com.         30      IN      A       10.96.0.53

;; Query time: 82 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Mon Oct 23 19:19:58 UTC 2023
;; MSG SIZE  rcvd: 512

// No retrying in TCP mode. 
kubectl exec dnsutils -- nslookup corednse2e.com 10.96.0.10
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
Name:   corednse2e.com
Address: 10.96.0.255
Name:   corednse2e.com
Address: 10.96.0.250
Name:   corednse2e.com
Address: 10.96.0.99
......
......


8.8.8.8 - There is TC flag and response is truncated. 
---------------------------------------------------------------------------------------------------
kubectl exec dnsutils -- dig corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @8.8.8.8
; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54576
;; flags: qr tc rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;corednse2e.com.                        IN      A

;; Query time: 14 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Oct 23 19:13:50 UTC 2023
;; MSG SIZE  rcvd: 32

// Retrying in TCP mode. 
kubectl exec dnsutils -- nslookup corednse2e.com 8.8.8.8
;; Truncated, retrying in TCP mode.
Server:         8.8.8.8
Address:        8.8.8.8#53


AzureDNS 168.63.129.16 - There is TC flag and response is NOT truncated. 
---------------------------------------------------------------------------------------------------
kubectl exec dnsutils -- dig corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @168.63.129.16
; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> corednse2e.com +ignore +noedns +search +noshowsearch +time=10 +tries=6 @168.63.129.16
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34304
;; flags: qr rd ra; QUERY: 1, ANSWER: 64, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;corednse2e.com.                        IN      A

;; ANSWER SECTION:
corednse2e.com.         600     IN      A       76.223.105.230
corednse2e.com.         600     IN      A       13.248.243.5
corednse2e.com.         600     IN      A       10.96.0.91
corednse2e.com.         600     IN      A       10.96.0.92
corednse2e.com.         600     IN      A       10.96.0.93
corednse2e.com.         600     IN      A       10.96.0.94
corednse2e.com.         600     IN      A       10.96.0.95
corednse2e.com.         600     IN      A       10.96.0.96
corednse2e.com.         600     IN      A       10.96.0.97
corednse2e.com.         600     IN      A       10.96.0.98
corednse2e.com.         600     IN      A       10.96.0.242
corednse2e.com.         600     IN      A       10.96.0.243
corednse2e.com.         600     IN      A       10.96.0.99
corednse2e.com.         600     IN      A       10.96.0.244
corednse2e.com.         600     IN      A       10.96.0.246
corednse2e.com.         600     IN      A       10.96.0.247
corednse2e.com.         600     IN      A       10.96.0.248
corednse2e.com.         600     IN      A       10.96.0.249
corednse2e.com.         600     IN      A       10.96.0.250
corednse2e.com.         600     IN      A       10.96.0.251
corednse2e.com.         600     IN      A       10.96.0.252
corednse2e.com.         600     IN      A       10.96.0.253
corednse2e.com.         600     IN      A       10.96.0.254
corednse2e.com.         600     IN      A       10.96.0.255
corednse2e.com.         600     IN      A       10.96.0.50
corednse2e.com.         600     IN      A       10.96.0.51
corednse2e.com.         600     IN      A       10.96.0.52
corednse2e.com.         600     IN      A       10.96.0.53
corednse2e.com.         600     IN      A       10.96.0.54
corednse2e.com.         600     IN      A       10.96.0.55
corednse2e.com.         600     IN      A       10.96.0.57
corednse2e.com.         600     IN      A       10.96.0.58
corednse2e.com.         600     IN      A       10.96.0.59
corednse2e.com.         600     IN      A       10.96.0.60
corednse2e.com.         600     IN      A       10.96.0.61
corednse2e.com.         600     IN      A       10.96.0.62
corednse2e.com.         600     IN      A       10.96.0.63
corednse2e.com.         600     IN      A       10.96.0.64
corednse2e.com.         600     IN      A       10.96.0.65
corednse2e.com.         600     IN      A       10.96.0.66
corednse2e.com.         600     IN      A       10.96.0.67
corednse2e.com.         600     IN      A       10.96.0.68
corednse2e.com.         600     IN      A       10.96.0.69
corednse2e.com.         600     IN      A       10.96.0.70
corednse2e.com.         600     IN      A       10.96.0.71
corednse2e.com.         600     IN      A       10.96.0.72
corednse2e.com.         600     IN      A       10.96.0.73
corednse2e.com.         600     IN      A       10.96.0.74
corednse2e.com.         600     IN      A       10.96.0.75
corednse2e.com.         600     IN      A       10.96.0.76
corednse2e.com.         600     IN      A       10.96.0.77
corednse2e.com.         600     IN      A       10.96.0.78
corednse2e.com.         600     IN      A       10.96.0.79
corednse2e.com.         600     IN      A       10.96.0.80
corednse2e.com.         600     IN      A       10.96.0.81
corednse2e.com.         600     IN      A       10.96.0.82
corednse2e.com.         600     IN      A       10.96.0.83
corednse2e.com.         600     IN      A       10.96.0.84
corednse2e.com.         600     IN      A       10.96.0.85
corednse2e.com.         600     IN      A       10.96.0.86
corednse2e.com.         600     IN      A       10.96.0.87
corednse2e.com.         600     IN      A       10.96.0.88
corednse2e.com.         600     IN      A       10.96.0.89
corednse2e.com.         600     IN      A       10.96.0.90

;; Query time: 79 msec
;; SERVER: 168.63.129.16#53(168.63.129.16)
;; WHEN: Mon Oct 23 19:24:14 UTC 2023
;; MSG SIZE  rcvd: 1056

What you expected to happen:
Querying CoreDNS, when the response is truncated, TC flag should be set.

How to reproduce it (as minimally and precisely as possible):
Create an AKS cluster
Change the coredns version to 1.10.1 or 1.11.1
Deploy a dnsutils pod, and run nslookup corednse2e.com.
run nslookup corednse2etestaks.com. -- Works on latest from master branch

Environment:
AKS, AzureDNS

  • the version of CoreDNS:
  • Corefile:
    .:53 {
        errors
        ready
        health {
          lameduck 5s
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
  • OS (e.g: cat /etc/os-release):
cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
@SriHarsha001 SriHarsha001 mentioned this issue Oct 24, 2023
Closed
@chrisohaver
Copy link
Member

Have you opened a bug with Azure DNS?

@SriHarsha001
Copy link
Contributor Author

SriHarsha001 commented Oct 26, 2023 •
edited

Yes Chris, I have already opened a bug with Azure DNS. But this issue can happen due to any upstream DNS (custom DNS) servers (e.g. sending >512 byte responses without an eDNS0 OPT RR).

So we cannot always rely on upstream DNS server and need to handle this scenario.

I will try my best to explain the scenario.

For example -

  • When domain name has 15 characters, offset after unpackQuestion will be 32.
  • Each loop adds 30 to offset and when number of A records in response is more than 15, then offset calculated in unpackRRslice will keep increasing in this sequence 62,92,122,152,182.....482,512
  • In 16th loop, offset (32 question bytes + 30 * 16) == 512 which will be == len(msg) for default UDPsize, so in this case, there is no Overflow error returned.
  • And the response will be truncated (ie response does not contain all the A records) and TC flag is false. This is a failure scenario.

Similarly,

  • When domain name has 19 characters, offset after unpackQuestion will be 36.
  • Each loop adds 34 to offset and when number of A records in response is more than 14, then offset calculated in unpackRRslice will keep increasing in this sequence 70,104,138,.....512
  • In 14th loop, offset (36 question bytes + 34 * 14) == 512 which will be == len(msg) for default UDPsize, so in this case, there is no Overflow error returned.
  • And the response will be truncated (ie response does not contain all the A records) and TC flag is false. This is a failure scenario.

@SriHarsha001
Copy link
Contributor Author

SriHarsha001 commented Oct 30, 2023 •
edited

Created a bug here - miekg/dns#1492

Submitted a PR here - miekg/dns#1493

So will close this.

@SriHarsha001 SriHarsha001 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 30, 2023
@SriHarsha001 SriHarsha001 mentioned this issue Oct 31, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SriHarsha001 @chrisohaver