We have one DNS server where queries time out twice, and then get an answer. This can be reproduced reliably by multiple users from multiple workstations or servers. Let's say the New York DNS server is working fine, and the Los Angeles DNS server is having this problem. The output looks like this:
C:\>nslookup fs1.contoso.com 10.1.0.10
Server: ladc01.contoso.com
Address: 10.1.0.10
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
Name: fs1.contoso.com
Address: 10.3.0.16
C:\>nslookup fs1.contoso.com 10.0.0.10
Server: nydc01.contoso.com
Address: 10.0.0.10
Name: fs1.contoso.com
Address: 10.3.0.16
I have tried it from a server with Network Monitor 3.4 installed and watched outgoing traffic. The strange thing is, when querying the Los Angeles server, I do not see the first few DNS queries going outbound. If I look to the New York server, I first see a PTR resolution of the 10.0.0.10 address, and then see a query for fs1.contoso.com. If I look to the Los Angeles server, the first packet I see is a mal-formed query for fs1.contoso.com.accounting.contoso.com (accounting.contoso.com is the third suffix on the list of alternate suffixes to try in the NIC configuration). It seems to be dropping the first 5 (or so) packets expected from the query.
I am doing the monitoring on the client side, and this behavior is very reliable. I can do it 10 times toward the NY DNS server and it will succeed every time. I can do it 10 times to the LA DNS server and it will have two time-outs and then a success every time. I have created a Windows Firewall rule to permit any outgoing traffic to 10.1.0.10. I made sure this was the first rule. There are no Deny rules conflicting with it.
Any idea what could make the first couple packets disappear before even reaching the NIC internally, based on the destination?