We are running a 2008R2 domain with 2008R2 DNS servers. We have security controls that only allow 2 of our DNS servers to handle recursion. Those (2)servers(primary.contoso.com & secondary.contoso.com) have (2) external forwarders and then will go to root hints if needed. All of our other DNS servers are set to forward unknown requests to primary.contoso.com and then to secondary.contoso.com I do have it checked on all servers to use the root hints if no forwarders are available, and recursion is not disabled on any of them, just in case security ever relaxes the firewall rules and lets me add them all in. We have had them in the firewall rules before.
This morning, primarydns.contoso.com went completely offline(no ping). secondarydns kept working fine and all our dhcp scopes were using it for secondary and clients were fine. I noticed that if I ran nslookup from all the other dns servers, that the queries failed. As soon as I went to their forwarding tab and moved the secondary.contoso.com server up to the top of the list, they started working. I confirmed this on (2)different servers. Any ideas why it would not resolve until I move the secondary forwarder up on the list?
Thanks,
Dan Heim