Server 2k8 r2: UDP traffic from over 30K IPs causing tcpip.sys to contend on spinlocks resulting in 1/2 second DPCs. Works fine in Win2k3.

We have a UDP server that accepts many connections.  In Win2k3 it can talk to anywhere from 90K to 120K IP addresses with no problem.

On Win2k8R2 it throttles and often tips over at 30K users.

I have xperf logs showing the problems.  DPCs start out taking reasonable amounts of time, but eventually grow to take half a second upwards of 3/4 of a second; at that point the network stack seems to tip over and start again.

I've appended some of the xperf data.  I don't see a way to provide an attachment.

The nic is the Intel 82576 with only one link active.  Again - same code same hardware runs fine on win2k3.

There are various routes to it, but huge swaths of code always end up in: NDIS.SYS!NdisAcquireReadWriteLock.

I would assume on the same lock, and I would guess it's a spin lock since it spends all its time in ntkrnlmp.exe!KeAcquireSpinLockAtDpcLevel.

In our sample the process had a weight of 540,714 units.  Of that at least 210,597 ended up trying to acquire that lock via WSASendTo.  Gory details appended.

From the DPC view, you can see the calls that were taking 1/2 a second or longer.  There are 110 of them in a 37 second sample.  The time to process varies randomly and there are many thousands in between.  This is appended as the second set of voluminous data.

OS is Win2k R2 SP1

The Intel 82576 NIC has the latest available drivers

I've already tried altering just about every network parameter to observe the results.  RSS enabled/disabled. RSS NIC configured for 1 or more CPUs.  Various timings for Interrupt deferral, etc...

The net effect of all this was that I could move the shape of the curves, and even stop the network from tipping over entirely; however the DPC times are still awful; the spinlocks are still spinning and the net throughput is still 1/4 of Win2k3.

Any suggestions?  What's the next step?

Joe

DPC Raw Info: