Performance enhancements make this C# application slower...any ideas what is going on?

Question

Hello I am working on a performance engineering project. The application is a C# web application using ASP.NET and IIS. I have found many things that I know can be refactored to enhance performance. For each one I first write a test driver and test it thoroughly in a stand alone test harness. The refactored version always runs anywhere from 4 to 80 times faster. Not the entire app the pattern that I am micro-benchmarking. I then refactor the application code to put in this pattern and the application runs SLOWER. I stopped working with the code itself and started capturing metrics at the OS level. One thing I saw was context switching of 8000 to 9000 per second. I found several places in the code that are likely culprits for this. A lock inside a while loop inside another lock was one. A method that gets called all the time which creates a new thread each time was another. My theory is that the reason that changes which should be performance enhancements actually make the code run slower is that any increase of throughput just makes whatever contention is causing the context switching worse. Has anyone ever seen behavior like this before? My next step is to use some tools to find out where the contention is. Thanks

Steve · Answer

I have done that already. Nothing shows up. There are no hotspots. No method invocations consume very much CPU time. Most of them are 2% or less. That is why I am thinking this is a contention issue.

Mahesh Chand · Answer

Try Red-Gate ANTS Profiler and profile your application. It tells you exact line of code where the application must be taking too much time. It comes with 14 days free trial and I think spending $200 is not a bad idea.