Inconvenient MVC Web Optimization Framework and High CPU Utilization Issue

What was the issue?

Last week, while working with the azure cloud services we tried to stress test it and guess what we found? CPU spikes!!

Initially we thought that this might be something related to load on the server as it was a small VM instance but at the same time the results were unbelievable as number of concurrent requests to the server were comparatively less. This gave us the kick start and we started to drill down to the real issue.

The approach:

It all started with several meetings with few brainy people around because I was clueless, some even directly said man your application got memory leaks (I found this quite exaggerative) but then
Thanks to the Visual Studio profiling framework which helped me a-lot to drill down into real issue of this CPU spikes. You can read about how to use this feature and how it helps in my blog post here profiling windows azure cloud services.

When I tried to stress test the service and tried to collect the profiled logs, found that there were these two methods which were taking almost more than 80% of CPU, wow!!

1. Scripts.Render
2. Styles.Render

Note: I was using ASP.NET MVC web optimization nuget v 1.0

Irony is, these methods are from the web optimization framework meaning, these are introduced to improve the performance of your web application by bundling the java-scripts and css files. More info about bundling here: Bundling and Minification.

All right - then of course the straight way to remove these calls from application and try to stress application again and check results. We did exact same thing and this time application performed pretty well!! No CPU spikes this time!! Man bingo, surprising but desirable result!!!!!!

Then why the optimization framework was doing something like this? Why? Why, why and why.......? This question even appeared in my dreams, lolz !!


I started search on it but didn't find anything helpful, I even started doubting my decision about removing those calls from the application unless I hit these posts here:

• MVC 4 on Azure - @Scripts.Render(“~/bundles/jquery”) slow and
• Azure Cache (Preview) OutputCache - High CPU - Slow

Here is why and *purely based on my assumption*

When you use this bundling framework (a.k.a. Web optimization framework), server uses the caching. All it needs to do is – bundle those scripts and CSS files, cache those on server and use it while sending response back to the clients.

Usually these caching settings are defined in the web configuration file of the application. These are the settings which talks about what type of caching needs to be used and what are its providers.

Things might have worked pretty fine if I was using the default caching and providers, however as this application was built for cloud, I was using the azure managed cache service which is really a distributed cache and hence the my caching settings in the web configuration file looked something like this:

<caching>

   <outputCache defaultProvider="DistributedCache">

      <providers>

         <add name="DistributedCache"

         type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider,

         Microsoft.Web.DistributedCache" cacheName="default" dataCacheClientName="default" />

      </providers>

   </outputCache>

</caching>

Now here is the trick, seems the web optimization framework (v1.0) didn’t know distributed cache providers and hence it simply ignored the caching, which indeed lead to recompilation of your bundles every time with each request and hence caused higher CPU utilization.

Now here is what I did to prove the assumption mentioned above -
Removed both from my application i.e. bundling as well as distributed caching settings - it worked fine and CPU utilization never touched the boundary.

Takeaways:

In case you are facing the same issue of high CPU consumption on the web server in your ASP.NET MVC application or cloud service then,

1. Run the visual studio profiler locally and get to the root cause.
   
   
2. If you find the Optimization framework APIs are taking most of the CPU, remove those and load test your application, see the results to validate your findings.

Latest developments:

According to Enable server cache for non default output cache providers - it’s the discussion on codeplex on this very same issue, but if you go and see the comments section then it is said that they have fixed it in the next version of nuget. (i.e. post version 1.0).

I haven’t tried using next version of this Nuget but when I will do it, I will update this post.