In this post, I’ll be talking about an issue which we recently encountered in one of our customers' SharePoint 2010 environments
Let me talk about the issue first, and how we can fix it. Later, we will discuss the lessons we learned from it. Issue we faced A couple of days back, we received a huge number of support incidents from the users across the globe stating that they were not able to access SharePoint portal. For some reason, the issue was intermittent. The site used to load fine at times, but all of a sudden, it stopped loading. We decided to login to each WFE servers in the farm and identify the one which was throwing the bad request. Listed below are the steps which we did initially to identify the root cause.the troubleshooting steps done by us, initially:
Now, let’s take a look on what Microsoft had to say to us about this issue …. Troubleshooting steps done by Microsoft We captured the ULS logs on the exact time the issue was reported and shared the same to Microsoft (please note that this issue which we are currently talking about is a non-reproducible one, meaning: we were not able to reproduce the same behavior to Microsoft as this happened only once. After the server reboot, everything looked normal). Microsoft analyzed the logs and this is what they found …. A huge performance issue was identified, as you can see in the logs below: AppDomain recycling was happening very frequently (shown below). The App domain recycling was happening on both the WFE’s, as shown in the ULS logs screenshot below: What we identified after analyzing the logs? Now, based on the above analysis, we identified that the root cause of this issue was the AppDomain recycling happening very frequently. This is an isolation process within the W3WP process of the web application. This process went on recycling and that caused the performance issue of the environment. The possible root cause for this App Domain recycle can be because of the below mentioned two reasons:
Lessons learned If you’re planning to install Antivirus software in your SharePoint farm, please make sure that all the folders mentioned in this article are excluded from getting scanned. These are SharePoint system related files & folders; and every time the AV scan engine tries to scan these files, it puts the farm on risk as the scanning process will interfere the SharePoint’s operations.
Essentials of SharePoint 2010: Business Intelligence Capabilities