Tag: Performance

Using PAL to Analyze SharePoint Performance Monitor Counters

Ever have a performance problem, but don’t know what performance counters to collect or how to analyze them? The PAL (Performance Analysis of Logs) tool is a powerful tool that reads in a performance monitor counter log and analyzes it using known thresholds.

Features

-Thresholds files for most of the major Microsoft products such as IIS, MOSS, SQL Server, BizTalk, Exchange, and Active Directory.
-An easy to use GUI interface which makes creating batch files for the PAL.ps1 script.
-A GUI editor for creating or editing your own threshold files.
-Creates an HTML based report for ease of copy/pasting into other applications.
-Analyzes performance counter logs for thresholds using thresholds that change their criteria based on the computer’s role or hardware specs.

Part 1 – Set Up Pal

1. Go to http://pal.codeplex.com/ and download the latest version of PAL onto your server.
2. Install the program via the Setup.exe file

Part 2 – Set Up Performance Counters

1. Open up the Performance Monitor snap-in (Start -> Run -> perfmon)
2. Expand the “Monitoring Tools” option and Right Click on “Performance Monitor” -> “New” -> “Data Collector Set“
3. Give the data collector set a name and click on Next
4. Give the location where you want to save the Perfmon Logs and click on Next
5. Here you can provide the Run As user of leave that as the Default user
6. Select the option “Save and Close” and click on Finish
7. Then go back to the Perfmon snap-in main window and you should see a Data Collector set with the same name that you created under Data Collector Sets -> User Defined
8. Click on the Data Collector Set and on the right hand pane, you should see a System Monitor Log Performance Counter. Right click on it and click on properties.
9. Under the Performance Counters tab, add the following SharePoint-specific performance counters required for your data collection. You can for some counters isolate the counters to specific web applications in order to narrow down things a bit:

11. Set the log format as binary, and the sample collection interval time to 5 seconds.
12. Under the File tab specify the file format name and logging properties for the file.

Part 3 – Process Your Performance Monitor Output Logs with PAL

1. Open up PAL from the Windows Start Menu:

2. The welcome page :

3. Select the .BLG binary PerfMon you collected earlier:

4. Choose the SharePoint 2010 Threshold File:

5. Answer the Questions! Select your relevant info including number of CPUs and Total Memory (PAL doesn’t detect automatically):

6. Choose your Output Options:

7. Select your file output variables and path location:

8. View the queue:

9. Execute!

10. Analysis in progress:

11. After it’s done processing, PAL outputs a beautiful, prioritized HTML page breaking down your performance pains and referencing solutions to fix them. By Default it saves it to PAL Reports:

This is a vastly useful tool which can similarly be jigged to work to turn the piles of data output by PerfMon into useful, actionable intelligencefor much more than just SharePoint. Thanks to Clint Huffman for creating and maintaining this!

The Top 10 Problems With SharePoint Performance Top 10 Lists

I’ve come across lot’s of helpful lists around the net and forums on the topic of performance tuning SharePoint however as time and performance projects march forward it’s clear that this subject, more than most in the already dense realm of SharePoint, deserves a more realistic outlook. The standard lists of performance tuning tips look at the forest for the trees. This list looks at the forest as a melee battle between good and evil and er.. anyhow, read on for some twists on the old tuning bag of tricks:

1. Wack a Mole Syndrome

Get verified, fully detailed user stories complete with IP address, browser version/settings/plugins, OS version, Fiddler, YSlow debug reports, etc. BEFORE making any analysis. Failure to do so (resulting on heresay, opinions or assumption) will lead to a non-productive game of wack-a-mole.

2. SharePoint performance tuning is incremental, not subtractive

1 + 1 does not equal 2 when untying complex relationships between SharePoint, IIS, Active Directory, SQL, Project Server or whatever else is in the mix in your SharePoint performance issue. For example, in one farm we found reducing the SharePoint object cache file size from 500 MB to 100 MB caused a corresponding reduction in the W3WP.exe process for that web application from 1.5GB to 300MB – sometimes resource problems are exacerbated in a bell curve (or even less predictable patterns) rather than an simple formula.

This requires multiple passes at problem resolutions, resolving one (or one group of) problems at a time rather then just running through a laundry list of to-do items and calling it a day. You have to be able to get buy-in from your client or team that the issue(s) may not be nailed with one go at it.

3. Scale up or scale out

We all know SharePoint plugs in as a web farm out of the box. You have your App Servers, your Web Front Ends, your Search Servers etc. What is more of a grey area is the threshold of, when does one add more RAM/better CPUS/Disks, versus adding a secondary App or WFE server? There are tools like the HP SharePoint Sizer which try and point you to the hardware required based on some fixed (in that case, vendor-specific) formulas. The only way to get an accurate assessment of when it’s time to order some new servers or whether you just need your VM guys to allocate some more RAM, is to have an SharePoint Consultant complete a full assessment of your situation.

Following a cookie cutter table of specs and platforms will just lead to grief.

4.Cumulative Updates, Service Packs, Regressions demand constant attention

You need to be in the mix with SharePoint day-in, day-out in order to keep on top of the various updates, regressions, bugs and workarounds. What may solve Performance Problem X may work on CU but not another. Committing to performance fixes without knowing how they sit in the living, moving behemoth of code that is the SharePoint framework is the path to frustration.

5. VMware abstracts the problem


Virtualization is popular. Virtualization allows IT guys to give the guys in suits what they need with the push of a button rather than with capital PO’s. Therefore: VM admins have an increasing importance in organizations as they are power brokers who can called on to quickly breath life into stakeholders projects by giving them the virtual juice they need. The problem seems to be that the new-found speed of infrastructure deployment has downplayed the old tendency of hyper-critical evaluation of infrastructure – when we used to pay for every nut and bolt of a new server farm deployment the specs involved would be of supreme importance and it seemed more thought was given as to what the applications deployed on the hardware actually require.

Particularily due to SharePoints tendency to be adopted as a fragile pilot project (and subsequently taking the company by storm and absorbing silos left right and center), there is a trait of starting off farms on VM’s starved for resources with the idea “well if this isn’t enough then we can just flip the switch and add more (RAM/Disk/CPU) later”. This is a recipe for a rocky SharePoint adolescence – there are many aspects of virtualization that do not add

For example, Disk Defragmentation of the guest OS is considered a laughable or even harmful tactic by many VM admins. I try to disprove that (with Bob Nolan of PerfectDisk’s generous help) in my blog post here.

Brent Ozar is a kingpin on the topic of SQL virtualization and he point’s out some reasons on why you should perhaps not virtualize SQL server at all.

So try explaining that stuff to your local Xerxes VMWare Admin

6. Company Politics


Company politics makes keeping a sensible amount of web parts and content on the page difficult. Design by Committee results in the inevitable ultra-busy intranet home page which will invariably contain:
– A Yahoo stock ticker
– A photo slideshow
– Pictures of the CEO and Board of Directors
– Some form of social company initiative e.g. employee of the week and their hobbies
– ..and so forth

Each one of these little pieces of real estate on the homepage has typically been fought for by stakeholders in meeting after meeting. Frustration levels rise as initiatives may get squashed from higher up. People get asked to come in on Saturday to fix up a webpart, emotional investment in particular web parts or content grows.

Now you, as the one assessing performance, come in and see that this assembly of intentions is actually overloading the Win XP / Internet Explorer IE7 workstations of the end users or there are too many SQL connections coming from some custom web part. Your recommendation of stripping off web parts and content falls in line with modern usability best practices. Be ready for the sparks to fly, or even worse, for nothing to happen at all as this emotional investment in content or functionality may preclude the client from giving you the ability to actually resolve the problem.

You can tell the trurth and share that time and time again you have seen a simple search interface on the home page, combined with with key navigation elements, result in the best user experience. Be prepared to be told to fix up performance some other way as the politics factor may preclude you from being able to get the job done.

7. Best practice or immediate fix? Guess which one impresses SharePoint stakeholders more.


Related to the 6. Company Politics section, opinions rarely sail on the same seas as performance tuning. Custom branding or development decisions may lead to Catch 22’s which prevent you from making the “right” choice. “This ain’t drywalling” is a nice saying of mine, but really, sometimes we are required to just slap some stuff on top of a problem area.

8. It’s the Content, Stupid

90% of web page “load” is typically getting content from the server to the client browser and the time it takes for the browser to render it.The following items are exacerbating the situation with SharePoint:
– Jquery-enabled web parts
– Content Editing staff who sneak in random Javascript/Jquery into Content Editor Web parts in order to achieve a quick fix to a layout or design issue, instead of creating custom branding or web parts. This Javascript is fired off at all kinds of random moments in the page life cycle if not carefully controlled and causes hiccup effects in the page load
– The ease of which people can upload images into SharePoint means more than ever we are seeing staff who upload non-web optimized images into SharePoint, resulting in the classic scenario of a home page slide show with 7 five megabyte images

So the problem reverts from being a “SharePoint” problem and becomes a plain ol’ web content management problem. This is an important distinction to make during a performance assessment as there are often stakeholders who have spent years perfecting their mastery over their “silos” (Lotus Notes, Oracle etc.) who now have to submit to the crushing (yet invigorating) force of complete corporate assimilation by SharePoint – and they would like very much to pin performance issues on the SharePoint boogeyman. Being able to isolate basic web content management issues is vital and could potentially lead to considerable savings in a performance roundup as web content consultants are generally cheaper then SharePoint consultants.

9. Maytag Repairman Syndrome

You can run through all the TechNet best practices and fix up loose ends that the original SharePoint farm architects may have missed or misunderstood. You can set up quotas, split up content DB’s, and generally soup up that farm for all she’s worth: however unless you carefully document what you’ve done and can provide accurate projections of before and after results no one will ever notice your work. You will end up racking up the hours and become an underappreciated lonely repairman without any work as the tuned up SharePoint farm sails smoothly away into the future.

10. Integration is Exacerbation

SharePoint’s key benefit to an organization is it’s integrated environment. It is a Document Managment system, Corporate Homepage, business process repository (workflows) and more. The good news is that more and more legacy and external systems are being rolled up or tied in to the SharePoint framework. The bad news is that when you wire up your Oracle, Project Server, or specialized industry-specific software suites into SharePoint you can compound performance problems. This is why using tools such as Red Gate’s Memory and Performance Profiler tools can help you isolate leaky or load-generating .DLL’s and connections.

Certificate Revocation Lists – Slow Spin up times

What

Most Microsoft assemblies and DLLs are digitally signed. Each time signed assemblies are loaded, default system behaviour is to check with the owner of the root certificate that the cert with which the assembly was signed is still valid. In the case of Microsoft assemblies, this means “phoning home” to read the Certificate Revocation List at crl.microsoft.com .

Whilst this is all very well and good if you have an Internet connection, sometimes you don’t have this luxury. Many web servers, for instance, don’t have outbound Internet accessibility. The CRL check will attempt to connect to Microsoft’s servers and then timeout, usually within 30-60 seconds.

With SharePoint, you’ll get a lot of delays in this scenario. One way to check if your server is affected by this condition is to open up a SharePoint Management Console PowerShell window and run the “STSADM -help” command. If it takes 30 seconds or more to display the usage instructions, then you will be experiencing really slow server performance.

So What

You make the first request of the day, or the first request after recycling the app pool because you are developing assemblies that sit in the GAC.There is a delay of about 2 minutes
While you are waiting, and tearing your remaining hair out because you know you have to do this at least 50 times today, there is no CPU activity, swapping or significant network traffic.

After the timeout the assembly is still loaded and the software works as expected, though very slow every time a new signed assembly is loaded for the first time, which happens a lot. The worst thing is that no entries are written to the event log and no exceptions are thrown so you are left completely in the dark about why your application is so bloody slow.

Now What

Our esteemed vendor Jereon from Muhimbi has a great explanation here and some options for you: http://blog.muhimbi.com/2009/04/new-approach-to-solve-sharepoints.html

Joel as well, and he even includes a PowerShell that will run it all down for you.
http://joelblogs.co.uk/2011/09/20/certificate-revocation-list-check-and-sharepoint-2010-without-an-internet-connection/

..and finally a SharePointBlues post on a bevy of Certificate-related issues:
http://www.sharepointblues.com/2012/01/09/sharepoint-certificate-errors/

vSharePoint March 15 – SharePoint Acceleration | Performance for the Rest of Us

Here is the roundup of the SharePoint Performance Presentation I gave at vSharePoint on March 15. Be sure to visit the recently revamped site at www.vsharepoint.com too!

SharePoint Performance MindMap (SWF)
March 2012 vSharePoint Meeting (PDF)

  • Keith Tuomi:  “SharePoint Acceleration | Performance for the Rest of Us”
  • Wolfi Holzmann: “SharePoint 2010 Workflows Authoring in Visio Premium 2010”

Optimize images for web directly inside Visual Studio 2010

Probably in the 90s sometime, you achieved insider web design knowledge learning that you should not really be referencing a 4MB image file as little content image in a web page. Friends, family and associates we’re amazed as you made their Geocities pages suddenly 10X faster by replacing the image bloat. These days, the people who shoot you your imagery are generally a little smarter about fixing up their icons/logos/layout element images but there is ALWAYS more blood available to be squeezed from the stone of web images.

You still don’t believe just saving at 75% quality “for web” in Photoshop is the universal fix for web image optimization, do you? It’s not exactly life’s little joyful task to take what Joe the Marketing guy down the cube farm sent you for content or icons, and turning it into SEO and visitor-friendly pixel greatness. Especially when you’re a Developer who knows that fractional performance tuning (in this case, shaving off file size KB) in web application content is rarely going to be noticed by your team, unless it’s missing.

I found a nice little plugin for VS 2010 at http://visualstudiogallery.msdn.microsoft.com/a56eddd3-d79b-48ac-8c8f-2db06ade77c3 that let’s you execute the tedious chore of image optimization directly inside VS 2010.

Adds a right-click menu to any folder and image in Solution Explorer that let’s you automatically optimize all PNG, GIF and JPEG files in that folder. The optimization doesn’t effect the quality of the images, but optimizes them using industry proven algorithms for removing EXIF and other metadata. The extension uses SmushIt and PunyPNG for optimizing the images. Check out the short video demo.