My checklist for optimizing SharePoint sites

Optimization is probably one of my favorite SharePoint topics – it seems there’s always a new trick to learn, and I love the fact that a minor tweak can have a dramatic impact on how well your site works. There are some great resources out there now for information on how to boost performance, but it strikes me that there isn’t one single paper or article which contains all the aspects I personally consider or have been relevant to me on past projects. Hence this article is essentially an aggregation of some of these sources. The aim isn’t for it to be “complete” (if there is such a thing) or authoritative in any way, and I know I wouldn’t be the first person to blog on this subject – it really is just a reminder list for me to refer to, but if it helps you also that’s great.

Key resources:

Before I break into the list, remember that different optimization measures will have different effects depending on your circumstances – having the best checklist in the world is really no substitute for the SharePoint architect’s ability to identify what is likely to be the bottleneck for a given implementation. Additionally my list is quite “developer-focused”, but if the infrastructure aspects of the implementation is the limiting factor (e.g. latency between web servers and SQL, amount of usable RAM etc.) you can probably optimize code until you’re blue in the face without solving your performance problem.

My list is broken down into sections, and some of the items have links to some useful resource or other which you might find helpful if the topic is new to you.

Configuration

Items in this section typically have  a big impact – in particular output caching, which is the first thing to consider on WCM sites for example.

Code/development

  • Test with SPDisposeCheck to ensure no memory leaks
  • Measure page payload weight (again, YSlow useful here) – aim to reduce as far as possible
  • Eliminate unnecessary ViewState! A good dev technique may be to turn off ViewState in web.config and then override it in controls/pages which really do need it (haven’t tried this, but keep meaning to).
  • Ensure not downloading core.js etc for anonymous users/delay-loading for authenticated.
  • Ensure image sizes are small/images are optimized for web (still comes up!)
  • Always ensure height/width attributes specified on images
  • Ensure custom CSS is factored correctly
  • Don’t forget client-side code efficiency e.g. CSS, jQuery selectors
  • Consider using a code-profiling tool such as ANTS profiler or dotTrace to identify sub-optimal code
  • Ensure general good coding practice e.g. your optimization work may well be undone if your code is badly-written. Accidentally doing unnecessary processing in a loop (in server-side OR client-side code is one example I’ve seen many times)
  • Be wary of performance impact of custom HTTP modules – I frequently see huge perf gains when running load tests with such modules enabled/disabled. Removing core.js as a post-processing step within a HTTP module is NOT the right approach kids! (No really, I’ve seen it done..)

Code/development – advanced

These tips might only really be necessary if you’re implementing a high-traffic site, but are worth considering:

  • Consider factoring custom JS/CSS into fewer files – fewer HTTP requests is better
  • Consider “minifying” JS
  • Consider using CSS to cluster background images (all images stitched into one for fewer HTTP requests) – AC might have mentioned somewhere an online service to automate this..

Architecture

You should consider this section absolutely incomplete – hopefully there’s enough here to dissuade anybody of the notion that it’s only about code though! The first two in particular are key.

  • 64-bit/sufficient available RAM etc.
  • Latency between web and SQL – MS suggest a ping response of less than 1 millisecond
  • Consider a CDN solution (e.g. Akamai, Limelight etc.) if users are geographically far away from servers (and y0ur client can afford it!)
  • Ensure not using web garden with BLOB cache (known BLOB cache issue)
  • Are app pool settings correct – check for excessive app pool recycling by enabling logging to the Windows event log using – cscript adsutil.vbs Set w3svc/AppPools/ <YourAppPoolName> /LogEventOnRecycle 255. Once or twice per day is probably the maximum you’re hoping for.
  • Is storage correctly architected e.g. are OS, data, logs on different disk spindles?
  • Is storage correctly configured (e.g. SAN configuration) – are your LUNS giving you sufficient IOPS?!


Using YSlow to help optimize websites

One particular tool worthy of discussion when we’re talking about optimizing websites is YSlow – if you haven’t come across it yet, this is a Firefox plugin which extends Firebug) developed by Yahoo! developers which uses their optimization ruleset (customizable) to report on and help further streamline your site. The tool focuses on communication between the server and browser – clearly a browser tool like this can’t identify any back-end issues, but it’s always interesting to run a site through the checks.

The first tab gives a ‘grade’ for your site based on the ruleset, helping you identify areas for improvement (I’ve highlighted where the page stops and the YSlow console starts with a red box):

YSlow_Grading 
The next tab provides a breakdown of files downloaded by the request, and allows me to see their size and headers etc. In particular I can see the expiry date and Etag and so on which the files are being tagged with so they can be cached locally:

YSlow_Components

The statistics tab provides some good analysis on the page weight and also illustrates the difference between a first visit and subsequent visits where appropriately tagged files will be served from the browser cache:

YSLow_Stats

Finally the Net tab in Firebug is also interesting, as this shows me how files were downloaded (sequential or in parallel) – the most recent browsers do a much better job of this, with IE8 and FF3 being able to open 6 channels per URL domain to download files in parallel, but note that IE7 could only open 2.

Firebug_NetAnalysis

From this, I also see the http://sharepoint.microsoft.com site (great site btw) I’ve been analyzing also displays the BLOB cache issue I discussed recently (now confirmed as a bug) where incorrect headers are added to files stored in the Style Library, causing lots of unnecessary HTTP 304s to go across the wire. So YSlow does give a great insight – however I do agree with Jeff Atwood’s reminder that some of the reported “issues” may not be the biggest concern for your site. As always, apply common sense.

Summary

Optimization is a deep topic, and this checklist is simply my reminder of some of the things to consider, though in reality a solid understanding of the nuts and bolts is required to really architect and develop high-performing sites. Tools like YSlow can also help with some aspects of optimization.

So which optimization nuggets did I miss which you consider? Feel free to leave a comment and add to this list..

Sample code from my top WCM tips presentation

Since AC suggested I post some sample code for some of the points I was making in my top 5 WCM tips presentation, I’ve now done just that. In the code you can find examples of the following:

  • Exception handler HTTP module – sends notification e-mails when an unhandled exception occurs and redirects the user to a configurable ‘friendly’ error page
  • Custom base class for master page – this is almost a skeleton class since your implementation will vary, but demonstrates how to use a custom base class
  • Trace helper class and matching ReSharper templates – for logging what happens in your code (this one’s a bonus since I recommend these classes implement trace!)

These files can be downloaded in a Visual Studio project from:

http://sharepointchris.googlepages.com/wcmsamplecode

But for those of you who’d just like a quick look at the code, see below.  The actual classes in the project have comments to help you ‘use’ these things in your site – these have been removed below for readability:

Exception handler HTTP module code:

using System;
using System.Diagnostics;
using System.Text;
using System.Threading;
using System.Web;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Utilities;
using COB.SharePoint.Utilities;

namespace COB.Demos.WcmSamples
{
public class ExceptionHandlerHttpModule : IHttpModule
{
#region -- Private members --

private TraceSwitch traceSwitch = new TraceSwitch("COB.Demos.WcmSamples.ExceptionHandlerHttpModule",
"Trace switch for the COB.Demos.WcmSamples.ExceptionHandlerHttpModule.");

private TraceHelper trace = null;

#endregion

#region -- Constructor

public ExceptionHandlerHttpModule()
{
trace = new TraceHelper(this);
}

#endregion

public void Init(HttpApplication context)
{
context.Error += context_Error;
}

private void context_Error(Object sender, EventArgs e)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "context_Error(): Entered error-handling module.");

// collect the last error..
var exception = HttpContext.Current.Server.GetLastError();

// process it/send e-mail..
processError(exception);

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "context_Error(): Finished handling error, " +
"about to redirect.");

// and finally clear the error and redirect to the friendly error page..
HttpContext.Current.Server.ClearError();
HttpContext.Current.Response.Clear();
string sErrorUrl = ConfigStore.GetValue("Errors", "FriendlyErrorPageUrl");
HttpContext.Current.Response.Redirect(sErrorUrl);
}

private void processError(Exception exception)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "processError(): Entered.");

string sMessage = buildEmailText(exception);
trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info, "processError(): Built error string '{0}', calling SendEmail.",
sMessage);

sendEmail(sMessage);

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "processError(): Leaving.");
}

private static string buildEmailText(Exception exception)
{
var messageBuilder = new StringBuilder();
messageBuilder.AppendLine("<strong>Date: </strong>" + DateTime.Now);
messageBuilder.AppendLine("<br /><strong>Message: </strong>" + exception.Message);
messageBuilder.AppendLine("<br /><strong>Source: </strong>" + exception.Source);
messageBuilder.AppendLine("<br /><strong>Current user: </strong>" + Thread.CurrentPrincipal.Identity.Name);
messageBuilder.AppendLine("<br /><strong>Machine name: </strong>" + Environment.MachineName);
messageBuilder.AppendLine("<br /><strong>Url: </strong>" + HttpContext.Current.Request.RawUrl);
messageBuilder.AppendLine("<br /><br /><strong>Exception details: </strong>" + exception);
return messageBuilder.ToString();
}

private void sendEmail(String message)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "sendEmail(): Entered.");

try
{
var errorToAddress = ConfigStore.GetValue("Errors", "SendToEmail");
var sErrorSubject = ConfigStore.GetValue("Errors", "EmailSubject");

trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info,
String.Format("SendEmail(): About to send error e-mail to recipient list '{0}' using SPUtility.SendEmail().",
errorToAddress));

SPUtility.SendEmail(SPContext.Current.Web, false, false, errorToAddress, sErrorSubject, message);
}
catch (Exception exception)
{
var traceMessage = String.Format("Exception thrown attempting to send error e-mail using SPUtility.SendEmail(). " +
"Exception Details: {0}", exception);
trace.WriteLineIf(traceSwitch.TraceError, TraceLevel.Error, String.Format("sendEmail(): {0}", traceMessage));
}

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "sendEmail(): Leaving.");
}

public void Dispose()
{
// nothing to do here..
}
}
}

Skeleton base class for master page:

using System;
using System.Diagnostics;
using System.Web;
using System.Web.UI;
using Microsoft.SharePoint;

namespace COB.Demos.WcmSamples
{
public class BasePage : MasterPage
{
#region -- Private members --

private TraceSwitch traceSwitch = new TraceSwitch("COB.Demos.WcmSamples.BasePage",
"Trace switch for the base page class.");

private TraceHelper trace = null;

#endregion

public BasePage()
{
trace = new TraceHelper(this);
}

protected override void OnLoad(EventArgs e)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "OnLoad(): Entered.");

// TODO: add custom code here..

base.OnLoad(e);

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "OnLoad(): Leaving.");
}

/// <summary>
/// Showing an example base page property..
/// </summary>
public bool UserIsAuthenticated
{
get { return HttpContext.Current.User.Identity.IsAuthenticated; }
}
}
}

My top 5 WCM tips presentation

Had a great time presenting my WCM tips presentation over the last week or so – first to a record attendance at the UK SharePoint user group (we hit 200+ attendees for the first time!) and then to a Gold partner audience at Microsoft today. I’d like to think the user group record was due in part to the great agenda, but suspect it was really down to the draw of free beer, pizza and curry from LBi 😉 Instead of posting a simple link to the slides, I want to run through the info here because:

  1. I can attempt to convey some of the info which was in my demos here.
  2. I know how many of you will read a blog post but not follow a link to a PowerPoint deck!

First off, this presentation isn’t about discussing the standard (but critical) WCM topics which you can easily find quality information on these days. Personally I think if you’re embarking on a WCM project your starting point for information should be Andrew Connell’s book (and his blog articles if you don’t already subscribe), and reading in detail some of the key MSDN articles which have been published on the subject (I list my favorite resources at the end). In these places, you’ll find discussion on some of the sub-topics (non-exhaustive list) which I see as the bedrock of WCM:

  • Security
  • Accessibility
  • Optimization
  • Deployment

So instead of covering these in detail, my tips focus on key techniques I’ve found to be powerful in delivering successful WCM projects. They won’t be suitable for all WCM projects, but hopefully they give some food for thought and provide some extra value to the WCM info already out there.

Tip #1 – Implement HTML markup in user controls, not page layouts in SPD

Explanation:

Instead of adding your page layout HTML to the page layout in SPD, create a ‘parent’ user control for each layout which then contains the HTML and child user controls for this layout. This means you have a 1-to-1 mapping between page layouts in SPD to ‘parent’ user controls within your VS web project. These parent user controls contain the actual markup and also any child user controls used on your page.

Slide bullets:

  • Faster development experience – less SPD pain [instantaneous save rather than 3-5 second save plus occasional unwanted changes to markup]
  • Page layout markup now in primary source control [alongside your other code]
  • Much simpler deployment of updates [simply XCOPY user controls when you deploy updates to other environments]

What it looks like:

So we see that our page layouts now have very little markup (note the ASP ContentPlaceholder controls must stay at the top level, so our parent user control gets referenced in PlaceholderMain):

PageLayoutMarkup

..and then all of our real markup code is in the parent user control and child user controls which it references:

UserControlMarkup

Tip #2 – create custom IIS virtual directory pointing to your web project files

Explanation:

Most project teams I see put their user controls under 12/CONTROLTEMPLATES (often in a sub-folder) so as to follow what SharePoint does. This is mandatory in some techniques (e.g. custom field controls, delegate controls), but is not required for the type of WCM page user controls discussed in tip #1, and there are arguments for not storing those in the 12 hive. In summary, having a custom IIS virtual directory pointing to your VS project means we avoid having separate design-time and run-time locations for the project files.

Slide bullets:

  • Store user controls/page furniture files (e.g. image/XSL) here. Remove code files (e.g. .cs files) for non-dev environments
  • Faster development experience – no files to copy, no post-build events. Just save and F5!
  • Important if using tip #1 – don’t want to have to compile project [for post-build event to copy files] just for a HTML change

What it looks like:

In our case we created a virtual directory with an alias of ‘MIW’ (the name of our website) which points to our project files:

CustomIisVirtualDir

All our user control/page furniture file paths then look like ‘/MIW/PageLayoutsControls/foo.ascx’  etc.

Tip #3 – make life easier for the site authors/admins [reduce their stress and they’ll be on your side]

Explanation:

This one is a non-technical tip I wanted to throw in there – whilst we’re busy getting the front-end of the website right, I think it pays to also think about how authors/admins will use the back-end of the website (N.B. here I mean ‘business admin’ rather than IT pro). Although this is probably verging on the ‘political’ side of things, I’d advocate making their life as easy as possible – they often have a loud voice within the client organisation, and if they have bad feedback on what you’re building that’s a black mark against you.

Slide bullets:

  • Consider providing custom tools if the ‘SharePoint way’ is not simple enough (e.g. user management)
  • If you use custom lists for site data, provide a link for authors to find them (e.g. using CustomAction)
  • Remember these people are rarely SharePoint gurus!

What it looks like:

Clearly this will look different for every project, but for our client we created a custom Site Actions sub-menu with some key links:

CustomSiteActionsOptions

This sub-menu provides navigation to a couple of key lists used to power the site, but also to some custom screens we created for user management. Here, we also did some work to simplify things by wrapping up the relatively complex process of creating a user in the membership provider, adding them to various security groups across 2 sites (we had 2 ‘sister’ MOSS sites with single sign-on) and setting certain profile fields, into some simple screens which give a convenient summary of what just happened after creating a new user:

CustomCreateUser

Finally, the ‘edit profile’ screens used by business administrators were adapted from those used by end users, so that the admins became very familiar with each step of the ‘profile wizard’ and were better able to support their users.

Tip #4 – plan for unexpected errors

This is an interesting area, partly because we’re talking about that category of things which ‘should never happen’, but sometimes do. Having this conversation with a client (or non-technical management within your own organization) can be fun because the typical response is "whaddya mean the website going to go wrong?", but anyone familiar with software development principles knows there is no such thing as bug-free software. So the important thing is how do we handle these cases when they do occur.

There are several tips here, so I’ll break them down into 4.1, 4.2 and 4.3:

 Tip #4.1 – implement ‘friendly’ pages for 404s and unhandled errors

Explanation:

In brief, users of a public-facing website should never see a .Net error if something goes wrong on the website! If this ever does happen, the user is left with the feeling that your website is unreliable and can lose confidence in the organisation – think about it, do you ever remember seeing this kind of error on amazon.com/eBay.com/microsoft.com?

Slide bullets:

  • Typically use custom HTTP module to override SharePoint’s default error-handling behaviour, checking for:
    • HttpContext.Current.Server.GetLastError()
    • HttpContext.Current.Response.StatusCode = 404

What it looks like:

On the Standard Chartered site, our ‘friendly’ message looks like:

CustomErrorScreen

Tip #4.2 – implement e-mail notifications to developers for errors

Explanation:

Sorting the user experience is one thing, but what about actually fixing the source of the problem? A key element of this is being alerted whenever a user does experience a problem, rather than relying on them reporting it. When I first told the team we’d be implementing this, I was really thinking about the UAT phase, and also that critical week or two after go-live when you’ll occasionally discover some latent issue which had managed to hide itself throughout all the testing. However, what we found is that we got even more value from this in the earlier dev/testing phases. At LBi, the test team sit near the development team, so when the Outlook ‘toast’ popped up with an exception message which wasn’t ‘known’ by the team, I’d use the information in the e-mail to work out which tester triggered the error, then armed with the stack trace and other information, shout over and ask exactly what they had done to arrive at the problem. Much more efficient than waiting for a full bug report at the end of the day/week!

Slide bullets:

  • Means an error cannot happen without the team being aware
  • We built this for production, but was even more useful in dev/testing!
  • Implemented in same custom HTTP module as error page redirection

What it looks like:

ExceptionEmail

Tip #4.3 – implement proper tracing in your code

Explanation:

So being alerted about unhandled exceptions is one thing, but the stack trace and other details in the e-mail are often not enough information to actually fix the bug easily. This can be because we can’t see exactly what happened in the code (e.g. values of variables) leading up to the error. The only real way to obtain this information from production servers (or anywhere where we can’t use the debugger) is to add trace/log statements to your code, which when enabled will write out these details as the code executes. Since this has to be done manually, clearly the trade-off here is the time to implement this robustness. I would strongly recommend using a code productivity tool such as ReSharper (e.g. going beyond Visual Studio snippets) to drop in these statements quickly rather than relying on typing or copy/paste.

Slide bullets:

  • Provides ability to quickly locate bugs in your code
  • Trade off is time/effort to implement
  • Consider productivity tools such as ReSharper/CodeRush to lessen impact

What it looks like:

This is showing trace output with DebugView – notice I’ve configured DebugView to show me ‘warning’ trace statements in yellow and ‘error’ statements in red:

TracingError

Tip #5 – design for flexibility

Again, this one is broken down into two tips:

Tip #5.1 – using SharePoint lists for configuration data

Explanation:

As I said in my presentation abstract, since the only certainties in life are death, taxes and clients changing their minds, we should always aim to develop a solution which is flexible enough to accommodate some of the changes we may be asked to implement. We’re probably all used to the idea of storing site data which may change in SharePoint lists, but I like to extend this to ‘configuration’ information which dictates how the site behaves. The ‘Config Store’ framework I wrote for this can be found on Codeplex at www.codeplex.com/SPConfigStore – this provides the list, content type, caching and API to retrieve ‘config items’ in code.  So to take an example from our project, we had a switch for whether newly-created users need to change their password on first logon. Clearly this is something we need enabled in production, but can be a pain in our test environments where the client needs to create users and then get on with running specific test scripts. So, by having such a switch in a SharePoint list, we get the benefit that key configuration of the site can be changed as easily as changing a SharePoint list item, as opposed to perhaps being stored in web.config where I’d need to RDP onto the server, open a file, trigger an app pool recycle etc.

We stored 130+ configuration settings in our Config Store list, and of course, applied appropriate permissions so that unauthorized users could not access the list.

Slide bullets:

  • Use SP lists for values the client may wish to edit, but consider caching

What it looks like:

ConfigStore

Tip #5.2 – implement a custom master page class

Explanation:

Although SharePoint gets master pages from .Net 2.0, these really deal with implementing a consistent look and feel only. If you want consistent functionality or behaviour across your pages, a custom master page class can help here. To implement this, the key is to modify the class which your master page derives from in the @Master directive. We used this approach to implement code which needs to execute on every page load, and even if you don’t have this requirement from the start, I’d advocate using it so you have a convenient place to put such code if the need arises.  

Slide bullets:

  • Use for any code which should execute on every page load
  • Examples on our site:
    • Check if trial user/when trial access ends
    • Check if accepted Terms & Conditions
    • Check has supplied their initial user profile info
    • Enforce use of HTTPS
  • Can also use to expose properties for commonly checked items (e.g. username, logged in time)

What it looks like:

<%@ Master language="C#" Inherits="MyCompany.MyClient.MyProject.BasePage" %>

Conclusion/resources

So that’s it for my tips, all feedback/suggestions welcome! In terms of key resources, in addition to AC’s book here are some selected articles and so on I’d recommend looking into:

Developer lessons learnt – SharePoint WCM in the finance sector

So in my recent SharePoint WCM in the finance sector post, I talked about what we built and why I think the result is kind of interesting. What I want to do today is share some of the technical lessons learnt, and give a sense of what worked and what didn’t. As I mentioned last time, UK-based folks will hopefully be able to gain more than I can provide here when the site gets presented at the SharePoint UK user group, meaning we’ll answer any question you care to come up with, not just some of the developer stuff I want to discuss today.

Now to frame all this, it’s important to consider the type of project this was – the terms mean slightly different things to different people, but to me the emphasis was on ‘development’, as opposed to ‘implementation’ or ‘customization’. In code terms, we ended up with the following:

  • 17 Visual Studio projects in total
  • 4 Windows services
  • 5 nightly batch processes
  • 5 supplementary SQL tables (outside of the SharePoint db)

Not bad for 8 weeks work. As an aside, although the first number seems surprisingly high, in the technical washup I did with the team nobody thought this wasn’t the right way to factor the code. This is partly explained by the fact this single project is actually part of a bigger program of projects being done for the client, and also partly by the complexity of the Endeca search implementation and batch processes we needed.

What worked well (in no particular order)

  • Using a ‘development farm’ amongst the developers – this means the content database is shared, and thus no effort is required for one developer to see the lists, site columns, content types, master pages/layouts etc. created by others. This is actually the only way to do team development in SharePoint for me, but worthwhile mentioning it as I know not all SharePoint shops do things this way.
  • Proper use of tracing – this is the idea of writing log statements throughout code to easily diagnose problems once the code has been deployed to other environments (e.g. QA/UAT/production). We used the standard .Net System.Diagnostics trace framework with levels of Verbose, Info, Warning and Error – this has been familiar to me for a long time but a couple of the devs were new to it and agreed it was invaluable. In particular, we had a lot of library code and it’s often difficult to find logic bugs when you can’t directly see the result of something on a screen. For me, tracing essentially gives you the power to find certain bugs in seconds or minutes which could otherwise take hours to resolve. Although adding the tracing code can slow down coding, to mitigate this we used..
  • ReSharper at the start of the project I created several ReSharper templates to call our common code e.g. for tracing, and got all the team to download trial versions of ReSharper. This meant we could add trace statements in just a few keystrokes, meaning the ‘I didn’t have time to add trace!’ excuse couldn’t be used 🙂
  • My Config Store framework based on a SharePoint list * – we stored over 130 ‘configuration items’, from ‘True’/’False’ config switches such as ‘enforce password change for users first logon,’ to known URLs, to certain strings displayed throughout the site. We also found a couple of areas for improvement (e.g. field not big enough to store XML fragments!) which will hopefully make it into the next release.
  • Implementing logging/notifications for unhandled exceptions – I know the MS Enterprise Library a component for this, but we developed our own using a HTTP handler which sits in front of SharePoint’s SPRequest handler. This means that whenever something happens in the code which we’re not expecting, we get to find out about it immediately and can see the stack trace and other debug info in the e-mail. This was invaluable when the testers got to work, as it meant we could proactively deal with bugs before they even got reported. As soon as we noticed a mail with a new exception, we shout over to the particular test guy (identified by the user ID) "What exactly did you just do?" (which impressed them greatly!), so we nail the exact set of circumstances/data which caused the bug right there and then. 
  • My Content Deployment Wizard tool * – I also played the ‘deployment/release manager’ role on this project, so I was probably the guy who benefited from this the most but I’ve actually used it somewhere on every single project I’ve done since building it. For releases when the team had updated x page layouts, x lookup lists and x Config Store items, the tool is invaluable for picking out just the changed items and deploying them to the other environments. For Config Store items in particular it was useful as some config is different between environments (similar to web.config keys) so you don’t want to overwrite the entire list. For early releases when the team had made lots of of complex ‘schema’ updates (such as lots of intricate changes to site columns/lookups/content types), due to the time pressures I elected to take the ‘everything will definitely work this way’ route and drop the site collection on the target and import the whole thing (since no valuable data to preserve) so there are some complex deployment scenarios I still haven’t fully tested personally, but with 3 environments on top of the dev environment to deploy to the Wizard was prettes tilitiy much a lifesaver.
  • Cross-site lookup field by SharePoint Solutions this solves the problem that a lookup column can only lookup data in the current web. We use this for several key sets of data so we get to have one copy and one copy only. Damn useful.
  • LINQ to SQL we use this for CRUD operations in our supplementary SQL tables, and the guys who used it agreed they saved significant time over the standard approach of writing ADO.Net code.

* hopefully it doesn’t come across as shameless self-promotion to include these – the very reason I built them was to solve recurring problems I saw on SharePoint dev projects, and both utilities really did help us here.

Project challenges (in no particular order)

  • Team Foundation Server weirdness – for reasons we still haven’t established, we found the .csproj file for the web project (i.e. the most critical VS project!) would be checked out whenever a developer compiled the solution. With multiple checkout enabled, this means that pretty much every developer had the project file checked out all the time, regardless of whether he/she was making any project changes (e.g. adding new classes). This meant we had many more merge issues than normal – not fun.
  • VM issues – for a while we thought ReSharper was the culprit here, but a VS hotfix brought more stability. A hunch says at least some of the issues are 64-bit related (our dev environment was matched to production in this respect), since often the problem would manifest itself via Visual Studio (a 32-bit application remember). Frequent VS crashes, "attempted to read or write protected memory" messages in the event log – oh joy.
  • Failure to identify shared code soon enough – often a concern on complex development projects when the team is working at high speed. We did daily standup meetings (similar to scrum) but I suspect we may have focused too much on issues rather than what was being ‘successfully’ developed. So we lost some time to refactoring to bring things back in line, but this is why I like to think of the approach as ‘Dangerously Rapid Application Development’ (for those who remember the term ;-))
  • Issues arising from sharing IP addresses on SSL – in several of our environments, we attempted to use the technique documented by Adrian Spear in To setup SSL on multiple Sharepoint 2007 web applications using host headers under IIS 6.0. I’ve used this successfully in the past but had some problems this time round – despite working fine in our QA environment, we had problems in other places. After carefully analyzing the differences, I worked out that this technique will only work if the SSL certificate being used is a wildcard certificate or is matched on the machine name rather than the site URL. This might be obvious to other people but wasn’t to me!

Hope someone finds this useful!

SharePoint WCM in the finance sector

You might have been wondering where I’ve been for the last few weeks, so I thought I’d show rather than just tell. I rarely talk about specific ‘daytime’ projects I work on, but I guess I’m quite proud of this one and you might be interested in what we’ve done. I’ve been working with LBi (Europe’s biggest design/technology agency) recently, leading a team of 5-7 developers on a WCM (Web Content Management) project for one of the global investment banks. You can see some aspects of what we’ve done with SharePoint in this post, and I’ll follow up with some developer-to-developer info about techniques we used in the next one. Additionally, for UK-based folks I think Riaz Ahmed (my current boss and Mr SharePoint at LBi) is hoping to do a real show and tell at the UK SharePoint user group in a couple of months, where there’ll be the opportunity for Q&A and to pick up far more info/lessons learnt (both technical and non-technical) than I can provide here.

The site is targeted at a closed audience over the internet (our client’s clients), so unfortunately you can’t see it for yourself – the site provides analysts with high-value information on financial markets, primarily in the form of market reports and newsflashes, though there are other content types. The user experience we were tasked with implementing (created by LBi’s design team) means the user gets to have some pretty impressive functionality around all this (IMHO) which differentiates the service from others.

The centrepiece is the market report data, which is held for many countries and topics:

MarketReport_b

This is imported nightly from a master datasource (thus lending itself nicely to caching), and our client can also add comments to the imported data via SharePoint WCM page editing. Perhaps the slickest piece of functionality is the facility for the user to create personalized ‘smart reports’ with specific markets and topics which interest him/her. These are saved against the user’s profile for quick access:

ConfigureSmartReportSmall_b

At runtime we build the personalized page from the same cache layer used by the standard pages in the first shot.

Since financial markets change quickly, our client’s authoring teams (distributed around the world), can quickly create newsflashes via SharePoint publishing functionality for display on the site – this is controlled via SharePoint workflow. Users can search and filter by country, date, high priority flag and can also store in a ‘My newsflashes’ bucket for quick retrieval:

NewsflashesSmall_b

Users also have the option of receiving e-mail alerts when newsflashes are added to the site (immediate for high priority newsflashes, nightly batch for others), and can tailor their e-mail preferences by country…

MyAccountCountryPreferencesSmall_b

..and topic:

MyAccountMailPreferencesSmall_b

The site also has a ton of other good stuff, such as:

  • single sign-on with a sister site
  • integration with a higher-level portal which can authenticate users itself (meaning users do not log-in separately on our site)
  • WCAG ‘A’ -level accessibility – accessibility wasn’t particularly a focus of the client for this site, but (happily) is just how things are done here anyhow. We use a lot of JavaScript to enhance the user experience but it always provides equivalent functionality when disabled – I have to say the LBi front-end developers use some incredibly innovative techniques to achieve this, I’ve rarely seen anything like it (see below for some examples).
  • integration with Endeca for search (including the newsflash landing page you see above – this is powered by search)
  • custom admin functionality for user creation/management
  • many many smaller but ‘interesting’ requirements such as newsflashes having user-selected ‘related items’, attachments etc etc!

Rich, accessible controls

In addition to all the personalization, one of the things that I love about the user experience is the funky controls we have. So why have a dull dropdown like this:

StandardDropdown

…when you can have one like this:

CountryControlExpandedMid 

This allows us to present only countries the user has told us they are interested in in the dropdown (thus avoiding clutter), whilst allowing them to easily see data for other countries if they have the occasional need to. To do this, clicking the ‘More…’ link expands the dropdown to show a second section containing the other countries (or more specifically, the other countries they are permitted to see):

CountryControlExpandedFull

Similarly for multiple selections, why have this:

StandardListbox

…when you can have this:

CountryControlMultiExpandedMid 

Again, clicking the ‘More…’ link allows the user to select a country not in their ‘preferred’ countries:

CountryControlMultiExpandedFull

Personally I think this makes for a great user experience, and it’s all accessibility-compliant.

Project plan : caffeine

The development time for everything you see here (and everything you don’t) was around 7-8 weeks, though this excludes some of the front-end development (HTML, CSS, JS). We’re not quite over the finishing line yet, but are in the final stages of user acceptance testing and security testing. To say it’s been tough would be something of an understatement, but I’m incredibly proud of my guys and it’s been a great team effort – fortunately for me they’re very talented 🙂

Next time I’ll detail some of the technical decisions we took and go through some developer bits which we felt worked well when building/deploying the site.