Command-line support for Content Deployment Wizard now available

I’m pleased to announce I’ve now completed initial development on the next version of the Content Deployment Wizard – this is a beta release for the next few weeks so if you need it "just work", you should continue to use the previous version (1.1), but I’m hoping some people out there are happy to test this beta. The tool has become fairly popular as a ‘handy tool to have in the SharePoint toolbox’, and hopefully this release extends it’s usefulness significantly for some scenarios. If you’re not familiar with the tool, it provides a way to import/export site collections, webs, lists, and files or list items, either between farms or between different sites in the same farm – the Codeplex site has more details. As previously mentioned, the key new additional functionality in this release is:

  • Command-line support
  • Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use
  • An installer

Having command-line support for the Wizard means that it can now be used in an automated way. Some key scenarios I think this might be useful in are:

  • Continuous integration/automated builds – if your site relies on SharePoint content, you can now move ‘real’ data as part of a build process, copying selected content from ‘dev’ to ‘build’ or ‘test’ for example. I often see static data (perhaps from an XML file or Excel spreadsheet) used in this way in nAnt/CruiseControl/MSBuild scripts, but for frequently changing data (config values, lookup lists etc.), this doesn’t work so well as there is always a static file to maintain separately. 
  • Deployment scripts – if you have deployment scripts to ‘bootstrap’ a website on developer machines, again pulling real data from a central ‘repository site’ can help here.
  • As part of a production ‘Content Deployment strategy’ – since out-of-the-box Content Deployment is restricted to deploying a web as the smallest item, the Wizard could be used to deploy selected lists/list items/files

Obviously you might have your own ideas about where it could slot into your processes too.

How it works

  1. First, we select the content to move as we would normally using the Wizard..

    SelectExportItems

  2. ..and select the options we want to use for this export..

    SelectExportSettings 

  3. On the final screen, the new ‘Save settings..’ button should be used to save your selections to an XML file: 

    SaveSettingsButton  
    This will then give you an XML file which looks like this:

  4. <ExportSettings SiteUrl="http://cob.publish.dev" ExcludeDependencies="False" ExportMethod="ExportAll" 
                    IncludeVersions="LastMajor" IncludeSecurity="None" FileLocation="C:\Exports" 
                    BaseFileName="BlogSubwebAndPageTemplates.cmp">
      <ExportObjects>
        <DeploymentObject Id="b0fd667b-5b5e-41ba-827e-5d78b9a150ac" Title="Blog" Url="http://cob.publish.dev/Blog" Type="Web" IncludeDescendants="All" />
        <DeploymentObject Id="cfcc048e-c516-43b2-b5bf-3fb37cd561be" Title="http://cob.publish.dev/_catalogs/masterpage/COB.master" Url="_catalogs/masterpage/COB.master" Type="File" IncludeDescendants="None" />
        <DeploymentObject Id="670c1fb3-12f3-418b-b854-751ba80da917" Title="http://cob.publish.dev/_catalogs/masterpage/COBLayoutSimple.aspx" Url="_catalogs/masterpage/COBLayoutSimple.aspx" Type="File" IncludeDescendants="None" />
      </ExportObjects>
    </ExportSettings>

  5. So we now have an XML ‘Wizard deployment settings file’ which has the IDs of the objects we selected and the export options. We’ll go ahead and show how this can be used at the command-line, but note also these settings can also be loaded into the Wizard UI on future deployments to save having to make the selections again – the key is the ‘Load settings..’ button on the first page (which we didn’t show earlier):

    LoadSettingsButton 

  6. For command-line use of the Wizard a custom STSADM command is used. We pass the settings file in using the -settingsFile switch. To run the export operation we showed above, our command would look like:
    stsadm -o RunWizardExport -settingsFile "C:\DeploymentSettings\ExportBlogSubwebAndTemplates.xml" -quiet

    The -quiet parameter is optional, and suppresses some of the progress messages which are returned during the operation.

  7. For an import operation, we follow the same process – go through the Wizard and select the settings for the import operation, then click ‘Save settings..’ at the end to get the file (N.B. note the ‘Import settings’ screen has been simplified slightly from previous versions):

    SelectImportSettings

  8. The command to import looks like this:
    stsadm -o RunWizardImport -settingsFile "C:\DeploymentSettings\ImportBlogSubwebAndTemplates.xml" -quiet

    So that’s both sides of it.

Using it for real

In real use of course, you may be deploying from one SharePoint farm to another. In this case, you also need to deal with copying the .cmp file from the source environment to the target if you’re going across farms – if you have network access between farms (e.g. you’re using it internally for automated builds/CI), a simple XCOPY in your scripts is the recommended way to do this. For production Content Deployment scenarios with no network connectivity, what I’m providing here will need to be supplemented with something else which will deal with the file transport. Clearly something web service based could be the answer.

Summary

Using the Wizard at the command-line may prove extremely useful if you need to move any SharePoint content regularly in an automated way. In contrast with other ways you might approach this, the XML definition file allows you to choose any number of webs/lists/list items/files to move in one operation, which may suit your needs better than shipping items around separately.

This is very much a beta release, but as a sidenote I’m expecting the initial issues to mainly be around the installer rather than core code – hence I’m providing a ‘manual’ install procedure which will get you past any such issues (see the readme). Needless to say, all the source code is also there for you on Codeplex if you’re a developer happy to get your hands dirty. I’m hoping a couple of friendly testers will try it out and help me iron out the wrinkles – please submit any issues to the Codeplex site linked to below.

You can download the 2.0 beta release of the Wizard (and source code) from:

A better Config Store for SharePoint sites

I’ve recently made some enhancements to my Config Store framework on Codeplex which I’m now ready to share. Many of these enhancements are a result of adapting the solution to a large project where we built a platform for 80-100 internet sites – hence it’s now become a bit more ‘enterprise’. With such things I generally make a deal with my employer where I do the work (or most of it) in my spare time and then get to share the code publicly, so here we go. Before I delve into the details, let’s have a reminder of what the Config Store is all about:

Recap – the SharePoint Config Store in a nutshell

Regular readers may remember this is a solution which allows use of a SharePoint list to store configuration values used by your SharePoint application – the idea is that your webparts/server controls/page layouts etc. store any strings/data they need for configuration in here as a more flexible alternative to web.config or similar. Since our values are now in a SharePoint list, configuration can be updated across the farm through the browser to administrators who have the appropriate permissions. We can also optionally take advantage of all the other things lists give us such as auditing, item-level security, alerts and version history etc. Finally, a caching layer is used to avoid round trips when your code fetches configuration values.

On the last couple of projects where my team has used it, we’ve finished up with 100+ config items in the list – storing all sorts of things from URLs, strings and ‘application behaviour’ switches:

ConfigStore

If you need a more complete overview, see:

Enhancements in the new release

  1. Optional "hierarchical" configuration model similar to web.config

    In the first release, since all the config values are stored in one list this means your configuration is stored in one ‘nominated’ site collection. This is fine for WCM sites which may only use one site collection, but may not easily map to certain enterprise requirements. What’s new in this release is that the framework can now use a ‘hierarchical’ model, where a Config Store list exists in whatever site collections should have one, but a ‘master’ site collection is nominated which contains the ‘master’ config values. What happens is that if the config item you request is in the ‘local’ Config Store you’ll get that value, and if not you’ll get the value from the master list. This allows local overriding of the parent values if required – in practice we found 95% of the config items would be stored in the master list only, but having the facility to override the other 5% was critical to supporting some of the functionality we developed.

    Needless to say, this is the implementation which best suited our requirements. It could be it doesn’t really suit yours, but developers could consider starting with my source code and modifying, since other aspects such as the caching layer, Feature files, event handlers etc. might not need to be changed much.

    If you want to continue to just use one Config Store list (even if you consume the config values in multiple site collections), the new code will continue to work just fine for this model too.

  2. Easier to use in ASPX markup

    Similar to my recent Language Store framework, the Config Store now supports easily dropping values into ASPX markup by implementing an expression builder. So if all you want to do is set the Text property of a control, you can now do with this without cluttering up the code-behind. Or, as an alternative example, here we’re retrieving an URL from the Config Store (stored in Category ‘PageUrls’ and key ‘MyAccountPage’) to assign to a hyperlink:
    <asp:HyperLink id="hyperlink1" NavigateUrl="<%$ SPConfigStore:PageUrls|MyAccountPage %>"
    Text="My account" ImageUrl="images/pict.jpg" runat="server"/>

  3. Fixed caching bug for farm environments

    Yes, something I have to hold my hands up to here – the caching layer in the initial release didn’t adequately deal with multiple servers. The effect was that it required an app pool recycle to pick up changes to config values, rather than them taking effect immediately. Not the end of the world, but certainly inconvenient and taking away some of the benefits of using a SharePoint list for configuration. So in this release, the implementation correctly relies on a back-end resource (using a CacheDependency) to invalidate the cache across all servers in the farm. The implementation I chose was to add the items to the cache with a CacheDependency on a text file – this needs to be located on a path which all servers can access – and the event handler now updates a timestamp in the file to invalidate the item in the cache across all servers.

    I went with using a file as the dependency item as I thought it was more lightweight than using a SqlCacheDependency – I didn’t really want to impose the SQL configuration etc. to be able to use the Config Store. However, with the file cache dependency I’ve chosen, be aware that some production configurations may have internal firewalls which prevent all WFEs from accessing a shared file in this way – check before you deploy. Of course the code is there for you to modify should you wish to make changes in this area.

  4. Amendment to Feature to prevent web.config modifications being made on Feature activation

    Another thing that got in the way when I tried to implement the Config Store on our enterprise project was the Feature event receiver which adds the required appSettings keys to web.config. The idea here is that, to help simplify installation and initial setup, the required web.config entries are added with empty values so all the developer has to do is plug in the appropriate values for his/her site. Sounds great – and it is for single site collection sites. But for multiple site collections, when we come to activate the Feature in the 2nd, 3rd, nth site collection – of course the receiver runs and adds the empty entries to web.config again, despite the fact that we’d already inserted the real values on the first activation. And since the new values come later in the file, guess which ones are used?

    So in this release there’s a Feature property which determines if web.config modifications are made – it’s set to ‘False’ by default but it’s there if you prefer to change it.

  5. ‘Config value’ column is now bigger

    Previously this column was a ‘single line of text’ but it’s now a ‘note’. This means you can happily store HTML/XML fragments or other larger values, which is very useful in some scenarios. However, note that if you’re already using the Config Store on your site and want to ‘upgrade’, this schema change is significant – I discuss it in the readme.txt file.

So that’s it. You can download the updates from www.codeplex.com/SPConfigStore – hope you find it as useful as we have.

Using .Net Expression Builders to set control properties

In my last post I introduced my Language Store solution for multi-lingual SharePoint sites, and showed the two ways it can be used:

In standard .Net procedural code:

string sButtonText = LanguageStore.GetValue("Search", "SearchGoButtonText");

Declaratively in HTML:

<asp:Button runat="server" id="btnSearch" Text="<%$ SPLang:Search|SearchGoButtonText %>" />

This declarative syntax is very useful as it means the developer doesn’t have to clutter up code-behind files just to call the method to retrieve a value, then assign it to the ‘Text’ property of various controls. I’ve also retrofitted this to my Config Store solution (along with some other enhancements) and this will be available on Codeplex soon. You might notice it’s the same syntax as the SPUrl token which can be used in master pages/page layouts to get a relative path to an image or CSS file, and that’s because I’m using the same .Net technique. Since I had to do some digging to work out how this was done, I’m guessing (could be wrong here!) many other developers haven’t come across this either, so here’s how it’s done.

Implementing an expression builder class

An expression builder is essentially a class which derives from System.Web.Compilation.ExpressionBuilder and contains logic to evaluate an expression at page parse time. The ‘secret’ is that the ASP.Net page parsing engine understands that it needs to call the class’s method whenever it encounters an expression in the appropriate form. These are the things that join this mini-framework together:

  • Class derived from ExpressionBuilder which overrides the EvaluateExpression() and GetCodeExpression() methods
  • Declaration in web.config which associates your prefix (‘SPLang’ in my case) with your expression builder class
  • Optional use of ExpressionPrefix attribute on class for designer support
  • An expression in declarative HTML (as per the example above)

Taking things step-by-step, here’s what my class looks like:

[ExpressionPrefix("SPLangStore")]
public class LangStoreExpressionBuilder : ExpressionBuilder
{
private static TraceSwitch traceSwitch = new TraceSwitch("COB.SharePoint.Utilities.LanguageStore",
"Trace switch for Language Store");

private static LangStoreTraceHelper trace = new LangStoreTraceHelper("COB.SharePoint.Utilities.LangStoreExpressionBuilder");

public static object GetEvalData(string expression, Type target, string entry)
{
trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Entered with expression '{0}'.",
expression);

string[] aExpressionParts = expression.Split('|');
string sCategory = aExpressionParts[0];
string sTitle = aExpressionParts[1];

if ((aExpressionParts.Length != 2) || (string.IsNullOrEmpty(sCategory) || string.IsNullOrEmpty(sTitle)))
{
trace.WriteLineIf(traceSwitch.TraceError, TraceLevel.Error, "GetEvalData(): Unable to parse expression '{0}' into " +
"format 'Category|Title' - throwing exception.",
expression);

throw new LanguageStoreConfigurationException("Token passed to Language Store expression builder was in the wrong format - " +
"expressions should be in form Language Store Category|Item Title e.g. Search|SearchGoButtonText");
}

string sValue = LanguageStore.GetValue(sCategory, sTitle);

trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info, "GetEvalData(): Retrieved '{0}' from Language Store.",
sValue);

trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Returning '{0}'.",
sValue);

return sValue;
}

public override object EvaluateExpression(object target, BoundPropertyEntry entry,
object parsedData, ExpressionBuilderContext context)
{
return GetEvalData(entry.Expression, target.GetType(), entry.Name);
}

public override CodeExpression GetCodeExpression(BoundPropertyEntry entry,
object parsedData, ExpressionBuilderContext context)
{
Type type1 = entry.DeclaringType;
PropertyDescriptor descriptor1 = TypeDescriptor.GetProperties(type1)[entry.PropertyInfo.Name];
CodeExpression[] expressionArray1 = new CodeExpression[3];
expressionArray1[0] = new CodePrimitiveExpression(entry.Expression.Trim());
expressionArray1[1] = new CodeTypeOfExpression(type1);
expressionArray1[2] = new CodePrimitiveExpression(entry.Name);
return new CodeCastExpression(descriptor1.PropertyType, new CodeMethodInvokeExpression(new
CodeTypeReferenceExpression(base.GetType()), "GetEvalData", expressionArray1));
}

public override bool SupportsEvaluate
{
get { return true; }
}
}

If you’re wondering why two methods are required, it’s because GetCodeExpression() is used where the page has been compiled and EvaluateExpression() is used when it is purely being parsed. My code follows the MSDN pattern which supports both modes and uses a third helper method (GetEvalData()) which both call into. It’s this GetEvalData() method which does the work of parsing the passed expression and then using it to obtain the value – in my case the expression is the ‘category’ and ‘title’ of the item to fetch from the Language Store. Notice that effectively, the key line in all of that is one line in GetEvalData() which calls my existing LanguageStore.GetValue() method – so effectively my expression builder is just a wrapper for this method.

My web.config entry looks like this:

<add expressionPrefix="SPLang" type="COB.SharePoint.Utilities.LangStoreExpressionBuilder, COB.SharePoint.Utilities.LanguageStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" />

And finally here’s how the component parts of the expression get used:

ExpressionBuilderSyntax

For SharePoint solutions, assuming we’re deploying our code as a Feature/Solution we’d generally want to add the web.config entry automatically by way of the SPWebConfigModification class. You can find the code to do this in on Codeplex in the source code for my Language Store solution (in the Feature receiver).

Finally, if you’re building an expression builder and this information doesn’t get you all the way, the MSDN documentation for the ExpressionBuilder class has some additional details.

Enhancing the design-time experience with a custom expression editor

This is something I haven’t looked at yet, but looks extremely cool! If the standard expression builder stuff wasn’t convenient enough for you, you can extend things further by providing a custom ‘ExpressionEditor’ for use in Visual Studio. If I understand the possibility correctly, this can provide a better experience in the VS properties grid in two ways:

  • a custom editor sheet (e.g. a dialog to enter the ‘category’ and ‘title’ of the Language Store item – let’s say ‘Search’ and ‘SearchGoButtonText’ was entered respectively, this would ‘build’ the string in the correct delimited ‘Search|SearchGoButtonText’ form required)
  • a custom picker using the Expressions collection – this could (I think) be used to query the Language Store list and display all the items, so that selecting the item to display the translation of is as simple as a few clicks, no typing!

I’d absolutely love to implement this for the Language Store/Config Store – so I might return to this at a later date!

Conclusion

Expression builders provide a powerful, clean way to inject method calls into your markup. In most cases we’re used to seeing them return strings as in my Language Store/Config Store implementations, and note that the following implementations are also present in the .Net framework:

However, one final thing to bear in mind is that the signature of the method returns an object – so theoretically it should be possible to do a whole host of other things, where the processsing returns a more complex object which gets assigned to the control property. An example could be data-binding scenarios where your method returns something which implements IEnumerable/IList – this could then be assigned to the DataSource property of your control declaratively. You might have other possibilities in mind, but hopefully that’s food for thought 😉

Building multi-lingual SharePoint sites – introducing the Language Store

If you’re ever asked to build a multi-lingual site in SharePoint, it quickly becomes apparent that there are a few extra considerations compared to a single language site. These could include:

  • Information architecture
  • Language/culture detection
  • Deciding whether to use variations or not
  • URL strategy

..and so on. Clearly these are decisions which will have a different ‘answer’ for every multi-lingual project, and typically your client’s specific requirements will steer your approach. However one challenge which is likely to remain constant across most such projects is this one:

  • How to deal with the many small strings of text which are not part of authored page content which need to be translated and displayed in the appropriate language

This is the challenge I’m focusing on here. To illustrate, here’s an example from the BBC site where I’ve highlighted all the strings which may need to be translated but which don’t belong to a particular page:

BBCExample

..and that’s just one page – it turns out a typical site will have many of these. If you have to translate additional strings shown to authors in edit mode only, you could easily find the total number stretching into the hundreds. So we start to need a framework for storage/retrieval of these ‘page furniture’ items. If we were dealing with a shrink-wrapped product, .Net resource files could be a good choice, but this approach is probably not flexible enough for a website and won’t allow content authors/power users to enter translations. Clearly something based around a SharePoint list is called for, so enter the ‘Language Store’ – my solution to the problem which you can now download from Codeplex (link at the end).

Introducing the Language Store

The Language Store is an adaptation of my earlier Config Store solution and follows some of the same principles:-

  • values are stored in a SharePoint list
  • an API is provided to retrieve values with a single method call
  • a caching framework is used for optimum performance
  • easily deployed as a .wsp

Items in the list are categorized, and have a column for each language translation:

LanguageStoreListNarrow

Note that each translation column in the list is named with the convention ‘LANG_<culture name>‘ (N.B. you might know a ‘culture name’ as a ‘locale ID’ or similar) – so when a new language needs to be added to the site, you simply create a new column with the appropriate name and add the translations. A list of culture names can be found in the MSDN documentation for the CultureInfo class.

Retrieving values

To retrieve a value, we simply call the GetValue() method and pass the category and title of the item to retrieve:

string sButtonText = LanguageStore.GetValue("Search", "SearchGoButtonText");

Also, since many of the items we might put in the Language Store are only used in the presentation of the page, it’s often a shame to have to switch to the code-behind just to fetch these values and assign them to a control’s ‘Text’ property. So I’ve provided a tokenized method similar to SPUrl, which allows you to simply drop Language Store values into your markup like this:

<asp:Button runat="server" id="btnSearch" Text="<%$ SPLang:Search|SearchGoButtonText %>" />

I like this because it means you don’t end up cluttering your code-behind with lots of lines just for fetching values from the Language Store and assigning them to ASP.Net labels or controls. For those who don’t know how this is done I’ll write more about it in the next post as I think it’s a cool, under-used facility in .Net.

How the Language Store determines which language to retrieve

In the current implementation, the regional settings of the SPWeb are used to determine which translations are retrieved. It’s a single method in the code (a single line in fact!), so this scheme could easily be changed if you have a different requirement. We’re using the Language Store on our current project, and using the SPWeb setting makes sense for us since we’re building around 100 different sites in ~30 languages, as opposed to one site which displays in the local language (according to the user’s thread culture or similar).

Note that if the Language Store doesn’t contain a value for the requested culture, a fallback process is used similar to .Net’s globalization framework:

  1. Check preferred culture  e.g. fr-CH for French (Switzerland)
  2. Check preferred culture’s parent e.g. fr for French
  3. Check default language (determined by configuration) e.g. en for English

This is useful where some items might have a different version for say American English (EN-US) and British English (EN-GB), but other items don’t require distinction so a single value can be entered into the parent column (EN).

By the way, if you’re wondering where the SPWeb regional settings are because you’ve never needed to change them, they’re here:

RegionalSettingsLink

RegionalSettingsPage

A checkbox allows the regional settings you make on a given web cascade down to child webs, so we simply set this at the root of the site as a one time operation.

Other bits and pieces

  • All items are wrapped up in a Solution/Feature so there is no need to manually create site columns/content types/the Language Store list etc. There is also an install script for you to easily install the Solution.
  • Caching implementation is currently based around a CacheDependency on a file – this enables the cache on all servers in your farm to be invalidated when an item is updated, but does require that all WFEs can write to this location (e.g. firewalls are not in the way).
  • The Language Store can also be used where no SPContext is present e.g. a list event receiver. In this scenario, it will look for values in your SharePoint web application’s web.config file to establish the URL for the site containing the Language Store (N.B. these web.config keys get automatically added when the Language Store is installed to your site). This also means it can be used outside your SharePoint application, e.g. a console app.
  • The Language Store can be moved from it’s default location of the root web for your site (to do this, create a new list (in whatever child web you want) from the ‘Language Store list’ template (added during the install), and modify the ‘LanguageStoreWebName’/’LanguageStoreListName’ keys which were added to your web.config to point to the new location. As an alternative if you already added 100 items which you don’t want to recreate, you could use my other tool, the SharePoint Content Deployment Wizard at http://www.codeplex.com/SPDeploymentWizard to move the list.)
  • All source code and Solution/Feature files are included, so if you want to change anything, you can.
  • Installation instructions are in the readme.txt in the download.

You can download the Language Store and all source code from www.codeplex.com/SPLanguageStore. All feedback welcome!

SharePoint dev strategies – it’s not all about Features!

Something I’ve been meaning to discuss for a long time is the decision to develop SharePoint artifacts using Features or some other approach. I actually discussed this back in May 2007 in my post SharePoint deployment options : Features or Content Deployment?, but feel it’s a topic worth revisiting/expanding on as I often see teams developing with Features without fully working out what exactly they are getting out of this approach. As you can guess by the article title, I’m not sold on the idea of using Features all the time (readers who have followed this blog from the start might find this surprising given I wrote many articles on how to work with Features) and want to put forward some points to consider when working out whether you need them or not.

Let’s first consider some (selected) characteristics of Features:

  • Provide a means of deploying SharePoint artifacts such as list templates/site columns/content types to multiple environments (e.g dev, test, production)
  • Currently the only way to deploy such artifacts across multiple site collections
  • Require some extra overhead to create, even with the community tools available (in comparison with creating artifacts directly in the SharePoint UI)
  • Little/no support for certain key updates (e.g. updating a content type which has already been deployed and is in use) – updates must be done through the user interface or the API, since modifying the original Feature files to make changes is unsupported.

Given these points, one scenario I really can’t see the benefit of Features for is when the solution consists of just one site collection – which is often the case for WCM sites. Why go through the extra hassle of packaging up artifacts into Features and be faced with difficulties managing updates when the artifacts will only ever exist in one site collection anyway? Sure, they may need to be deployed between environments but we have other ways of doing that.

N.B. The same applies to site definitions – why go to the trouble of creating a custom site definition when only one site will ever be created from it?

The alternative

If you aren’t forced into using Features to deal with multiple site collections, not using them could be the ‘most valid’ choice. In my recent WCM projects, I haven’t used Features for anything which doesn’t require a Feature (e.g. a VS workflow, a CustomAction etc.) for a long time now, including the project I discussed recently in SharePoint WCM in the finance sector and Developer lessons learnt – SharePoint WCM in the finance sector. Certainly given the extremely tight timescale on that project, I actually feel we could have failed to deliver on time if we had used Features.

Instead, my approach is to create a blank site in the dev environment, and do all the list/site column/content type/master page development there using the SharePoint UI and SPD. My next step (perhaps not surprising to regular readers) is to use my Content Deployment Wizard tool to move all the SharePoint artifacts to the other environments when ready. Equally, you could choose to write your own code which does the same thing using the now well-documented Content Deployment API. You’ll need to deal with any filesystem and .Net assets separately (generally before you import the SharePoint content on the destination), but in my view we’ve at least drastically simplified the SharePoint side of things. This seems to work well for many reasons:

  • More efficient since no development time lost to building Features
  • The update problem described earlier is taken care of for you (by the underlying Content Deployment API) – as an example, add a field to a content type in dev, deploy the content which uses it and the field will be added on the import site
  • Concept of a ‘package’ is maintained, so .cmp files produced by the Wizard can be handed to a hosting company for them to import using the Wizard at their end. I hear of quite a few people doing this.
  • We can store the .cmp files in source control and use them as part of a ‘Software Development Lifecycle’ approach. My approach (and I’d guess that of others using the tool in this way) is to store the .cmp file alongside the filesytem files such as .ascx files for the current ‘release’, and import them as part of the deployment process of moving the release to the next environment.

As an aside, when I decided to write a tool which would simplify dealing with dev/QA/UAT/production environments on SharePoint projects, I was initially torn between ‘solving the content type update problem’ and something based around the Content Deployment API. One reason why I decided on the latter was because the CD API already seemed to have solved the other issue!

Now I’m certainly not saying it works perfectly every time (it doesn’t, though is much improved following SP1 and infrastructure update), but in my experience I seem to spend less time over the course of a project resolving deployment issues than I would do building/troubleshooting Features. Additionally, using Content Deployment allows deployment of, well, content – if your solution relies on pre-created publishing pages or you have a scenario such as your client creating some content in UAT which needs to be moved to production before go-live, Features won’t help you here. The Content Deployment mechanism however, is designed for just that.

Where do Solutions (.wsp) fit in all this?

So to summarize the above, my rule of thumb for projects which aren’t built around multiple site collections is don’t use Features for things which don’t absolutely require them. So where does that leave Solution packages (.wsp files) – should they be abandoned too? Well no, definitely not in my view. Solutions solve a slightly different problem set:-

  • Deploying files to SharePoint web servers such that each server in a farm is a mirror of another. Ensuring all web front-ends have the same files used by SharePoint is, of course, a key requirement for SharePoint farms – this applies to Feature files when using them, but also to assemblies, 12 hive files etc.
  • Web.config modifications e.g. the ‘SafeControls’ entry required for custom web parts/controls
  • Code Access Security config modifications e.g. those required for controls not running from the GAC
  • Some other tasks, such as deployment of web part definition files (.webpart)

Really, there’s nothing stopping you from doing all this manually if you wanted to (especially if you’re always deploying to a single server, so there are less things to keep in sync). But the point here is that Solutions genuinely do make your life easier for comparatively little effort, so the ‘cost/benefit’ ratio is perhaps different to Features for me – the key is using one of the automated build approaches such as WSP Builder. So, my recommendation would generally be to always use Solutions for assemblies, 12 hive files etc., particularly in multiple server farm environments.

Conclusion

My rules of thumb then, are:

  • Consider not using Features (and site definitions) if your site isn’t based around multiple site collections – using the Wizard or some other solution based on Content Deployment can be the alternative
  • Use Solutions if you have multiple servers/environments, unless you’re happy to have more work to do to keep them in sync
  • If you are using Features, plan an approach for dealing with updates such as content type updates

My message here possibly goes against some of the guidance you might see other folks recommend, but I’m just going on the experience I’ve had delivering projects using different approaches. As always, the key is to consider deployment approach before you actually come to do it!

P.S. Also remember, deploying using backup and restore is a bad idea 😉

Source code for SP Content Deployment Wizard now released!

Back from holiday now, and hopefully with something useful for those of you who use my Content Deployment Wizard tool to move SharePoint content around. I’ve now released the source code for the tool onto Codeplex, so if you have an interest in the Content Deployment API or just want to see how the wizard works, it’s now all there for you. This is kind of a big step for me as I saw the tool more as a "free product" (built over many late nights after Suzanne was asleep!) rather than an open source project – I figured I would release the code at some point, but intended to wait a little longer until the next release, since I’m actually halfway through changing the code over for the new functionality. However, my hand was slightly forced because the Codeplex administrators suspended the project due to me not having supplied source code – apparently this is a rule of the site which I had missed. I wasn’t happy with the tool not being available for people who wanted it, so wanted to rectify this as soon as possible. So this explains why anybody attempting to download the tool over the last 3 or 4 days was unable to – sincerely apologize for the inconvenience people!

In any case, it had been on my mind because there seems to be an increase recently in people building their own tools and functionality around the Content Deployment API – it’s certainly an area with a lot of scope for custom solutions, and folks occasionally drop me a line with a request for a help. So it seems right to stop being precious about it and release the code sooner rather than later. I also fixed an annoying bug which caused an unhandled exception if a user with insufficient SharePoint permissions used the tool.

Anyway, hope it’s useful to someone!

Content Deployment Wizard – Codeplex site homepage
Content Deployment Wizard – Release 1.1 with source code

P.S. I have a couple of things temporarily ahead of Wizard development in the queue, but as mentioned previously the focus for next version will be to add support for command-line use/scripting. This means the Wizard can be used in scheduled/automated deployments, with the benefit of the granular selection of content to deploy which standard Content Deployment does not provide.

Config Store FAQs

So last week I introduced my Config Store ‘framework’ on Codeplex, which provides a complete solution (including source code!) for storing config values in a SharePoint list, but skipped over some of the details, including the optimizations which I think make it so much better than a simple implementation of this concept. This post aims to shed a bit more light on what the Config Store is about, and hopefully provides some key information to help you work out whether it’s something which would be useful in your projects.

What kind of configuration values can I put in the Config Store?
In short, any value you wish to avoid hardcoding into your SharePoint application’s C#/VB.Net code. Values are stored in the list as strings, but if they are actually integers, booleans, DateTime values etc., you would just cast them as appropriate after retrieval. Note also that we can also store complex data as XML, and either deserialize into a class or otherwise process with standard XML methods.

How do I retrieve config values in code?
For a single value it’s as simple as:

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

To take advantage of the optimization (described later) for retrieving multiple values in a single control/page/web part/whatever, use the ConfigStore.GetMultipleValues() method – see the Config Store introduction post or Codeplex site for details.

What are the main components?
The main pieces are:

  • Infrastructure – the list, site columns, and content type
  • Event receivers – responsible for managing the config cache
  • Feature and Feature receiver – responsible for provisioning the infrastructure (Feature) and also adding web.config entries and hooking up event receivers (Feature receiver)
  • Code – the main class is the ConfigStore class, this has the GetValue() and GetMultipleValues() methods.

How do I install it?
Simply run the install script which will do the work of installing the Solution (.wsp) to your site. Once that’s complete, there are a couple of things to check got installed correctly – a checklist is included in the readme.txt in the download.

Performance is important on our site – how is the Config Store optimized?
There are several levels of optimization – taking these one at a time:

  • all values are cached in memory, so in general retrieval is lightning fast and doesn’t require any kind of database lookup.
  • even the first ever request for a particular value will be lightning fast, since the value will already be in the cache. This is possible because of the event receiver on the list – when the admin added the config value to the list, we executed some code to proactively add the item to the memory cache at this point. The caveat here of course, is that IISResets/app pool recycles will still clear down the cache (since it is the ASP.Net memory cache), so this only holds for values added since the last reset/recycle. All that said, you could certainly add code to the global.asax file to cache all values automatically after a reset/recycle if you want ultimate performance.
  • if a query is required, the retrieval code uses the current SPContext if one is available – this avoids creating an SPSite/SPWeb object unless absolutely necessary. Needless to say, any objects which are created are also disposed.
  • the query is performed using a CAML query (SPQuery) as opposed to iterating through the whole config list.
  • the GetMultipleValues() method allows the developer to do just that whilst ensuring there is only one underlying query, rather than one for each value retrieved. 

Developers familiar with caching will recognise that this alone provides a huge boost to performance.

Where can I retrieve values from in code?
It doesn’t have to be a SharePoint page, control or web part. The framework is designed to work in other scenarios even when there is no SPContext available – examples are event receivers, InfoPath managed code or even outside the SharePoint web app (e.g. console app, custom STSADM command).

I work in a large SharePoint deployment and my admin doesn’t want to deploy assemblies to the GAC or run under Full Trust. Can I still use the Config Store?
By default the Config Store Solution installs the assembly to the GAC. This provides access to the functionality from outside the SharePoint web app as described above. However, it is possible to run the assembly from your site’s bin directory, though remember you will then need to define CAS policy in your web.config code as appropriate.

I’ve installed the Config Store using the supplied .wsp and install script, how can I quickly tell it’s working?
The quickest way is to add the supplied ‘ConfigTestControl’ to one of your pages in SharePoint Designer. You’ll need the Register directive to link the assembly, and also the control declaration:

<%@ Register Tagprefix="COB" Namespace="COB.SharePoint.Utilities" Assembly="COB.SharePoint.Utilities.ConfigStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" %>
. <!-- normal page markup here -->
.
.
<COB:ConfigTestControl runat="server"></COB:ConfigTestControl>

How does the Config Store work in terms of permissions? Do I need to apply any permissions to the list to make it work?
No specific permissions need to be configured, unless you choose to lock down write access to the config list beyond the default – this is for the list to inherit the permissions of the root web of your site. This means  anyone with ‘Contribute’ permission in this web (for example) will also be able to work with items in the config list. If you do wish to restrict who can modify config values, simply add whatever security you require to the config list – the Config Store framework will still continue to work since the code retrieves values under an admin context, via use of SPSecurity.RunWithElevatedPrivileges().

Our site is an internet site, will the Config Store work with anonymous users?
Yes. Again, since values are actually retrieved under an admin context, it doesn’t matter that your anonymous users do not have direct permissions to read from the list.

Anything else?
It’s worth remembering that we can also take advantage of services offered by the list infrastructure when using the Config Store. Some ideas could be auditing ("we need to log any changes to our app’s config"), alerts ("if something changes, I want to know about it!"), version history ("let’s see what the setting was when the site was having problems") and workflow ("I want all changes to go through me for change management").

The Config Store framework can be downloaded from www.codeplex.com/SPConfigStore.

Introducing the SharePoint Config Store for developers

Today I want to introduce something I’ve been working on recently which could be of use to you if you’re a SharePoint developer. Often when developing SharePoint solutions which require coding, the developer faces a decision about what to do with values he/she doesn’t want to ‘hardcode’ into the C# or VB.Net code. Example values for a SharePoint site/application/control could be:

‘AdministratorEmail’ – ‘bob@somewhere.com’
‘SendWorkflowEmails’ – ‘true’

Generally we avoid hardcoding such values, since if the value needs to be changed we have to recompile the code, test and redeploy. So, alternatives to hardcoding which folks might consider could be:

  • store values in appSettings section of web.config
  • store values in different places, e.g. Feature properties, event handler registration data, etc.
  • store values in a custom SQL table
  • store values in a SharePoint list

Personally, although I like the facility to store complex custom config sections in web.config, I’m not a big fan of appSettings. If I need to change a value, I have to connect to the server using Remote Desktop and open and modify the file – if I’m in a server farm with multiple front-ends, I need to repeat this for each, and I’m also causing the next request to be slow because the app domain will unload to refresh the config. Going through the other options, the second isn’t great because we’d need to be reactivating Features/event receiver registrations every time (even more hassle), though the third (using SQL) is probably fine, unless I want a front-end to make modifying the values easier, in which case we’d have some work to do.

So storing config values in a SharePoint list could be nice, and is the basis for my solution. Now I’d bet lots of people are doing this already – it’s pretty quick to write some simple code to fetch the value, and this means we avoid the hardcoding problem. However, the Config Store ‘framework’ goes further than this – for one thing it’s highly-optimized so you can be sure there is no negative performance impact, but there are also some bonuses in terms of where it can used from and the ease of deployment. So what I hope to show is that the Config Store takes things quite a bit further than just a simple implementation of ‘retrieving config values from a list’.

Details (reproduced from the Codeplex site)

The list used to store config items looks like this (N.B. the items shown are my examples, you’d add your own):

ConfigStoreList.jpg
There is a special content type associated with the list, so adding a new item is easy:

ConfigItemContentType.jpg

..and the content type is defined so that all the required fields are mandatory:

AddNewConfigItem.jpg

Retrieving values

Once a value has been added to the Config Store, it can be retrieved in code as follows (you’ll also need to add a reference to the Config Store assembly and ‘using’ statement of course):

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

Note that there is also a method to retrieve multiple values with a single query. This avoids the need to perform multiple queries, so should be used for performance reasons if your control/page will retrieve many items from the Config Store – think of it as a best practise. The code is slightly more involved, but should make sense when you think it through. We create a generic List of ‘ConfigIdentifiers’ (a ConfigIdentifier specifies the category and name of the item e.g. ‘MyApplication’, ‘AdminEmail’) and pass it to the ‘GetMultipleItems()’ method:

List<ConfigIdentifier> configIds = new List<ConfigIdentifier>();

ConfigIdentifier adminEmail = new ConfigIdentifier("MyApplication", "AdminEmail");
ConfigIdentifier sendMails = new ConfigIdentifier("MyApplication", "SendWorkflowEmails");
configIds.Add(adminEmail);
configIds.Add(sendMails);
Dictionary<ConfigIdentifier, string> configItems = ConfigStore.GetMultipleValues(configIds);
string sAdminEmail = configItems[adminEmail];
string sSendMails = configItems[sendMails];

..the method returns a generic Dictionary containing the values, and we retrieve each one by passing the respective ConfigIdentifier we created earlier to the indexer.

Other notes

  • All items are wrapped up in a Solution/Feature so there is no need to manually create site columns/content types/the Config Store list etc. There is also an install script for you to easily install the Solution.
  • Config items are cached in memory, so where possible there won’t even be a database lookup!
  • The Config Store is also designed to operate where no SPContext is present e.g. a list event receiver. In this scenario, it will look for values in your SharePoint web application’s web.config file to establish the URL for the site containing the Config Store (N.B. these web.config keys get automatically added when the Config Store is installed to your site). This also means it can be used outside your SharePoint application, e.g. a console app.
  • The Config Store can be moved from it’s default location of the root web for your site. For example sites I create usually have a hidden ‘config’ web, so I put the Config Store in here, along with other items. (To do this, create a new list (in whatever child web you want) from the ‘Configuration Store list’ template (added during the install), and modify the ‘ConfigWebName’/’ConfigListName’ keys which were added to your web.config to point to the new location. As an alternative if you already added 100 items which you don’t want to recreate, you could use my other tool, the SharePoint Content Deployment Wizard at http://www.codeplex.com/SPDeploymentWizard to move the list.)
  • All source code and Solution/Feature files are included, so if you want to change anything, you can
  • Installation instructions are in the readme.txt in the download

Summary

Hopefully you might agree that the Config Store goes beyond a simple implementation of ‘storing config values in a list’. For me, the key aspects are the caching and the fact that the entire solution is ‘joined up’, so it’s easy to deploy quickly and reliably as a piece of functionality. Many organizations are probably at the stage where their SharePoint codebase is becoming mature and perhaps based on a foundation of a ‘core API’ – the Config Store could provide a useful piece in such a toolbox.

You can download the Config Store framework and all the source code from www.codeplex.com/SPConfigStore.