Incremental deployment support added to CD Wizard

Back now from a short break from blogging for a holiday and some preparation for SharePoint 2010. One of the things I’ve been working on recently is the latest release of my SharePoint Content Deployment Wizard – this has just been uploaded to Codeplex. In this release (2.5 beta) I focused on adding support for a couple of things that were common feature requests, and which should hopefully make the tool more useful for users which have these requirements. As you’ll have guessed from the title of this post, probably the biggest one is incremental deployments, but there are a couple of other useful things in there too.

Support for incremental deployments

This is something that some folks have asked for for a long time. If you are doing regular updates using the Wizard for a set of content (perhaps using the command-line support added in release 2.0), it’s often preferable to only deploy the content which has actually changed. This is now possible, and is done by selecting ‘ExportChanges’ from the ‘Export method’ dropdown on the export settings screen:

ExportChanges  
Confusingly you might have noticed the option was in the dropdown in previous releases, but it now does what it should.

How it works

Incremental deployment relies on SharePoint’s change log – the queryable store which several pieces of SharePoint use to know what content changed when. A change token can be used as a “point in time reference” which is used to identify changes after certain point, and the Wizard now stores a change token each time you perform an incremental export. When you next perform an incremental export for the same scope, the Wizard retrieves the last token for this content which is then used to ensure only changed content is deployed. Some key points here are:

  • The last change token is stored as an SPPersistedObject in the respective content database
  • Since the Wizard allows selection of multiple items to export, the change token of the ‘largest’ scope object which is selected for export is used

Support for disabling compression

Another new feature is the option for turning off the file compression which happens by default in SharePoint content deployment. When exporting a large amount of content, you will have seen that a series of files are generated, each of which is no larger than 25GB (note this is actually a configurable number, though I’ve not yet seen the requirement to expose this in the Wizard). Most of the time, you might not care about this, but the situation where it does become important is where the server doesn’t have enough memory to deal with the compression process. This could be because an extremely large site collection is being exported, or because the server just doesn’t have a lot of available memory.

To disable file compression, use the new checkbox shown below:

DisableCompression

Support for exporting the root web only

Previous versions of the Wizard actually didn’t support the scenario where one wants to export only the root web of a site collection. This is because the treeview only gave one option for the root node – ‘Export entire site’. This meant that when you selected this but really only wanted the root web, what you ended up getting was the entire site collection – which unsurprisingly has confused people in the past. This has now been fixed, the context menu now has an extra level where it’s possible to select just the root web:

ExportRootWebOnly

Conclusion

The new features of this release are:

  • Support for incremental deployments
  • Support for disabling file compression
  • Support for exporting the root web only

The latest 2.5 beta of the Content Deployment Wizard can be downloaded from Content Deployment Wizard Release 2.5 beta. As soon as I feel enough people have used this version without issues I’ll remove the beta label, as I know some folks care about this even for Codeplex utilities. As always, if you run into any problems leave me a message on the Codeplex site and I’ll do my best to help.

Command-line support for Content Deployment Wizard now available

I’m pleased to announce I’ve now completed initial development on the next version of the Content Deployment Wizard – this is a beta release for the next few weeks so if you need it "just work", you should continue to use the previous version (1.1), but I’m hoping some people out there are happy to test this beta. The tool has become fairly popular as a ‘handy tool to have in the SharePoint toolbox’, and hopefully this release extends it’s usefulness significantly for some scenarios. If you’re not familiar with the tool, it provides a way to import/export site collections, webs, lists, and files or list items, either between farms or between different sites in the same farm – the Codeplex site has more details. As previously mentioned, the key new additional functionality in this release is:

  • Command-line support
  • Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use
  • An installer

Having command-line support for the Wizard means that it can now be used in an automated way. Some key scenarios I think this might be useful in are:

  • Continuous integration/automated builds – if your site relies on SharePoint content, you can now move ‘real’ data as part of a build process, copying selected content from ‘dev’ to ‘build’ or ‘test’ for example. I often see static data (perhaps from an XML file or Excel spreadsheet) used in this way in nAnt/CruiseControl/MSBuild scripts, but for frequently changing data (config values, lookup lists etc.), this doesn’t work so well as there is always a static file to maintain separately. 
  • Deployment scripts – if you have deployment scripts to ‘bootstrap’ a website on developer machines, again pulling real data from a central ‘repository site’ can help here.
  • As part of a production ‘Content Deployment strategy’ – since out-of-the-box Content Deployment is restricted to deploying a web as the smallest item, the Wizard could be used to deploy selected lists/list items/files

Obviously you might have your own ideas about where it could slot into your processes too.

How it works

  1. First, we select the content to move as we would normally using the Wizard..

    SelectExportItems

  2. ..and select the options we want to use for this export..

    SelectExportSettings 

  3. On the final screen, the new ‘Save settings..’ button should be used to save your selections to an XML file: 

    SaveSettingsButton  
    This will then give you an XML file which looks like this:

  4. <ExportSettings SiteUrl="http://cob.publish.dev" ExcludeDependencies="False" ExportMethod="ExportAll" 
                    IncludeVersions="LastMajor" IncludeSecurity="None" FileLocation="C:\Exports" 
                    BaseFileName="BlogSubwebAndPageTemplates.cmp">
      <ExportObjects>
        <DeploymentObject Id="b0fd667b-5b5e-41ba-827e-5d78b9a150ac" Title="Blog" Url="http://cob.publish.dev/Blog" Type="Web" IncludeDescendants="All" />
        <DeploymentObject Id="cfcc048e-c516-43b2-b5bf-3fb37cd561be" Title="http://cob.publish.dev/_catalogs/masterpage/COB.master" Url="_catalogs/masterpage/COB.master" Type="File" IncludeDescendants="None" />
        <DeploymentObject Id="670c1fb3-12f3-418b-b854-751ba80da917" Title="http://cob.publish.dev/_catalogs/masterpage/COBLayoutSimple.aspx" Url="_catalogs/masterpage/COBLayoutSimple.aspx" Type="File" IncludeDescendants="None" />
      </ExportObjects>
    </ExportSettings>

  5. So we now have an XML ‘Wizard deployment settings file’ which has the IDs of the objects we selected and the export options. We’ll go ahead and show how this can be used at the command-line, but note also these settings can also be loaded into the Wizard UI on future deployments to save having to make the selections again – the key is the ‘Load settings..’ button on the first page (which we didn’t show earlier):

    LoadSettingsButton 

  6. For command-line use of the Wizard a custom STSADM command is used. We pass the settings file in using the -settingsFile switch. To run the export operation we showed above, our command would look like:
    stsadm -o RunWizardExport -settingsFile "C:\DeploymentSettings\ExportBlogSubwebAndTemplates.xml" -quiet

    The -quiet parameter is optional, and suppresses some of the progress messages which are returned during the operation.

  7. For an import operation, we follow the same process – go through the Wizard and select the settings for the import operation, then click ‘Save settings..’ at the end to get the file (N.B. note the ‘Import settings’ screen has been simplified slightly from previous versions):

    SelectImportSettings

  8. The command to import looks like this:
    stsadm -o RunWizardImport -settingsFile "C:\DeploymentSettings\ImportBlogSubwebAndTemplates.xml" -quiet

    So that’s both sides of it.

Using it for real

In real use of course, you may be deploying from one SharePoint farm to another. In this case, you also need to deal with copying the .cmp file from the source environment to the target if you’re going across farms – if you have network access between farms (e.g. you’re using it internally for automated builds/CI), a simple XCOPY in your scripts is the recommended way to do this. For production Content Deployment scenarios with no network connectivity, what I’m providing here will need to be supplemented with something else which will deal with the file transport. Clearly something web service based could be the answer.

Summary

Using the Wizard at the command-line may prove extremely useful if you need to move any SharePoint content regularly in an automated way. In contrast with other ways you might approach this, the XML definition file allows you to choose any number of webs/lists/list items/files to move in one operation, which may suit your needs better than shipping items around separately.

This is very much a beta release, but as a sidenote I’m expecting the initial issues to mainly be around the installer rather than core code – hence I’m providing a ‘manual’ install procedure which will get you past any such issues (see the readme). Needless to say, all the source code is also there for you on Codeplex if you’re a developer happy to get your hands dirty. I’m hoping a couple of friendly testers will try it out and help me iron out the wrinkles – please submit any issues to the Codeplex site linked to below.

You can download the 2.0 beta release of the Wizard (and source code) from:

Update on next version of Content Deployment Wizard

Generally I only ever talk about SharePoint tools I’m working on once they’re 100% complete and ready for use, but recently I had a conversation with someone at a user group which made me think about a policy change. Regular readers will know the main tool I’m associated with is the SharePoint Content Deployment Wizard which has become fairly popular (over 7000 downloads) – occasionally I’ve mentioned that one goal was to implement a command-line version, since this opens up all sorts of deployment possibilities. However I’ve not talked about this for a while, and just recently I’ve spoken to a couple of people who assumed I dropped this/didn’t have the time to look at it, so here I am to tell you this is not the case!

For anybody that cares, the good news is I’ve actually been working on this since December interspersed with blogging, and am nearly done. The yicky refactoring work is complete, and I got chance to write the custom STSADM command on the front of it on the flight to the MVP summit last week. I need to do more testing first, but I’m hoping to release a beta to Codeplex over the next couple of weeks – if you’re interested in the idea of scripted deployment of specific sites/webs/lists/list items between sites or farms (remember MOSS Content Deployment only does sites/webs and requires HTTP(S) connectivity), I’m hoping some friendly beta testers will help me screw the last bits down. The key aspects of this release are:

  • Command-line support
  • Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use

Shortly after this release, I’m hoping to add support for incremental deployments (so only the content which has actually changed in the sites/webs/lists/you select will be deployed), but that’s not going to make into this next cut unfortunately.

Keep tuned for further updates 🙂

Other stuff

Whilst I’m at it, other things in the pipeline from me include:

Needless to say, there are plenty of other blog articles on my ‘ideas list’ too.

Sidenote – reflecting on 2 years of SharePoint blogging

Bizarrely, I’m into my 3rd year of SharePoint blogging now. I’ve no idea how this happened. Having done some interesting work with SharePoint’s Feature framework, the initial idea was to write 4 or 5 articles I had material for – as a record for myself more than anything – and be done with it. Since then, although I do write the odd ‘easy’ post (like this one), generally my articles seem to take a long time to get completed, but I know they could be better. Occasionally I get reminded of this! So there’s a long way to go for me to become a better blogger, but I’m fully hoping to still be at it in another 2 years time – and I’ll have plenty more to say when the next version of SharePoint approaches 🙂

SharePoint dev strategies – it’s not all about Features!

Something I’ve been meaning to discuss for a long time is the decision to develop SharePoint artifacts using Features or some other approach. I actually discussed this back in May 2007 in my post SharePoint deployment options : Features or Content Deployment?, but feel it’s a topic worth revisiting/expanding on as I often see teams developing with Features without fully working out what exactly they are getting out of this approach. As you can guess by the article title, I’m not sold on the idea of using Features all the time (readers who have followed this blog from the start might find this surprising given I wrote many articles on how to work with Features) and want to put forward some points to consider when working out whether you need them or not.

Let’s first consider some (selected) characteristics of Features:

  • Provide a means of deploying SharePoint artifacts such as list templates/site columns/content types to multiple environments (e.g dev, test, production)
  • Currently the only way to deploy such artifacts across multiple site collections
  • Require some extra overhead to create, even with the community tools available (in comparison with creating artifacts directly in the SharePoint UI)
  • Little/no support for certain key updates (e.g. updating a content type which has already been deployed and is in use) – updates must be done through the user interface or the API, since modifying the original Feature files to make changes is unsupported.

Given these points, one scenario I really can’t see the benefit of Features for is when the solution consists of just one site collection – which is often the case for WCM sites. Why go through the extra hassle of packaging up artifacts into Features and be faced with difficulties managing updates when the artifacts will only ever exist in one site collection anyway? Sure, they may need to be deployed between environments but we have other ways of doing that.

N.B. The same applies to site definitions – why go to the trouble of creating a custom site definition when only one site will ever be created from it?

The alternative

If you aren’t forced into using Features to deal with multiple site collections, not using them could be the ‘most valid’ choice. In my recent WCM projects, I haven’t used Features for anything which doesn’t require a Feature (e.g. a VS workflow, a CustomAction etc.) for a long time now, including the project I discussed recently in SharePoint WCM in the finance sector and Developer lessons learnt – SharePoint WCM in the finance sector. Certainly given the extremely tight timescale on that project, I actually feel we could have failed to deliver on time if we had used Features.

Instead, my approach is to create a blank site in the dev environment, and do all the list/site column/content type/master page development there using the SharePoint UI and SPD. My next step (perhaps not surprising to regular readers) is to use my Content Deployment Wizard tool to move all the SharePoint artifacts to the other environments when ready. Equally, you could choose to write your own code which does the same thing using the now well-documented Content Deployment API. You’ll need to deal with any filesystem and .Net assets separately (generally before you import the SharePoint content on the destination), but in my view we’ve at least drastically simplified the SharePoint side of things. This seems to work well for many reasons:

  • More efficient since no development time lost to building Features
  • The update problem described earlier is taken care of for you (by the underlying Content Deployment API) – as an example, add a field to a content type in dev, deploy the content which uses it and the field will be added on the import site
  • Concept of a ‘package’ is maintained, so .cmp files produced by the Wizard can be handed to a hosting company for them to import using the Wizard at their end. I hear of quite a few people doing this.
  • We can store the .cmp files in source control and use them as part of a ‘Software Development Lifecycle’ approach. My approach (and I’d guess that of others using the tool in this way) is to store the .cmp file alongside the filesytem files such as .ascx files for the current ‘release’, and import them as part of the deployment process of moving the release to the next environment.

As an aside, when I decided to write a tool which would simplify dealing with dev/QA/UAT/production environments on SharePoint projects, I was initially torn between ‘solving the content type update problem’ and something based around the Content Deployment API. One reason why I decided on the latter was because the CD API already seemed to have solved the other issue!

Now I’m certainly not saying it works perfectly every time (it doesn’t, though is much improved following SP1 and infrastructure update), but in my experience I seem to spend less time over the course of a project resolving deployment issues than I would do building/troubleshooting Features. Additionally, using Content Deployment allows deployment of, well, content – if your solution relies on pre-created publishing pages or you have a scenario such as your client creating some content in UAT which needs to be moved to production before go-live, Features won’t help you here. The Content Deployment mechanism however, is designed for just that.

Where do Solutions (.wsp) fit in all this?

So to summarize the above, my rule of thumb for projects which aren’t built around multiple site collections is don’t use Features for things which don’t absolutely require them. So where does that leave Solution packages (.wsp files) – should they be abandoned too? Well no, definitely not in my view. Solutions solve a slightly different problem set:-

  • Deploying files to SharePoint web servers such that each server in a farm is a mirror of another. Ensuring all web front-ends have the same files used by SharePoint is, of course, a key requirement for SharePoint farms – this applies to Feature files when using them, but also to assemblies, 12 hive files etc.
  • Web.config modifications e.g. the ‘SafeControls’ entry required for custom web parts/controls
  • Code Access Security config modifications e.g. those required for controls not running from the GAC
  • Some other tasks, such as deployment of web part definition files (.webpart)

Really, there’s nothing stopping you from doing all this manually if you wanted to (especially if you’re always deploying to a single server, so there are less things to keep in sync). But the point here is that Solutions genuinely do make your life easier for comparatively little effort, so the ‘cost/benefit’ ratio is perhaps different to Features for me – the key is using one of the automated build approaches such as WSP Builder. So, my recommendation would generally be to always use Solutions for assemblies, 12 hive files etc., particularly in multiple server farm environments.

Conclusion

My rules of thumb then, are:

  • Consider not using Features (and site definitions) if your site isn’t based around multiple site collections – using the Wizard or some other solution based on Content Deployment can be the alternative
  • Use Solutions if you have multiple servers/environments, unless you’re happy to have more work to do to keep them in sync
  • If you are using Features, plan an approach for dealing with updates such as content type updates

My message here possibly goes against some of the guidance you might see other folks recommend, but I’m just going on the experience I’ve had delivering projects using different approaches. As always, the key is to consider deployment approach before you actually come to do it!

P.S. Also remember, deploying using backup and restore is a bad idea 😉

Source code for SP Content Deployment Wizard now released!

Back from holiday now, and hopefully with something useful for those of you who use my Content Deployment Wizard tool to move SharePoint content around. I’ve now released the source code for the tool onto Codeplex, so if you have an interest in the Content Deployment API or just want to see how the wizard works, it’s now all there for you. This is kind of a big step for me as I saw the tool more as a "free product" (built over many late nights after Suzanne was asleep!) rather than an open source project – I figured I would release the code at some point, but intended to wait a little longer until the next release, since I’m actually halfway through changing the code over for the new functionality. However, my hand was slightly forced because the Codeplex administrators suspended the project due to me not having supplied source code – apparently this is a rule of the site which I had missed. I wasn’t happy with the tool not being available for people who wanted it, so wanted to rectify this as soon as possible. So this explains why anybody attempting to download the tool over the last 3 or 4 days was unable to – sincerely apologize for the inconvenience people!

In any case, it had been on my mind because there seems to be an increase recently in people building their own tools and functionality around the Content Deployment API – it’s certainly an area with a lot of scope for custom solutions, and folks occasionally drop me a line with a request for a help. So it seems right to stop being precious about it and release the code sooner rather than later. I also fixed an annoying bug which caused an unhandled exception if a user with insufficient SharePoint permissions used the tool.

Anyway, hope it’s useful to someone!

Content Deployment Wizard – Codeplex site homepage
Content Deployment Wizard – Release 1.1 with source code

P.S. I have a couple of things temporarily ahead of Wizard development in the queue, but as mentioned previously the focus for next version will be to add support for command-line use/scripting. This means the Wizard can be used in scheduled/automated deployments, with the benefit of the granular selection of content to deploy which standard Content Deployment does not provide.