A better Config Store for SharePoint sites

I’ve recently made some enhancements to my Config Store framework on Codeplex which I’m now ready to share. Many of these enhancements are a result of adapting the solution to a large project where we built a platform for 80-100 internet sites – hence it’s now become a bit more ‘enterprise’. With such things I generally make a deal with my employer where I do the work (or most of it) in my spare time and then get to share the code publicly, so here we go. Before I delve into the details, let’s have a reminder of what the Config Store is all about:

Recap – the SharePoint Config Store in a nutshell

Regular readers may remember this is a solution which allows use of a SharePoint list to store configuration values used by your SharePoint application – the idea is that your webparts/server controls/page layouts etc. store any strings/data they need for configuration in here as a more flexible alternative to web.config or similar. Since our values are now in a SharePoint list, configuration can be updated across the farm through the browser to administrators who have the appropriate permissions. We can also optionally take advantage of all the other things lists give us such as auditing, item-level security, alerts and version history etc. Finally, a caching layer is used to avoid round trips when your code fetches configuration values.

On the last couple of projects where my team has used it, we’ve finished up with 100+ config items in the list – storing all sorts of things from URLs, strings and ‘application behaviour’ switches:

ConfigStore

If you need a more complete overview, see:

Enhancements in the new release

  1. Optional "hierarchical" configuration model similar to web.config

    In the first release, since all the config values are stored in one list this means your configuration is stored in one ‘nominated’ site collection. This is fine for WCM sites which may only use one site collection, but may not easily map to certain enterprise requirements. What’s new in this release is that the framework can now use a ‘hierarchical’ model, where a Config Store list exists in whatever site collections should have one, but a ‘master’ site collection is nominated which contains the ‘master’ config values. What happens is that if the config item you request is in the ‘local’ Config Store you’ll get that value, and if not you’ll get the value from the master list. This allows local overriding of the parent values if required – in practice we found 95% of the config items would be stored in the master list only, but having the facility to override the other 5% was critical to supporting some of the functionality we developed.

    Needless to say, this is the implementation which best suited our requirements. It could be it doesn’t really suit yours, but developers could consider starting with my source code and modifying, since other aspects such as the caching layer, Feature files, event handlers etc. might not need to be changed much.

    If you want to continue to just use one Config Store list (even if you consume the config values in multiple site collections), the new code will continue to work just fine for this model too.

  2. Easier to use in ASPX markup

    Similar to my recent Language Store framework, the Config Store now supports easily dropping values into ASPX markup by implementing an expression builder. So if all you want to do is set the Text property of a control, you can now do with this without cluttering up the code-behind. Or, as an alternative example, here we’re retrieving an URL from the Config Store (stored in Category ‘PageUrls’ and key ‘MyAccountPage’) to assign to a hyperlink:
    <asp:HyperLink id="hyperlink1" NavigateUrl="<%$ SPConfigStore:PageUrls|MyAccountPage %>"
    Text="My account" ImageUrl="images/pict.jpg" runat="server"/>

  3. Fixed caching bug for farm environments

    Yes, something I have to hold my hands up to here – the caching layer in the initial release didn’t adequately deal with multiple servers. The effect was that it required an app pool recycle to pick up changes to config values, rather than them taking effect immediately. Not the end of the world, but certainly inconvenient and taking away some of the benefits of using a SharePoint list for configuration. So in this release, the implementation correctly relies on a back-end resource (using a CacheDependency) to invalidate the cache across all servers in the farm. The implementation I chose was to add the items to the cache with a CacheDependency on a text file – this needs to be located on a path which all servers can access – and the event handler now updates a timestamp in the file to invalidate the item in the cache across all servers.

    I went with using a file as the dependency item as I thought it was more lightweight than using a SqlCacheDependency – I didn’t really want to impose the SQL configuration etc. to be able to use the Config Store. However, with the file cache dependency I’ve chosen, be aware that some production configurations may have internal firewalls which prevent all WFEs from accessing a shared file in this way – check before you deploy. Of course the code is there for you to modify should you wish to make changes in this area.

  4. Amendment to Feature to prevent web.config modifications being made on Feature activation

    Another thing that got in the way when I tried to implement the Config Store on our enterprise project was the Feature event receiver which adds the required appSettings keys to web.config. The idea here is that, to help simplify installation and initial setup, the required web.config entries are added with empty values so all the developer has to do is plug in the appropriate values for his/her site. Sounds great – and it is for single site collection sites. But for multiple site collections, when we come to activate the Feature in the 2nd, 3rd, nth site collection – of course the receiver runs and adds the empty entries to web.config again, despite the fact that we’d already inserted the real values on the first activation. And since the new values come later in the file, guess which ones are used?

    So in this release there’s a Feature property which determines if web.config modifications are made – it’s set to ‘False’ by default but it’s there if you prefer to change it.

  5. ‘Config value’ column is now bigger

    Previously this column was a ‘single line of text’ but it’s now a ‘note’. This means you can happily store HTML/XML fragments or other larger values, which is very useful in some scenarios. However, note that if you’re already using the Config Store on your site and want to ‘upgrade’, this schema change is significant – I discuss it in the readme.txt file.

So that’s it. You can download the updates from www.codeplex.com/SPConfigStore – hope you find it as useful as we have.

Code/Feature samples for common SharePoint development tasks

There are lots of tools available to help developers, but I always think the single most useful thing is actually seeing a good example of the thing I’m trying to develop. This is why SharePoint blogs are so popular right? As well as checking my favorite blogs or past articles, I’m always going back to previous work – perhaps to see what values I used in a certain SharePoint file or to grab a chunk of code I wrote which I can amend to fit my current requirement. So today I want to point you in the direction of some useful samples which could be good reference material if you’re a SharePoint developer – in actual fact I’m not releasing anything new here, but highlighting that my recent ‘Config Store’ solution provides a great "one-stop shop" of downloadable examples of several common SharePoint development tasks. Examples of the following dev tasks can all be found in the solution:

  • Deploy site columns, content types as Feature
  • Deploy a list (with default list items) as a Feature
  • Using Feature receivers
  • Using SPWebConfigModification to make web.config changes in code/as part of a Feature
  • Using SPQuery to efficiently find items in a list
  • Build a Solution package (.wsp) without WSP Builder
  • Write a Solution deployment script to retract Solution, deactivate Feature, redeploy Solution, activate Feature etc.
  • Using event receivers
  • As a .Net bonus, how to use the HttpRuntime class to work with ASP.Net caching outside of a web context

The entire set of code/Feature files can be downloaded from Codeplex, making it a great reference project containing samples of all the tasks listed above. If this is useful to you, the list below details the file to open to find the code sample, and some notes which might help you to get to grips with the task:

Deploy site columns, content types as Feature:

<table class="MsoNormalTable" style="margin-left: -0.75pt; width: 298pt; border-top-style: none; border-right-style: none;
border-left-style: none; border-collapse: collapse; border-bottom-style: none” cellspacing=”0″ cellpadding=”0″ width=”650″ border=”1″>

File

Notes

ConfigStoreElements.xml          

This is accomplished by use of the Field/ContentType Feature elements. Note these should be before any ListTemplate/ListInstance items in the same elements file – if using both sets of elements in different files, ensure the field/content type file is listed first in manifest.xml.

Deploy a list (with default list items) as a Feature:

File

Notes

ConfigStoreElements.xml          

This is accomplished by use of the ListTemplate/ListInstance Feature elements.

Feature receivers:

File

Notes

ConfigStoreFeatureReceiver.cs

In my case, the Feature receiver makes modifications to web.config and programmatically adds list event receivers to a list.

web.config modifications

File

Notes

ConfigStoreFeatureReceiver.cs

This is accomplished by use of the SPWebConfigModification class. I add three entries to the web.config to support my solution.

 

Using SPQuery to find items in a list:

File

Notes

ConfigStore.cs                                 

I have two methods in the ConfigStore class which perform queries in this way (also known as a CAML query).

 

Build a Solution package (.wsp) without WSPBuilder:

File

Notes

makecab.ddf/                                  
manifest.xml                          

Whilst WSPBuilder offers a great automated way to build Solutions, occasionally there is a need to step outside the tool to build the package ‘manually’. In my case, I had a problem where WSPBuilder was adding elements in the ‘wrong’ order to the generated manifest.xml file, and this was causing the Solution deployment to fail.

 

Solution deployment script to retract Solution, deactivate Feature etc:

File

Notes

COB.SharePoint.Utilities.ConfigStore_Install.bat      

This is the script I’ve often used to deploy my Solutions. It has error checking at each stage so I can easily tell at what point the install fails for simpler problem diagnosis.

 

Using event receivers:

File

Notes

ConfigStoreListEventReceiver.cs

In the Config Store solution I use a list event receiver to add items to the cache when the list contents change. This code shows using the ItemAdded and ItemUpdated events on the list.

 

Using the HttpRuntime class to work with ASP.Net caching outside of ASP.Net:

File

Notes

ConfigStoreListEventReceiver.cs

Something to remember with list event receivers is that we actually don’t have access to HttpContext.Current or SPContext.Current – both are null. If it wasn’t for the HttpRuntime class, this would be a problem for my Config Store code since I wanted to add/remove items from the cache in the list event receiver (i.e. manage the cache as soon as list items are changed). Fortunately, HttpRuntime allows us access to the ASP.Net framework from any code location (e.g. console app).

 

The link to download all this stuff is at http://www.codeplex.com/SPConfigStore. Hope you find the samples useful, feel free to leave a comment if anything doesn’t make sense.

Config Store FAQs

So last week I introduced my Config Store ‘framework’ on Codeplex, which provides a complete solution (including source code!) for storing config values in a SharePoint list, but skipped over some of the details, including the optimizations which I think make it so much better than a simple implementation of this concept. This post aims to shed a bit more light on what the Config Store is about, and hopefully provides some key information to help you work out whether it’s something which would be useful in your projects.

What kind of configuration values can I put in the Config Store?
In short, any value you wish to avoid hardcoding into your SharePoint application’s C#/VB.Net code. Values are stored in the list as strings, but if they are actually integers, booleans, DateTime values etc., you would just cast them as appropriate after retrieval. Note also that we can also store complex data as XML, and either deserialize into a class or otherwise process with standard XML methods.

How do I retrieve config values in code?
For a single value it’s as simple as:

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

To take advantage of the optimization (described later) for retrieving multiple values in a single control/page/web part/whatever, use the ConfigStore.GetMultipleValues() method – see the Config Store introduction post or Codeplex site for details.

What are the main components?
The main pieces are:

  • Infrastructure – the list, site columns, and content type
  • Event receivers – responsible for managing the config cache
  • Feature and Feature receiver – responsible for provisioning the infrastructure (Feature) and also adding web.config entries and hooking up event receivers (Feature receiver)
  • Code – the main class is the ConfigStore class, this has the GetValue() and GetMultipleValues() methods.

How do I install it?
Simply run the install script which will do the work of installing the Solution (.wsp) to your site. Once that’s complete, there are a couple of things to check got installed correctly – a checklist is included in the readme.txt in the download.

Performance is important on our site – how is the Config Store optimized?
There are several levels of optimization – taking these one at a time:

  • all values are cached in memory, so in general retrieval is lightning fast and doesn’t require any kind of database lookup.
  • even the first ever request for a particular value will be lightning fast, since the value will already be in the cache. This is possible because of the event receiver on the list – when the admin added the config value to the list, we executed some code to proactively add the item to the memory cache at this point. The caveat here of course, is that IISResets/app pool recycles will still clear down the cache (since it is the ASP.Net memory cache), so this only holds for values added since the last reset/recycle. All that said, you could certainly add code to the global.asax file to cache all values automatically after a reset/recycle if you want ultimate performance.
  • if a query is required, the retrieval code uses the current SPContext if one is available – this avoids creating an SPSite/SPWeb object unless absolutely necessary. Needless to say, any objects which are created are also disposed.
  • the query is performed using a CAML query (SPQuery) as opposed to iterating through the whole config list.
  • the GetMultipleValues() method allows the developer to do just that whilst ensuring there is only one underlying query, rather than one for each value retrieved. 

Developers familiar with caching will recognise that this alone provides a huge boost to performance.

Where can I retrieve values from in code?
It doesn’t have to be a SharePoint page, control or web part. The framework is designed to work in other scenarios even when there is no SPContext available – examples are event receivers, InfoPath managed code or even outside the SharePoint web app (e.g. console app, custom STSADM command).

I work in a large SharePoint deployment and my admin doesn’t want to deploy assemblies to the GAC or run under Full Trust. Can I still use the Config Store?
By default the Config Store Solution installs the assembly to the GAC. This provides access to the functionality from outside the SharePoint web app as described above. However, it is possible to run the assembly from your site’s bin directory, though remember you will then need to define CAS policy in your web.config code as appropriate.

I’ve installed the Config Store using the supplied .wsp and install script, how can I quickly tell it’s working?
The quickest way is to add the supplied ‘ConfigTestControl’ to one of your pages in SharePoint Designer. You’ll need the Register directive to link the assembly, and also the control declaration:

<%@ Register Tagprefix="COB" Namespace="COB.SharePoint.Utilities" Assembly="COB.SharePoint.Utilities.ConfigStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" %>
. <!-- normal page markup here -->
.
.
<COB:ConfigTestControl runat="server"></COB:ConfigTestControl>

How does the Config Store work in terms of permissions? Do I need to apply any permissions to the list to make it work?
No specific permissions need to be configured, unless you choose to lock down write access to the config list beyond the default – this is for the list to inherit the permissions of the root web of your site. This means  anyone with ‘Contribute’ permission in this web (for example) will also be able to work with items in the config list. If you do wish to restrict who can modify config values, simply add whatever security you require to the config list – the Config Store framework will still continue to work since the code retrieves values under an admin context, via use of SPSecurity.RunWithElevatedPrivileges().

Our site is an internet site, will the Config Store work with anonymous users?
Yes. Again, since values are actually retrieved under an admin context, it doesn’t matter that your anonymous users do not have direct permissions to read from the list.

Anything else?
It’s worth remembering that we can also take advantage of services offered by the list infrastructure when using the Config Store. Some ideas could be auditing ("we need to log any changes to our app’s config"), alerts ("if something changes, I want to know about it!"), version history ("let’s see what the setting was when the site was having problems") and workflow ("I want all changes to go through me for change management").

The Config Store framework can be downloaded from www.codeplex.com/SPConfigStore.

Introducing the SharePoint Config Store for developers

Today I want to introduce something I’ve been working on recently which could be of use to you if you’re a SharePoint developer. Often when developing SharePoint solutions which require coding, the developer faces a decision about what to do with values he/she doesn’t want to ‘hardcode’ into the C# or VB.Net code. Example values for a SharePoint site/application/control could be:

‘AdministratorEmail’ – ‘bob@somewhere.com’
‘SendWorkflowEmails’ – ‘true’

Generally we avoid hardcoding such values, since if the value needs to be changed we have to recompile the code, test and redeploy. So, alternatives to hardcoding which folks might consider could be:

  • store values in appSettings section of web.config
  • store values in different places, e.g. Feature properties, event handler registration data, etc.
  • store values in a custom SQL table
  • store values in a SharePoint list

Personally, although I like the facility to store complex custom config sections in web.config, I’m not a big fan of appSettings. If I need to change a value, I have to connect to the server using Remote Desktop and open and modify the file – if I’m in a server farm with multiple front-ends, I need to repeat this for each, and I’m also causing the next request to be slow because the app domain will unload to refresh the config. Going through the other options, the second isn’t great because we’d need to be reactivating Features/event receiver registrations every time (even more hassle), though the third (using SQL) is probably fine, unless I want a front-end to make modifying the values easier, in which case we’d have some work to do.

So storing config values in a SharePoint list could be nice, and is the basis for my solution. Now I’d bet lots of people are doing this already – it’s pretty quick to write some simple code to fetch the value, and this means we avoid the hardcoding problem. However, the Config Store ‘framework’ goes further than this – for one thing it’s highly-optimized so you can be sure there is no negative performance impact, but there are also some bonuses in terms of where it can used from and the ease of deployment. So what I hope to show is that the Config Store takes things quite a bit further than just a simple implementation of ‘retrieving config values from a list’.

Details (reproduced from the Codeplex site)

The list used to store config items looks like this (N.B. the items shown are my examples, you’d add your own):

ConfigStoreList.jpg
There is a special content type associated with the list, so adding a new item is easy:

ConfigItemContentType.jpg

..and the content type is defined so that all the required fields are mandatory:

AddNewConfigItem.jpg

Retrieving values

Once a value has been added to the Config Store, it can be retrieved in code as follows (you’ll also need to add a reference to the Config Store assembly and ‘using’ statement of course):

string sAdminEmail = ConfigStore.GetValue("MyApplication", "AdminEmail");

Note that there is also a method to retrieve multiple values with a single query. This avoids the need to perform multiple queries, so should be used for performance reasons if your control/page will retrieve many items from the Config Store – think of it as a best practise. The code is slightly more involved, but should make sense when you think it through. We create a generic List of ‘ConfigIdentifiers’ (a ConfigIdentifier specifies the category and name of the item e.g. ‘MyApplication’, ‘AdminEmail’) and pass it to the ‘GetMultipleItems()’ method:

List<ConfigIdentifier> configIds = new List<ConfigIdentifier>();

ConfigIdentifier adminEmail = new ConfigIdentifier("MyApplication", "AdminEmail");
ConfigIdentifier sendMails = new ConfigIdentifier("MyApplication", "SendWorkflowEmails");
configIds.Add(adminEmail);
configIds.Add(sendMails);
Dictionary<ConfigIdentifier, string> configItems = ConfigStore.GetMultipleValues(configIds);
string sAdminEmail = configItems[adminEmail];
string sSendMails = configItems[sendMails];

..the method returns a generic Dictionary containing the values, and we retrieve each one by passing the respective ConfigIdentifier we created earlier to the indexer.

Other notes

  • All items are wrapped up in a Solution/Feature so there is no need to manually create site columns/content types/the Config Store list etc. There is also an install script for you to easily install the Solution.
  • Config items are cached in memory, so where possible there won’t even be a database lookup!
  • The Config Store is also designed to operate where no SPContext is present e.g. a list event receiver. In this scenario, it will look for values in your SharePoint web application’s web.config file to establish the URL for the site containing the Config Store (N.B. these web.config keys get automatically added when the Config Store is installed to your site). This also means it can be used outside your SharePoint application, e.g. a console app.
  • The Config Store can be moved from it’s default location of the root web for your site. For example sites I create usually have a hidden ‘config’ web, so I put the Config Store in here, along with other items. (To do this, create a new list (in whatever child web you want) from the ‘Configuration Store list’ template (added during the install), and modify the ‘ConfigWebName’/’ConfigListName’ keys which were added to your web.config to point to the new location. As an alternative if you already added 100 items which you don’t want to recreate, you could use my other tool, the SharePoint Content Deployment Wizard at http://www.codeplex.com/SPDeploymentWizard to move the list.)
  • All source code and Solution/Feature files are included, so if you want to change anything, you can
  • Installation instructions are in the readme.txt in the download

Summary

Hopefully you might agree that the Config Store goes beyond a simple implementation of ‘storing config values in a list’. For me, the key aspects are the caching and the fact that the entire solution is ‘joined up’, so it’s easy to deploy quickly and reliably as a piece of functionality. Many organizations are probably at the stage where their SharePoint codebase is becoming mature and perhaps based on a foundation of a ‘core API’ – the Config Store could provide a useful piece in such a toolbox.

You can download the Config Store framework and all the source code from www.codeplex.com/SPConfigStore.