Category Archives: 17724

Problems with Microsoft Fake Stubs and IronPython

I have been changing the mocking framework used on a project I am planning to open source. Previously it had been using Typemock to mock out items in the TFS API. This had been working well but used features of the toolset that are only available on the licensed product (not the free version). As I don’t like to publish tests people cannot run I thought it best to swap to Microsoft Fakes as there is a better chance any user will have a version of Visual Studio that provides this toolset.

Most of the changes were straightforward but I hit a problem when I tried to run a test that returned a TFS IBuildDetails object for use inside an IronPython DSL.

My working Typemock based test was as follows

[Test]
     public void Can_use_Dsl_to_get_build_details()
     {
        // arrange
         var consoleOut = Helpers.Logging.RedirectConsoleOut();

         var tfsProvider = Isolate.Fake.Instance<ITfsProvider>();
         var emailProvider = Isolate.Fake.Instance<IEmailProvider>();
         var build = Isolate.Fake.Instance<IBuildDetail>();

         var testUri = new Uri("vstfs:///Build/Build/123");
         Isolate.WhenCalled(() => build.Uri).WillReturn(testUri);
         Isolate.WhenCalled(() => build.Quality).WillReturn("Test Quality");
         Isolate.WhenCalled(() => tfsProvider.GetBuildDetails(null)).WillReturn(build);

         // act
         TFSEventsProcessor.Dsl.DslProcessor.RunScript(@"dsl\tfs\loadbuild.py", tfsProvider, emailProvider);

        // assert
         Assert.AreEqual("Build ‘vstfs:///Build/Build/123′ has the quality ‘Test Quality’" + Environment.NewLine, consoleOut.ToString());
     }

Swapping to Fakes I got

[Test]
     public void Can_use_Dsl_to_get_build_details()
     {
        // arrange
         var consoleOut = Helpers.Logging.RedirectConsoleOut();
         var testUri = new Uri("vstfs:///Build/Build/123");

         var emailProvider = new Providers.Fakes.StubIEmailProvider();
         var build = new StubIBuildDetail()
                         {
                             UriGet = () => testUri,
                             QualityGet = () => "Test Quality",
                         };
         var tfsProvider = new Providers.Fakes.StubITfsProvider()
         {
             GetBuildDetailsUri = (uri) => (IBuildDetail)build
         };

         // act
         TFSEventsProcessor.Dsl.DslProcessor.RunScript(@"dsl\tfs\loadbuild.py", tfsProvider, emailProvider);
     

        // assert
         Assert.AreEqual("Build ‘vstfs:///Build/Build/123′ has the quality ‘Test Quality’" + Environment.NewLine, consoleOut.ToString());
     }

But this gave me the error

Test Name:    Can_use_Dsl_to_get_build_details
Test FullName:    TFSEventsProcessor.Tests.Dsl.DslTfsProcessingTests.Can_use_Dsl_to_get_build_details
Test Source:    c:\Projects\tfs2012\TFS\TFSEventsProcessor\Main\Src\WorkItemEventProcessor.Tests\Dsl\DslTfsProcessingTests.cs : line 121
Test Outcome:    Failed
Test Duration:    0:00:01.619

Result Message:    System.MissingMemberException : ‘StubIBuildDetail’ object has no attribute ‘Uri’
Result StackTrace:   
at IronPython.Runtime.Binding.PythonGetMemberBinder.FastErrorGet`1.GetError(CallSite site, TSelfType target, CodeContext context)
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at Microsoft.Scripting.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0 arg0, T1 arg1)
at IronPython.Compiler.PythonScriptCode.RunWorker(CodeContext ctx)
at IronPython.Compiler.PythonScriptCode.Run(Scope scope)
at IronPython.Compiler.RuntimeScriptCode.InvokeTarget(Scope scope)
at IronPython.Compiler.RuntimeScriptCode.Run(Scope scope)
at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink errorSink)
at Microsoft.Scripting.SourceUnit.Execute(Scope scope)
at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope)
at TFSEventsProcessor.Dsl.DslProcessor.RunScript(String scriptname, Dictionary`2 args, ITfsProvider iTfsProvider, IEmailProvider iEmailProvider) in c:\Projects\tfs2012\TFS\TFSEventsProcessor\Main\Src\WorkItemEventProcessor\Dsl\DslProcessor.cs:line 78
at TFSEventsProcessor.Dsl.DslProcessor.RunScript(String scriptname, ITfsProvider iTfsProvider, IEmailProvider iEmailProvider) in c:\Projects\tfs2012\TFS\TFSEventsProcessor\Main\Src\WorkItemEventProcessor\Dsl\DslProcessor.cs:line 31
at TFSEventsProcessor.Tests.Dsl.DslTfsProcessingTests.Can_use_Dsl_to_get_build_details() in c:\Projects\tfs2012\TFS\TFSEventsProcessor\Main\Src\WorkItemEventProcessor.Tests\Dsl\DslTfsProcessingTests.cs:line 141

If I altered my test to not use my IronPython DSL but call the C# DSL Library directly the error went away. So the issue lay in the dynamic IronPython engine – not something I am going to even think of trying to fix.

So I swapped the definition of the mock IBuildDetails to use Moq (could have used the free version of Typemock or any framework) instead of a Microsoft Fake Stubs and the problem went away.

So I had

[Test]
     public void Can_use_Dsl_to_get_build_details()
     {
         // arrange
         var consoleOut = Helpers.Logging.RedirectConsoleOut();
         var testUri = new Uri("vstfs:///Build/Build/123");

         var emailProvider = new Providers.Fakes.StubIEmailProvider();
         var build = new Moq.Mock<IBuildDetail>();
         build.Setup(b => b.Uri).Returns(testUri);
         build.Setup(b => b.Quality).Returns("Test Quality");

         var tfsProvider = new Providers.Fakes.StubITfsProvider()
         {
             GetBuildDetailsUri = (uri) => build.Object
         };

        // act
         TFSEventsProcessor.Dsl.DslProcessor.RunScript(@"dsl\tfs\loadbuild.py", tfsProvider, emailProvider);


         // assert
         Assert.AreEqual("Build ‘vstfs:///Build/Build/123′ has the quality ‘Test Quality’" + Environment.NewLine, consoleOut.ToString());
     }

So I have a working solution, but it is a bit of a bit of a mess, I am using Fake Stubs and Moq in the same test. There is no good reason not to swap all the mocking of interfaces to Moq. So going forward on this project I only use Microsoft Fakes for Shims to mock out items such as  TFS WorkItem objects which have no public constructor.

Cannot run Microsoft Fakes based test if Typemock Isolator enabled

With Microsoft Fakes moving to the Premium SKU of Visual Studio in 2012.2 (CTP4 is now available) more people will be looking at using them.

I have just installed CTP4 and have seen a behaviour I don’t think I have not seen in the previous version of Visual Studio (I need to check because as well as CTP4  I have recently installed the new version of Typemock Isolator 7.3.0 that addresses issues with Windows 8 and Visual  Studio 2012).

Anyway the error you see when you run a fakes based test is ‘UnitTestIsolation instrumentation failed to initialialize, Please restart Visual Studio and rerun this test’

image

The solution is to disable Typemock Isolator (Menu Typemock > Suspend Mocking), when this is done, without a reboot, the Fakes based test run.

Does mean you can’t have a solution using both Fakes and Isolator, but why would you?

For those hard to mock moments – Microsoft Fakes or Typemock Isolator?

About a year ago I wrote a post ‘Now that VS11 has a fake library do I still need Typemock Isolator to fake out SharePoint?’. Well this discussion becomes relevant for more people as with Visual Studio 2012.2 (currently available as a CTP) the Microsoft Fakes move from the Ultimate SKU to the Premium SKU.

From my experience the Ultimate SKU is not present on too many developer’s PCs. It is most commonly found on the PCs of the team leads, software architects or test developers (managing coded UI/load testing etc. efforts). If a team was historically going to use Microsoft Fakes then they had to buy more Ultimate SKUs; as what is the point of a unit test using a mocking framework that only part of the team can run?

The Premium SKU of Visual Studio is far more common, I would go as far as to say it is the corporate standard for development. Now as this SKU contains Test Manager (since 2012 RTM) it covers most jobs most developers do. Ultimate is just needed for the specialists in the team. Adding fakes to the Premium SKU really makes sense if Microsoft want to drive adoption.

So now the question of whether to use Microsoft Fakes or Typemock Isolator (or Telerik JustMock a product I have to admit I have not used in anger) is rebalanced as there is a fair chance a development team may all be licensed for Microsoft Fakes as they have the premium SKU. The question becomes is the cost of Isolator justified by the features it offers over and above Microsoft Fakes?

This is not an uncommon form of question for any third party add-in to Visual Studio. Visual Studio offers refactoring, but I think few would argue that Resharper or RefactorPro! don’t offer more features that justify their cost.

For me the big advantage of Typemock is ease of use and consistent syntax across all usage patterns. This could be just due to familiarity, but the fact I don’t need to manually generate the fake assembly is a bonus. Also that Isolator’s fluent API is basically the same as Moq and FakeItEasy so causes less friction when coming to advanced mocking from these tools. A team can use the free basic version of Typemock Isolator until they need the advanced features when they need to license it.

Fakes is a different way of working to most other frameworks, working at a different level inside Visual Studio. A disadvantage of this is that it does not lend itself well to refactoring, you are probably going to have to regenerate the fake assemblies after any refactor, which can be slow. Also this makes refactoring a bit more risky, as you also have to touch unit tests, a manual operation.

I think at this time for me Isolator still offers advanced features and easy of use advantages that justifies the license cost. However, as with all tools this is an ever changing field, I expect to see new features and changes for all the players in the fakes market as they all aim to better address the problems cause by the poorly architecture of applications/frameworks such as SharePoint and of course our own poorly designed legacy code.