Fixing WinHEC – Part 2


In my last posting, I presented how to fix WinHEC from the attendee’s point of view.  This post will look at why it is critical for Microsoft to get it right.  For the last 10 years, the Windows group has been waging a war to improve driver quality.  In that time, many great tools and initiatives have aided the development of better drivers.   But there are still a lot missing items, and unfortunately, Redmond seems disconnected from some of the problems of the driver community.   Yes, a number of us try to call this out, but even the most respected professional will not carry the impact of a room with over a hundred driver developers saying “This tool or program has a problem”.


Right now, I cannot think of an effort from Redmond that does not need some improvement, but when I talk to Microsoft developers, many of them do not believe the problems are important.  The problems can range from the breaking of most corporate (and many personal) driver build environments by the incompatible changes to the setenv script, to complexity of Driver Test Manager.    Worse yet, some efforts that were initially praised by the community are now despised.   Take WPP tracing, for example: almost everyone viewed this as a fantastic tool when it came out, but with incompatible revisions and inability of most of the community to get the more advanced features to work, developers are now ripping the traces out of their drivers.


Redmond seems to be charging ahead with new programs and tools, many which appear good, but actually miss the mark terribly.  It was revealed on the Microsoft newsgroups recently that they are working on Visual Studio integration for device driver development.  While the community talks about this a lot, at last year’s WinHEC, when the question was asked (in the one feedback forum that was held), most people rated this extremely low when compared to other tools and fixes to the existing tools.


Microsoft, you need to get the input of the community.  Surveys such as the recent DTM survey are worthless, since they structure the answers into preconceived categories and do not allow the community to interact with each other and Microsoft.   The great strength of Windows is the large number, and for most part, good quality of drivers available.  Please don’t waste your time on efforts without getting the community involved early on. 


So, let’s see a WinHEC that is exciting enough to attract a large audience, and filled with feedback sessions attended by Microsoft developers and managers to hear about the problems the community needs fixed.

Fixing WinHEC

As anyone who has been to WinHEC over the years knows, its content swings between being a heavy technical conference and being a marketing conference.   This year’s WinHEC is scheduled for November 4-7, 2008, again in Los Angeles, California.  I do not expect much from this venue, since neither of the two WinHEC’s previously in LA had a strong technical presence.   Going to LA requires putting engineers on airplanes and into hotel rooms and not just for a few hours in a Seattle conference center.    


So what makes a great techie WinHEC?  


1.      Lots of technical presentations – Some of the best conferences limited the keynotes to two.  Good WinHEC’s offer repeats of talks and go until late the last day to allow people to get as much as possible.  


2.      Ask the Experts – Starting in 2001, Microsoft added Ask the Experts, where presenters were available after their talks to take questions.  Sometimes this was in a separate room, sometimes this was over lunch.  Personally, I like a separate room for an hour or so at each lunch break. 


3.      Birds of a Feather – For a number of years, there were Birds of a Feather (BOF) sessions where, for example, driver developers could comment on any topic to the WDK team.  Sometimes there was a direction provided by Microsoft; much of the feedback for WDF came in BOF’s, but mostly BOF’s were wide open.  


4.      Developer Lounge – Unfortunately, this only happened one year in Seattle.  This was a quiet location near the trade show entrance where you could find many Microsoft developers hanging out.  It was a great place to drop by and ask a question, and had the advantage of allowing people to make the conversation fairly private when needed (versus the wide open Ask the Experts).  


 Last year’s WinHEC did not do a good job: it had five keynotes, and shut down early giving techies only 12 hours of presentations, versus as many as 21 some years.  There were no Ask the Experts, or BOF’s, or a developer lounge.   Unfortunately, there is no sign this will change in 2008.  Much of the problem is that it seems that Microsoft figures that WinHEC’s don’t need to be techie when they do not have a lot of new technology to present.  


It is time for the techies to take back WinHEC.  After all, it is supposed to be an engineering conference.  It should be easy to add back the Ask the Experts, BOF’s and a developer lounge.  On the technical side, there is nothing wrong with their presenting new things, but I believe most of the community would also be interested in some deep drill-downs on existing technologies.  As a bonus, being able to give input for improvement on existing technologies could improve the quality of drivers for all of us.   


For deep drill-downs, may I suggest we have multi-hour looks at subsystems like the memory manager and the I/O manager?  How about having feedback discussions for some of the technologies that do not thrill developers, for instance DTM, WPP/ETW and the lack of support for things that cause some dev’s to hook?    


Come on Microsoft, last year’s WinHEC was poorly attended and technically boring.  The above suggestions on information to present touches on just a few items, and I am sure the developer community can add many more.  There are plenty of topics that would excite developers and make the conference the event the place to be.  But, without a change in direction, I think most of us in the driver community will stay away from “Los Angeles, the land of fruits, nuts and lousy conferences”


Improving the documentation

 If you haven’t heard, Microsoft is now updating the WDK documentation monthly and is now providing a way to download these updated docs to your computer.  This update, just for documentation, can be found at   Microsoft should be congratulated for this effort to improve the documentation in a timely manner.


Currently there are some flaws to be aware of.  First the install process does not take into account that many of us would just like to update the documentation we got with the WDK with the new docs.  In fact, the installer does not even know about the WDK installation, and provides no way to update the documentation.  I find that the installation of the WDK’s and its documentation are the least friendly parts of the product.  After several attempts to install the updates, I finally pulled the doc files out of the underlying CAB file and put them in the help directory of the WDK where they belong.


The second problem is that the documentation does not have a feedback link so you can send in bug reports.  This will be corrected in the next version, but for now the feedback alias is   Having a well documented feedback alias is significant.  Those who have been in the industry awhile may remember that when OSR hosted the documentation bug finding contest, the quality of the documentation went up significantly.


With all the new material in the documentation, there are errors.  Now that there is a mechanism for providing feedback, and with frequent updates, we have a chance to fix these problems.  Let’s roll up our sleeves and get to it.  I am going to cut this entry short so I can send some mail about the technical mistakes I know of in the doc set. 


Coding Guidelines

When I get a new client, I ask for a copy of their driver coding guidelines.   Typically I get one of several responses.

·         We trust our developers to do the right thing.

·         The last time we brought this up there was almost open revolt.

·         They hand me a corporate standard for coding applications, most of which is worthless for drivers.

Whether companies recognize it or not, they need a set of coding guidelines for drivers.  There are a lot of ways of doing things in drivers that can cause problems, and a good set of guidelines can help.  Consider some of the following areas:

·         C++ – There can be an argument for this in many cases, but if C++ is used for drivers, there needs to be limits placed upon the language for it to work in the kernel.

·         Undocumented or unusual routines – What is your company’s policy on the use of undocumented kernel interfaces?  What about routines not normally used in the kernel, but documented for user space?  How about functions from the IFS kit for a regular driver?  There are a lot decisions to make here.

·         WDM, WDF or 3rd party toolkits – What is your policy on drivers:  should they be the traditional WDM model, the new WDF models or a 3rd party model? 

·         Development Environment – What do you require for a development environment?  This should be simple; namely a specific version of the DDK/WDK, but if you don’t spec it you can get anything.

·         Windows Logo – Do you want your drivers to have a Windows Logo?  This impacts the routines, models, and methodology for developing drivers.

·         Static Verification tools – What is your policy on code passing static and dynamic verification tools?  Do you require code to be PREfast clean?  Does your code need to run through Static Driver Verifier?  Do you require annotations to aid the above tools in finding bugs?  Do you use PC-Lint and do you have your own definitions file? 

·         Testing – Does the code need to pass Driver Verifier?  For Server 2003, does your code need to pass Call Usage Verifier? Is there a code coverage tool and standards to be met?  What other tests are required?

·         Diagnostics – What is your model for tracing and debug prints?  Do you require event logging?  Do you expect data to be presented to the performance monitor?  Is there information going to Windows Management Instrumentation?

·         Coding style – This is the one that causes the biggest fights.  You should probably not be dictating the location of braces or indentation, but there are things that make code easier to review or for the next person to work on it.

The above is not a complete list, but even more needs to be considered.  If you do not have the resources in-house, consider engaging a consultant to develop the guidelines based on your current practice. 

You do not want to end up like one firm I know that hired a contractor to develop a driver.  When the driver was delivered, the company discovered the contractor had used a toolkit the firm did not own and would cost over $100,000 to buy for the developers and train them on.  The choice between slipping the product schedule and buying the tool was not a pleasant one.  With some preparation up front in specifying coding guidelines, such scenarios can be prevented and time and money saved in the end.


Tag, you’re it

I’ve been spending the last couple of days tracking down a bug in a driver I am writing. The effort reminded me of how great tags on memory allocations and frees can be. Also, the work reminded me that there are at least a couple of features Microsoft does not promote and I rarely see.

For the uninitiated, tags are a four character value that is passed as an argument in memory allocation calls. The tag gives you a way to identify what the memory was allocated for by having a different tag for each common structure allocated. Here is a common problem: many driver developers commonly assume that they have only one tag for their whole driver because so many of the common samples do this. There is nothing stopping you from having multiple tags, and in fact there is a strong reason to have them.

Right now I am developing a file system mini-filter that has twenty-one different tags it uses for allocations. All the major buffer types and context blocks each have a unique tag for their allocations and frees.

Yes, I said frees. Part of the reason to have multiple tags is that you can also put the tag on the free of the memory by using the call ExFreePoolWithTag. Unfortunately, this call has been described as worthless in Walter Oney’s Programming the Windows Driver Model and is incorrectly documented by Microsoft.

The value of ExFreePoolWithTag is when you combine it with a tag with the PROTECTED_POOL bit set. This bit requires that you free the memory with ExFreePoolWithTag, and the OS will bug check if the memory being freed does not have the matching tag. Unfortunately, PROTECTED_POOL is not documented except in include files, and is not used by any Microsoft sample. Using ExFreePoolWithTag with PROTECTED_POOL tags gives you an automatic check that you are freeing what you intended to.

You do have to be careful on memory you allocate that the system will be freeing, since Windows will not know what tag you are using, so the system frees everything without tags. For everything except the rare instance where the system frees the memory, using multiple tags with PROTECTED_POOL is worthwhile.

Document explorer versus useful data

There is a trend in the tools coming out of Microsoft that is driving me nuts and in my opinion significantly hurting productivity.  This trend is the shrinking of the amount of data that appears on the display and requiring more mouse clicks to get there.

For driver writers this trend is most obvious in Document Explorer 8 which is used to display the Vista WDK documentation.   Comparing it to HTML Help Control 5 that was used for Windows Server 2003 SP1 DDK the previous version will show you what I mean. 

On opening the DDK documentation you have two major panes, the pane on the right goes from the toolbar to the bottom of the window, giving you a large amount of information, on my display (1280×1024) about 60 lines or a full page.  When you open the WDK you see an extra pane on the right, called index results that takes away about 12 lines on my display or roughly 20% less useful data.  

Now the second assault on data occurs when you search, with the DDK I get 36 terms on the screen, with the WDK I get 27 or a 25% drop in data.  Worse yet, the search display is now a tabbed window in the data area making toggling between several elements of the search more cumbersome.

Of course index results mentioned previously also make thing more painful, since a large portion of the time you click on a kernel API in the index, instead of seeing the API in the data pane, you see some page about a use of the API, so you have to go down and click the index results to get the API.  As an example go into the Vista DDK documentation, and choose ZwCreateFile in the index, I doubt most of us are looking for TDI Kernel-Mode Client Interactions” when we look up ZwCreateFile but that is what we get.

Microsoft is doing all this at the same time that displays are going to wide screen models with 13% less vertical pixels.  All of this adds up to taking longer to get the information needed to develop or check a driver, making it more likely that the implementation will be rushed and unreliable. 

I don’t blame the WDK documentation group, the tool is a corporate standard.  Perhaps the folks who designed and approved this trend have become so used to Power Point that more that six lines of data at a time is too much for them.   Hopefully, Microsoft will wake up to the problem, at the recent WinHEC a feedback session was asked “How do you search for WDK help?” and the overwhelming answer was Google!


Crossing the Undocumented Line

As a consultant who has more than once taken on projects Microsoft has said are impossible, many people assume I often use undocumented calls in Windows in my work.  In fact, I try to avoid them if at all possible, and am extremely careful in crossing the undocumented line.

A developer should ask a set of questions when considering using an undocumented technique; these are:

  1. Is it really needed?  Before anything else, ask yourself if there is any other way to do this.  Be sure to not constrain your design when you ask this question.  For instance, requiring something be done in the kernel may force an undocumented technique where building a helper application could allow a documented approach.

  2. How undocumented is it?  There is a lot of variation here; there are plenty of undocumented calls that are never mentioned by Microsoft.  These I consider pretty undocumented even if there are examples on the web. 
There are calls that have been in the Microsoft DDK include files for years, but are not documented.  For example, a number of the ZwXXX calls have options that are not described in the documentation, but are in the includes with all the needed data structures.  

Another variant of undocumented is the call that is documented in later OS’s but works for your driver in previous versions.  In the early days of the DDK, there were calls in the samples that were not documented.  These calls are not completely safe, but they are better than the totally undocumented calls.

  1. Which OS’s require the undocumented stuff?  If you are looking at an undocumented call for an older OS, the function you need in later OS’s is not present, so you are probably safer than expecting a call to be there in the future.

  2. What is the scope of the input and usage of the undocumented stuff?  If you are using the undocumented calls in a constrained environment you are in a safer position than a having a widespread and flexible use.  It is hard to test any piece of code, but if the inputs or patterns of use vary, the testing of this area just got harder.

  3. What is the fallback plan? If you do go ahead and do this, what will you do if it breaks tomorrow?  Has your design kept the undocumented stuff limited so it can be replaced in the code?  Is there likely to be an alternative for the replacement?  What is the plan if there is no replacement?

If you decide you have to cross the line and use an undocumented call, there are some things you should be doing:

·         Let the customer know – Document the call and the answers to the above questions to the customer or your management.  Crossing the line should be a decision made by more than just the developer.

·         Plan for a long testing cycle – You have chosen to use something you cannot rely on, so your testing pain just went up.  You really need to test all the likely OS configurations the end user could have.  This means not just testing on the current version of a Windows OS, but instead the base, all service packs, and the current version , i.e., all the latest hotfixes and downloads.  For 32-bit Windows, that is sixteen full tests if you want to support Windows 2000 up to Vista!

There are times to cross the line, but be sure you have a plan and a strong justification before you enter the undocumented territory.


Bleeding edge and far from it

I am just back from WinHEC and while there I realized that many people including a number from Microsoft don’t distinguish developing for the leading edge from living there. I am known as a guy who has done a number of things that Redmond had said “Windows is not capable of doing” and technologies that Microsoft was later proud to show off once they were working. 

When it comes to my tools, I am far from the bleeding edge.  For instance, though I recently started using the latest Visual Studio, most of my work is still done with VS6.   I like the new Studio, but I trust VS6 and that version produces code that for the most part needs no libraries other than the standard system lib’s to support it.  I use Office XP and probably will not move for a long time, since I once had a terrible experience of not being able to send a customer a promised document, since the conversion back to the format the client was using did not work well.


I am not alone in this attitude; many of the developers I respect the most stay far away from the leading edge for their work environments.  In this way we are like Seymour Cray, who required that the parts for his designs were in production and testable even though the systems he was designing were years off.


The challenge here is that Microsoft does not seem to understand this.   The WDK team was proudly pointing out the features and fast updates with online MSDN, when most developers use the local documents.  One of the new features with the MSDN stuff is a wiki capability that is supposed to be monitored by using an RSS feed.   Most of the senior developers I know do not monitor this, since Microsoft’s previous generation of tools do not support RSS!  Another RSS problem is that when Microsoft “improved” its blogging site, it dropped email notification of blog postings, because RSS was available in all the brand new tools.


Another example from WinHEC was the Windows Driver Testing Framework.  This tool assumes you are using the latest Visual Studio to develop with managed languages using the full strength of COM.  I know of almost no respected driver writers who know how to do this, and even fewer who would want to. 


It has to be confusing for the WDK team because their core product is the exception that proves the rule.  Over the years, the development community has found that BETA and just released versions of the DDK are excellent so we trust using these in our day to day work.  Unfortunately, few other teams in Redmond inspire this confidence.


So, Microsoft, if you want us to keep developing on the bleeding edge, give us tools that do not force us to live there.


Never stop learning

One thing I always recommend to my customers is to keep their developers up to speed on the latest Windows driver technologies.  I’m thinking about this right now, since last week saw the release of Developing Drivers with Windows Driver Foundation and we are less than two weeks away from the start of this year’s WinHEC. 

It is scary how many managers and developers believe that they don’t need to know this new stuff.  Even if you are still required to do drivers for NT 4.0 (hopefully not), learning the new stuff can improve your code.  Just seeing where Microsoft has put its effort to improve things may suggest to you places in your code you need to review and improve.

Also, just because you can’t use the latest Windows versions does not mean you cannot use the latest tools.  I had a customer a few years ago who had been chasing a bug for around six months.   I used PREfast on the code and found the bug in the legacy driver in a couple of minutes!

Yes, you can in theory learn this stuff from the Windows Driver Kit documentation, but that is the hard way to do it.  There are a lot of great papers on Windows Driver and Hardware Central and at OSR Online .   I check them weekly to see what is new.  The challenge for papers and documentation is to know how to approach all the information.  That is where a book, WinHEC, or one of the excellent classes offered on Windows drivers can help.

This year will be my 10th WinHEC, and at every one of them I have learned something new that I now apply to my work.  More importantly, WinHEC is a chance to meet many of the experts in the field. You don’t want to abuse this, but it is extremely useful to be able to drop an email to an expert and ask a question.   

People ask on forums, “What book should I get to learn device drivers? “  The real answer is, all of them.  Looking up at my bookshelf, I see 17 books on Windows internals and drivers, including each version of some books.  This does not count all the class notes, the booklets from WinHEC, and many years of OSR’s NT Insider.  It also does not count the draft of the excellent WDF book I mentioned in the first paragraph.  I was a technical reviewer of the book and cannot wait to get my hands on the production version.

I’ve been writing drivers and system software for 35 years, and it surprises some of my clients when I mention I’ve taken five classes on Windows driver development.  These clients think if you’ve written drivers before, you only have to tweak what you know.  Nope.  Windows is an incredibly rich development environment for drivers, and there is a lot to learn.  Right now if I had the time, I would like to sign up for a couple of classes from OSR and a couple from Azius, about Windows driver development, because there are areas I know I could learn more about.

Never stop the learning process for yourself or your developers.   If you do, you are crippling your product’s capabilities and/or time to market, and your future employability as a developer.