The Case of the Context-Menu Shell Extension That Doesn’t Get Invoked by Explorer

A developer was working on a context-menu shell extension.

He wrote all the skeleton infrastructure code, including some code to display the selected files when the shell extension gets invoked by Windows Explorer. It’s kind of the “Hello World” for context-menu shell extensions.

He happily builds his code within Visual Studio, copies the shell extension DLL to a virtual machine, registers the extension in the VM, right-clicks on a bunch of files… but nothing happens. His extension’s menu items just don’t show up in the Explorer context-menu.

What’s going wrong, he thought?

He starts thinking to all sorts of hypotheses and troubleshooting strategies.

He once again manually registers the COM DLL that implements the shell extension from the Command Prompt:

regsvr32 CoolContextMenuExtension.dll

A message box pops up, informing him that the registration has succeeded.

He tries right-clicking some files, but, once again, his extension’s menu items don’t show up.

He then checks that the extension was properly registered under the list of approved shell extensions (this was part of his shell extension infrastructure C++ code):

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved

He finds the GUID of the shell extension in there, as expected.

He then checks that the extension was registered under the proper subkey in HKEY_CLASSES_ROOT. Found it there, too!

He starts wondering… what may be going so wrong?

He checks the C++ infrastructure code generated by the ATL Visual Studio wizard, including his own additions and modifications. The implementations of DllRegisterServer and DllUnregisterServer seem correct.

He also checks his own C++ code in the C++ class that implements the shell extension’s COM object. He has correctly implemented IShellExtInit::Initialize, and the three methods of IContextMenu: GetCommandString, InvokeCommand, and QueryContextMenu.

He takes a look at the class inheritance list: both IShellExtInit and IContextMenu are listed among the base classes.

The IShellExtInit and IContextMenu COM interfaces are correctly listed among the base classes.
The IShellExtInit and IContextMenu COM interfaces are correctly listed among the base classes.

He feels desperate. What’s going wrong?? He wrote several shell extensions in C++ in the past. Maybe there’s some bug in the current version of Visual Studio 2019 he is using?

Why wasn’t his context-menu shell extension getting called by Explorer?

 

I took a look at that code.

At some point, I was enlightened.

I gave a look at the COM interface map in the header file of the C++ shell extension class.

There’s a bug in the shell extension C++ object’s COM interface map.
There’s a bug in the shell extension C++ object’s COM interface map.

Wow! Can you spot the error?

Something is missing in there!

In fact, in addition to deriving the C++ shell extension class from IContextMenu, you also have to add an entry for IContextMenu in the COM map!

I quickly added the missing line:

    COM_INTERFACE_ENTRY(IContextMenu)

The COM interface map, with both the IShellExtInit and IContextMenu entries.
The COM interface map, with both the IShellExtInit and IContextMenu entries.

I rebuilt the extension, registered it, and, with great pleasure and satisfaction, the new custom items in the Explorer’s context-menu showed up! And, after selecting a bunch of files to test the extension, the message box invoked by the shell extension C++ code was correctly shown.

All right! 😊

So, what was going wrong under the hood?

Basically, since the context-menu shell extension was correctly registered under the proper key in the Windows Registry, I think that Windows Explorer was actually trying to invoke the shell extension’s methods.

In particular, I think Explorer called QueryInterface on the shell extension’s COM object, to get a pointer to IContextMenu. But, since IContextMenu was missing from the COM map, the QueryInterface code implemented by the ATL framework was unable to return the interface pointer back to Explorer. As a consequence of that, Explorer didn’t recognize the extension as a valid context-menu extension (despite the extension being correctly registered in the Windows Registry), and didn’t even call the IShellExtInit::Initialize method (that was actually available, as IShellExtInit had its own entry correctly listed in the COM map from the beginning).

The bug is in the details!

So, the moral of the story is to always check the COM map in addition to the base class list in your shell extension’s C++ class header file. The COM interfaces need to both be in the base class list and have their own entries in the COM map.

And, in addition, to me this experience has suggested that it would have been great if Visual Studio had issued at least a warning for having the IContextMenu COM interface listed among the base classes, but missing from the COM map! This would be an excellent time-and-bug-saving addition, Visual Studio IDE/Visual C++/Visual Assist X teams!

 

Where Should I Register My Context-Menu Shell Extension to Operate on All Files and Folders?

A developer was working on a context-menu shell extension. He registered the extension under:

HKEY_CLASSES_ROOT\*\shellex\ContextMenuHandlers

The developer noted that his context-menu extension was called for all files. But he also wanted his extension to be called for all directories.

If you want your context-menu shell extension to be invoked on all files and all file folders, the correct key to use for registration is HKCR\AllFileSystemObjects, not HKCR\*.

For more information, MSDN has a list of Predefined Shell Objects.

 

Building In-Process Shell Extensions? C++ Is The Right Tool For The Job

In today’s world, there are so many programming languages to choose from when  developing software projects. So, oftentimes, this question arises: “With so many simpler and higher-level programming languages, why should I choose C++ to do X?”

Well, programming languages (and frameworks) are just tools. And my usual guidance is: use the right tool for the job. So, if for your project a simpler/more productive/higher-level programming language is well suited, then just go for it!

But, there are cases in which C++ is just The Best Tool For The Job.

C++ great for perf
C++ great for perf

One of these contexts is the development of in-process shell extensions for Windows. There is a whole MSDN web page discussing “Guidance for Implementing In-Process Extensions”.

A key point is that the .NET Framework/CLR is a high-impact runtime, with associated performance issues for in-proc extensions:

“Performance issues can arise with runtimes that impose a significant performance penalty when they are loaded into a process. The performance penalty can be in the form of memory usage, CPU usage, elapsed time, or even address space consumption. The CLR, JavaScript/ECMAScript, and Java are known to be high-impact runtimes. Since in-process extensions can be loaded into many processes, and are often done so at performance-sensitive moments (such as when preparing a menu to be displayed the user), high-impact runtimes can negatively impact overall responsiveness. […]”

The aforementioned MSDN documentation continues with a brief discussion of issues associated to high resource consumption as well.

Then, another paragraph about “Issues Specific to the .NET Framework” briefly touches on COM interop related problems. It’s important to keep in mind that the in-proc shell extension model was designed around native code, and there is a kind of “impedance mismatch” between that and the managed .NET world.

Note that the use of .NET is considered acceptable for other types of extensions, like out-of-process extensions.

A common type of shell extensions are the context-menu extensions.

Example of customized context-menu via shell extensions
Example of customized context-menu via shell extensions

If you are curious about the development  of this type of extensions using C++, you may find my Pluralsight course on the subject interesting.

(A brief description of the course content is offered in this blog post.)

 

Pluralsight Course: Building Context-Menu Shell Extensions in C++

I “wrote” a couple of video courses published on Pluralsight (a third one is work in progress, stay tuned!).

My first Pluralsight course was “Building Context-Menu Shell Extensions in C++”. It’s a slightly less than three-hour course, in which I teach you how to build context-menu shell extensions in C++, using Visual Studio.

The course starts with a brief introduction to COM: just to those COM concepts required for the remaining course modules.

Then, in the following module, I introduce the use of IExecuteCommand to build a simple context-menu shell extension. In this module, I use just “raw” C++, without any frameworks (like ATL). This approach gives the opportunity to show how some things work “under the hood”.

In the next module, I revisit the IExecuteCommand technique, but this time with the help of ATL. ATL is a very useful productive framework for C++/COM programmers: comparing the work done in the previous module with the ATL-based approach presented in this module will make you appreciate the productivity improvements brought by ATL (and Visual Studio ATL Wizards).

In the final module I introduce you to an IContextMenu-based technique for building context-menu shell extensions. There are pros and cons in using IExecuteCommand vs. IContextMenu. For example, while IContextMenu is available in Windows XP, IExecuteCommand is a Win7+ COM interface. So, if you need to develop a context-menu shell extension that supports XP, you have to use IContextMenu.

Moreover, while IExecuteCommand simplifies some common operations, more advanced techniques like building fancy UIs in the context-menu (for example, implementing owner-drawn menu items) require the use of IContextMenu and its later incarnations (like IContextMenu3).

I hope you enjoy the course.