In the first part of this article series I proposed a small program in the form of a “RunDll32-callable” DLL that executes the Open With dialog with the “Always associate this program” checkbox unchecked for the sake of running a file with a particular program without the risk of a permanent association. In that blog post I left as a question if we can do better, because the method I presented doesn’t work when the file is not already associated with a particular program. In this post I’m going to describe a somewhat “hacky” procedure that works with unknown file extensions also. First of all, we have to dig into the theory and explain why the “standard” use of the API does not completely fulfill our requirements.

When you invoke the “Open With” dialog by calling the SHOpenWithDialog API, what you are really invoking is one of a few different dialog box templates; the template that gets invoked is decided internally by Windows depending on a few factors. The main of those factors is if the file you are trying to open has a known or unknown file type. Indeed, you can see that the “Open With” dialog box is aesthetically a bit different if you’re dealing with a known file type than if you’re dealing with an unknown file type:

You can see that the difference (marked in a red rectangle) that tell us that they are different dialog boxes is the description text field in the dialog box on the left. By design, this dialog box also places a mark next to the “Always use the selected program” checkbox even if we don’t set the OAIF_FORCE_REGISTRATION flag in the OPENASINFO structure.

Windows considers that a file extension is “known” when there is a class key in the registry associated with that extension. In plain words, there must be a HKEY_CLASSES_ROOT\.ext registry key, with its (Default) value pointing to ExtProgID, where ExtProgID is the program identifier for that file extension (HKEY_CLASSES_ROOT\ExtProgID). For the interested reader, there’s much more information about file associations and program identifiers in this MSDN article: http://msdn.microsoft.com/en-us/library/windows/desktop/cc144104(v=vs.85).aspx. Therefore, if we could “mislead” the system and make it think that an unknown file type is known, at least momentarily, we might get our desired result. Of course, we should be neat and clear our tracks afterwards, to leave the system in the same situation as it was before.

MSDN describes an interface for file association queries: IQueryAssociations. We’re going to consume this interface in order to ask the operating system whether the file type we’ve right clicked is known or not. Here is a possible code snippet to accomplish the task:

IQueryAssociations* pqa;
DWORD cch;
BOOL bUnknownType = TRUE;

if (SUCCEEDED(AssocCreate(CLSID_QueryAssociations, IID_PPV_ARGS(&pqa))))
{
    if (SUCCEEDED(pqa->Init(ASSOCF_INIT_DEFAULTTOSTAR, pszFileExt, NULL, NULL)) && 
        FAILED(pqa->GetString(ASSOCF_IGNOREBASECLASS, ASSOCSTR_COMMAND, L"openas", NULL, &cch)))
    {
        if (SUCCEEDED(pqa->GetKey(ASSOCF_IGNOREBASECLASS, ASSOCKEY_CLASS, NULL, &hKey)))
        {
	    bUnknownType = FALSE;
            RegCloseKey(hKey);
        }
    }
}

The semantics are pretty straightforward. First, we set up a pointer to IQueryAssocations interface and then we use the GetKey method to get the registry key associated with the file extension. The FAILED(pqa->GetString(ASSOCF_IGNOREBASECLASS, ASSOCSTR_COMMAND, L"openas", NULL, &cch) line is not totally obvious. It avoids that a file extension that has one or several applications in its “open with” section is considered a known file extension. If you omit this test, some extensions would be considered known extensions even though they have not a default program associated to them; that’s not what we want.

Once we have determined that the file extension is unknown, we have to “deceive" the system a bit before calling SHOpenWithDialog as we did in the first version of the DLL. The goal is to make Windows think that the file extension is known so let’s create a HKEY_CURRENT_USER\Software\Classes\.ext and HKEY_CURRENT_USER\Software\Classes\Test registry keys and set the (Default) value of the former to “Test”. In code:

HKEY hKey, hAssocKey, hProgIDKey;
PCWSTR pszFileExt = PathFindExtension(lpszCmdLine);
LPWSTR lpszClassesSubkey = TEXT("Software\\Classes\\");
WCHAR szClassesExtKey[256] = {0};
WCHAR szClassesProgIDKey[256] = {0};

StringCchCopy(szClassesExtKey, ARRAYSIZE(szClassesExtKey), lpszClassesSubkey);
StringCchCopy(szClassesProgIDKey, ARRAYSIZE(szClassesProgIDKey), lpszClassesSubkey);
StringCchCat(szClassesExtKey, ARRAYSIZE(szClassesExtKey), pszFileExt);
RegCreateKeyEx(HKEY_CURRENT_USER, szClassesExtKey, 0, NULL, REG_OPTION_NON_VOLATILE, KEY_ALL_ACCESS, NULL, &hAssocKey, NULL);
RegSetValueEx(hAssocKey, NULL, 0, REG_SZ, (BYTE*)TEXT("Test"), (4 + 1) * sizeof(TCHAR)); // Set the "(Default)" value
StringCchCat(szClassesProgIDKey, ARRAYSIZE(szClassesProgIDKey), TEXT("Test"));
RegCreateKeyEx(HKEY_CURRENT_USER, szClassesProgIDKey, 0, NULL, REG_OPTION_NON_VOLATILE, KEY_ALL_ACCESS, NULL, &hProgIDKey, NULL);
RegCloseKey(hAssocKey);
RegCloseKey(hProgIDKey);

For demonstration purposes, I omitted all error handling in this snippet. Another improvement could be to create a uniquely named registry key, and not simply “Test”, in order to avoid name clashes. This is left as an exercise to the reader. Now it’s time to call SHOpenWithDialog and, much to our surprise, the checkbox remains unchecked, just as we want. There are no conflicts if we decide to make a permanent association, because associations created by users are registered in a different key, HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.ext\UserChoice, and our registry key is deleted after we close the Open With dialog box.

This second version of the DLL is available at my website: http://www.wintecnico.com/Ficheros/MyOpenAs_v2.zip. Of course, as it was the case with the previous version, the DLL is provided “as is”.

To sum up, I’ve described the aspects of the Open With dialog that were relevant to build a solution to our problem. We started using nothing more than the public API (a not so popular public API, though). After that, we used a debugger to demonstrate that some tricks published on the Internet have no scientific basis, and finally we suggested another solution that works with unknown file types too. The downside is that it is “hackier” than the first one and not something that you would implement in a production enviroment, but it may help you if you’re bothered by this design limitation in Windows.

One of the most common user interface problems in Windows occurs when the user unintentionally associates a file extension with the wrong application. This was not particularly problematic in Windows XP, because you had the option to revert everything using the File Types dialog, but since Windows Vista and Windows 7 don’t come with that dialog or a replacement (possibly because rewriting it to be UAC-compatible was not deem the effort for Microsoft), end users are forced to solve the problem by editing the Registry. And, you know, when I give users Registry instructions or a script to solve a problem like this, I feel Microsoft has failed and there’s room for improvement.

My fellow MVP, Ramesh Srinivasan, developed a fantastic tool to unassociate file extensions in Windows Vista and Windows 7, you can download it from here: http://www.winhelponline.com/articles/231/1/An-Utility-to-Unassociate-File-Types-in-Windows-7-and-Vista.html. At least we have a simple UI replacement for one of the most used features of the old File Types dialog.

But what if we want to prevent the problem from happening? I went to one of the best sources for usability studies, the Windows forums and newsgroups. Some people discussed that if the “Always use the selected program to open this kind of file” checkbox were unchecked by default, it would help decrease the number of cases where users want to open a file with a different program, only once, but they end up with a permanent association that they don’t know how to undo.

Exploring possible implementations

Now that we’ve determined a way to prevent the problem, we need to implement it. I searched the Internet and the articles I found didn’t convince me. All of them of them either disable or remove the checkbox (so you can’t register an association even if you want to). For example, the tweak explained in this page is more a side effect caused by passing wrong arguments to a DLL function than a valid solution.

In my search, I discovered that Windows Vista introduced a new API that looked promising: SHOpenWithDialog. The second parameter of this function is a pointer to a constant structure, OPENASINFO, that specifies the filename, description, and some flags that control the characteristics of the dialog box. The flag OAIF_FORCE_REGISTRATION, per the MSDN documentation, forces the “Always use this program” checkbox to be checked. After reading carefully the semantics of the other flags I determined that I needed to combine the OAIF_ALLOW_REGISTRATION, OAIF_REGISTER_EXT and OAIF_EXEC flags:

OPENASINFO oainfo = {0};
LPWSTR  lpwszCmdLine; // This will be a function parameter
HWND hWnd; // This will be a function parameter
oainfo.pcszFile = lpwszCmdLine;
oainfo.oaifInFlags = (OAIF_ALLOW_REGISTRATION | OAIF_REGISTER_EXT | OAIF_EXEC);
SHOpenWithDialog(hwnd, &oainfo);

And that’s it, we have almost finished our very simple program, but we still need to know how to pass in the file path and the window handle. That’s not hard if we do some inspection on the “(Default)” value of the registry key HKEY_CLASSES_ROOT\Unknown\shell\opendlg\command. In general, inspection is a valuable skill to detect patterns in existing designs, and to avoid engineering things from scratch. When you invoke the “open with” action, Windows calls %SystemRoot%\system32\rundll32.exe %SystemRoot%\system32\shell32.dll,OpenAs_RunDLL %1, i.e., it calls a function in Shell32.dll by using the Rundll32.exe executable. To be callable from Rundll32.exe, functions must conform to a particular signature. This is it for Unicode:

void CALLBACK EntryPointW(HWND hwnd, HINSTANCE hinst, LPWSTR lpszCmdLine, int nCmdShow);

So the final result is:

void CALLBACK EntryPointW(HWND hwnd, HINSTANCE hinst, LPWSTR lpszCmdLine, int nCmdShow)
{
OPENASINFO oainfo = {0};
oainfo.pcszFile = lpszCmdLine;
oainfo.oaifInFlags = (OAIF_ALLOW_REGISTRATION | OAIF_REGISTER_EXT | OAIF_EXEC);
SHOpenWithDialog(hwnd, &oainfo);
}

This is basically all the code we need. Compile it and create a DLL or download the DLL from my server: http://www.wintecnico.com/Ficheros/MyOpenAs.zip. One question you may ask: “Can we do better? Why it doesn’t work with unknown file types?” This question and some in-depth analysis and debunking of some of the tricks that you may find  on the Internet will be the topic of the next article.