Preventing a DLL from being unloaded by the app that uses it

Update: See 'The right way, conclusion' for an update from Mike Grier. 

Last week I was debugging a Windows hook function, and I discovered a side effect of installing a hook: If a function is installed as a hook function, the system will prevent its containing DLL from being unloaded by the application until that shuts down.

If you stop to think about it, it makes perfect sense. The hook function is in the DLL. If it would be unloaded before the hook is removed, there would be an access violation.

An immediate use for this side effect came to mind.

Several years ago, I was working on a measurement system where the main application was some sort of scheduler. It would contain a recipe of actions to take. Each of those actions was a function in a DLL. The DLLs could be created using the development environment that came with the system.

That development environment was a wrapper around MSVC, and it came with libraries to access all of the system’s test and measurement hardware.

This system works great, but there was one problem: it was not possible to persist data in memory in between function calls. I.e. it was not possible to use global data across function calls.

The reason was that the DLLs are loaded and unloaded each time. Or at least you had to assume this was the case. Maybe the scheduler would keep the DLL in memory if the next function was in the same DLL, but you couldn’t trust this, and it would not work if they were in different modules.

Anyhow, this all came back to mind when I was debugging that windows hook. I did some research, and I have found 2 solutions to the problem I just described.

The right way

The right way is perfectly safe. It is the correct way to do things.

First of all, you should read my previous post if you are not very familiar with the DllMain callback function. DllMain is a very unsafe place to do things, but in our case, I will demonstrate that everything we do is safe.

The magic happens inside the DllMain function when the DLL is loaded into memory.

BOOL APIENTRY DllMain( HMODULE hModule,

                       DWORD  ul_reason_for_call,

                       LPVOID lpReserved

                               )

{

  switch(ul_reason_for_call)

  {

  case DLL_PROCESS_ATTACH:

    LockLibraryIntoProcessMem(hModule, &g_Self);

    break;

  case DLL_PROCESS_DETACH:

    break;

  }

    return TRUE;

}

DllMain is safe if anything in it is safe. In this case there is only 1 function call.

int LockLibraryIntoProcessMem(

HMODULE DllHandle,

HMODULE *LocalDllHandle)

{

  if(NULL == LocalDllHandle)

    return ERROR_INVALID_PARAMETER;

 

  *LocalDllHandle = NULL;

  TCHAR moduleName[1024];

  if(0 == GetModuleFileName(

DllHandle,

moduleName,

sizeof(moduleName)/ sizeof(TCHAR)))

    return GetLastError();

  *LocalDllHandle = LoadLibrary(moduleName);

  if(NULL == *LocalDllHandle)

    return GetLastError();

  return NO_ERROR;

}

This function will increment the DLL reference count by calling LoadLibrary on its containing DLL.

hModule is the handle to the containing DLL. We use this to get the full qualified name to the DLL in order to be able to call LoadLibrary.

LocalDllHandle is the variable that will receive the extra handle. This is factually redundant, because it will be the same handle as hModule. We just foresee that this could change in the future.

Since GetModuleFileName is located in kernel32.dll, it is safe to call it from within DllMain.

That’s it. As simple as that, our dll remains in memory even if the calling app tries to unload it, as can be seen by the code of a demo application.

typedef void  (*Function)(void);

int _tmain(int argc, _TCHAR* argv[])

{

  HMODULE module = LoadLibrary(L"Dll1.dll");

  Function function = (Function) GetProcAddress(module, "Function");

  (function)();

  FreeLibrary(module);

  //FreeLibrary(module);

  __try

  {

    (function)();

  }

  __except(EXCEPTION_EXECUTE_HANDLER)

  {

    MessageBox(GetTopWindow(NULL), L"access violation", L"", MB_OK);

  }

  return 0;

}

The app loads the DLL and gets function pointer to an exported function. It calls the function simply to demonstrate that it works. Then it calls FreeLibrary.

If our DLL would not have prevented itself from being unloaded, using the function pointer again would cause an access violation. This does not happen. Instead, you will get a successful function call.

The right way: Conclusion

This method solves the original stated problem, and does not have any nasty side effects. A DLL can safely load itself again, because it is already thereJ. The only net action is that the ref count increments.

It is now possible to read and write global variables inside the DLL – either directly or through accessor functions – and the data will remain in memory, even if the calling application thinks it has unloaded that DLL. Hip hip hooray.

But maybe it has already occurred to you: if the extra handle is the same one that the Calling application gets, then what prevents the application from calling FreeLibrary twice?

Nothing. In fact, you can test this for yourself. Call FreeLibrary twice, the DLL gets unloaded and reusing the function pointer causes an access violation.

There is nothing the DLL itself can do about it. But remember: we are trying to change the behavior of our DLL in the context of an existing application. It does not know what we are trying to do, nor can it notice it without some real black hacking magic.

And it is not like we are subverting the application for malicious reasons. We are changing behavior for legitimate design purposes.

But like all non-obvious behavior, the cardinal rule of programming applies: if you do things like this, document it thoroughly. Document why you have to do it, how it works exactly, and whether there are possible complications or not. Failure to do so can cause loads of problems later on.

Update from Mike Grier:

I got a reaction from Mike Grier himself. The right way that I described is indeed the right way on pre-XP systems. On XP or later you can use GetModuleHandleEx with the GET_MODULE_HANDLE_EX_FLAG_PIN flag to prevent unloading of the DLL.

The advantage is that the calling app cannot unload the DLL by calling FreeLibrary twice. The disadvantage is that you cannot unload the DLL anymore, even if you should want to.

The windows hook hack

This is what triggered this article in the first place.

As mentioned earlier, a correctly installed hook will cause its containing DLL to be fixed into the memory of its containing process.

LRESULT CALLBACK CallWndProc(

    int nCode,

    WPARAM wParam,

    LPARAM lParam

)

{

  return CallNextHookEx(NULL, nCode, wParam, lParam);

}

BOOL APIENTRY DllMain( HMODULE hModule,

                       DWORD  ul_reason_for_call,

                       LPVOID lpReserved)

{

  switch(ul_reason_for_call)

  {

  case DLL_PROCESS_ATTACH:

    g_Hook = SetWindowsHookEx(

WH_CALLWNDPROC,

CallWndProc,

hModule, 0);

    break;

  case DLL_PROCESS_DETACH:

    if(NULL != g_Hook)

      UnhookWindowsHookEx(g_Hook);

    break;

  }

  return TRUE;

}

The hook function is a dummy that does nothing else except calling the next hook. It has to do nothing except to exist.

DllMain will only be called with DLL_PROCESS_DETACH when the application exits. At that point the DLL has to unhook the hook.

The hack: conclusion

Once the hook is installed, the DLL will be locked into the process memory of its calling application. However, this method has its problems.

Most importantly: It is only safe if the calling application has already loaded User32.dll in memory. That is the DLL in which the hook functions reside. If it is not already in memory, there is NO guarantee that User32.dll has already been initialized at this point.

In that case. It will most likely not cause any problems until something subtle changes, at which point there will be all sorts of weird and difficult to diagnose bugs.

But if there is a guarantee that User32.dll is already there, this will not be an issue.

The calling application can still remove the DLL by calling FreeLibrary twice, but this will cause an access violation sometime after doing so when the system tries to use the pointer to the hook function that is not resident anymore.

This method really should not be used if you can use ‘the right way’ (and you can) but I only wanted to show the trick that got this whole article started.

You can – of course – download the demo project and see for yourself. Have fun.

What every programmer should know about DLL loading

Every programmer who ventures beyond his / her first hello world application has to deal with DLLs sooner or later. Maybe you even have to create your own DLLs, or load and unload DLLs on demand.

In that case – if you have not already done so – you have to go and read mgrier's articles on the NT DLL loader over here.

mgrier is one of the project leaders who maintained the NT DLL loader for some time. This information can prevent you from doing stupid things that might come back to haunt you or someone else, after years of no problems at all.

Fun with templates: implementing policy based error handling

This article is a follow up of my third article on interoperability for LabVIEW arrays in C++. That article provided a method for allowing programmers to index into multi dimensional arrays.

As far as functionality went, the solution was OK, but the error handling left a lot to be desired. I also wanted the error handling strategy to be policy based. I.e. you should be able to choose at compile time whether the classes use return value error handling or exceptions, or maybe something else entirely, like structured exceptions.

After some experimenting, I now have a decent solution that is elegant and maintainable.

The general idea

After writing my previous article in this series, I was kicking around with a few ideas, but I got a great suggestion from my fellow MVP Eugene Gershnik.

Basically, it boils down to the following idea:

struct EBThrowing

{

  typedef void RetVal;

 

  static RetVal ReturnError(int ErrorCode)

  {

    if(0 != ErrorCode)

      throw Lvtl_Exception(ErrorCode);

  }

};

 

struct EBRetVal

{

  typedef int RetVal;

 

  static RetVal ReturnError(int ErrorCode)

  {

    return ErrorCode;

  }

};

template<class EB> class CClass()

{

  typename EB::RetVal foo(void)

  {

    int errorCode = 0;

    //…

    return EB::ReturnError(errorCode);

  }

};

This example defines 2 error handling policies. One policy specifies that error handling constitutes returning error codes. The other policy specifies that errors will be reported as exceptions.

Each of those policies has a typedef to define the ‘RetVal’ type, as well as a function called ‘ReturnError’.

Each class that wants to implement policy based error handling has to be a template class that takes a policy class as input.

Return value types

This is where the magic happens: all methods that could trigger an error have to have a return value type that is defined by the policy class. Since these types are resolved at compile time, the function

typename EB::RetVal foo(void);

is equivalent to

int foo(void);

if the error handling policy class is EBRetval. If the policy class is EBThrowing, the function is equivalent to

void foo(void);

As a result, it is hard to accidentally misuse an instance of CClass, as far as error handling goes.

Suppose you try to use an instance of CClass that throws exceptions, in a situation where you expect return values will generate a compiler error because

int ret = foo();

cannot compile if foo is of type void.

Because the type identity of CClass is determined in part by the error policy, it is impossible to pass an instance of CClass<EBRetVal> where an instance of CClass<EBThrowing> is expected or vice versa.

Raising errors

Part of the policy contract is that the consuming classes never raise an error directly. I.e. they never return an actual value or throw an exception directly.

Instead, all errors will be raised by calling RaiseError.

return EB::ReturnError(errorCode);

The ReturnError function has a return value of type EB::RetVal. It depends on the policy. This is necessary because the methods of the consuming class also have this return type, and they have to match.

If the error policy prescribes error handling via return values, ReturnError simply does this:

return ErrorCode;

If the policy prescribes error handling via C++ exceptions, ReturnError simply does this:

if(0 != ErrorCode)

  throw Lvtl_Exception(ErrorCode);

Not raising false errors

Consider the following snippet:

typename EB::RetVal foo(void)

  {

    int errorCode = 0;

    //…

    return EB::ReturnError(errorCode);

  }

The error code will be determined by what happens in foo. However, if there was no error, no error should be raised either. With standard exception handling, this would not be a problem, because foo would simply not raise an error if there was no error.

But if return value error handling is used, foo always has to return a value, even if there was no error. In that case the general behavior is to return 0.

CClass should never return a value directly, since this would break the policy contract. Instead it must always use ReturnError. This means that ReturnError will be called with 0 as the error code if there was no error. ReturnError should then check if the error code really indicates an error before doing anything.

Providing a default policy

Template classes allow template parameters to be specified with default values. This makes it possible to use the template class without having to specify the default parameters. This makes the code more readable, and the program easier to write.

However, in this case the code is meant to be used as a toolkit by other developers. If we provide a default policy, it will be wrong for half of the users.

template<class EB=EBThrowing> class CClass();

If I want to use the EBRetVal policy, I would still have to provide a value for the EB parameter.

To solve this, I have changed CClass.

#ifndef LVTL_OVERRIDE_EB_DEFAULT

typedef EBThrowing     EBDefault;

#endif

template<class EB=EBDefault> class CClass();

This construct allows me to specify a default ‘default’ policy, while still allowing the user to override it with another default by supplying a custom typedef.

Usually, only 1 policy is used throughout the whole dll or application, so allowing the user to provide a default policy should in practice mean that they never have to supply one explicitly.

Limitation 1

If you design classes that implement a policy based error handling strategy, you should take care that cleanup actions are policy agnostic.

I.e. you must make sure that cleanup is done before you raise the error. For example, if you would like to enable the user to provide his own policy class that uses structured exceptions, you have to be aware that you cannot guarantee that objects will be properly destructed when an error is raised

Limitation 2

You should also be aware that it is difficult to layer policy based classes on top of each other while retaining the policy scheme. This is demonstrated by the this implementation of foo, where bar is an instance of a CBar class that implements policy based error handling.

typename EB::RetVal foo(void)

  {

    int errorCode = 0;

    //…

    bar->SomeMethod();

    //…

    return EB::ReturnError(errorCode);

  }

The problem is obvious. How would you know that you need to cleanup and bail out if bar raised an error?

You cannot check its return value, nor catch an exception, because you don’t know which of those you will get. Remember, using policies means that you don’t know what bar will do in advance.

If you want to use CBar, you have to fix its policy in order to be able to use it reliably.