Just another Microsoft MVPs site

Preventing a DLL from being unloaded by the app that uses it

Update: See 'The right way, conclusion' for an update from Mike Grier. 

Last week I was debugging a Windows hook function, and I discovered a side effect of installing a hook: If a function is installed as a hook function, the system will prevent its containing DLL from being unloaded by the application until that shuts down.

If you stop to think about it, it makes perfect sense. The hook function is in the DLL. If it would be unloaded before the hook is removed, there would be an access violation.

An immediate use for this side effect came to mind.

Several years ago, I was working on a measurement system where the main application was some sort of scheduler. It would contain a recipe of actions to take. Each of those actions was a function in a DLL. The DLLs could be created using the development environment that came with the system.

That development environment was a wrapper around MSVC, and it came with libraries to access all of the system’s test and measurement hardware.

This system works great, but there was one problem: it was not possible to persist data in memory in between function calls. I.e. it was not possible to use global data across function calls.

The reason was that the DLLs are loaded and unloaded each time. Or at least you had to assume this was the case. Maybe the scheduler would keep the DLL in memory if the next function was in the same DLL, but you couldn’t trust this, and it would not work if they were in different modules.

Anyhow, this all came back to mind when I was debugging that windows hook. I did some research, and I have found 2 solutions to the problem I just described.

The right way

The right way is perfectly safe. It is the correct way to do things.

First of all, you should read my previous post if you are not very familiar with the DllMain callback function. DllMain is a very unsafe place to do things, but in our case, I will demonstrate that everything we do is safe.

The magic happens inside the DllMain function when the DLL is loaded into memory.

BOOL APIENTRY DllMain( HMODULE hModule,

                       DWORD  ul_reason_for_call,

                       LPVOID lpReserved

                               )

{

  switch(ul_reason_for_call)

  {

  case DLL_PROCESS_ATTACH:

    LockLibraryIntoProcessMem(hModule, &g_Self);

    break;

  case DLL_PROCESS_DETACH:

    break;

  }

    return TRUE;

}

DllMain is safe if anything in it is safe. In this case there is only 1 function call.

int LockLibraryIntoProcessMem(

HMODULE DllHandle,

HMODULE *LocalDllHandle)

{

  if(NULL == LocalDllHandle)

    return ERROR_INVALID_PARAMETER;

 

  *LocalDllHandle = NULL;

  TCHAR moduleName[1024];

  if(0 == GetModuleFileName(

DllHandle,

moduleName,

sizeof(moduleName)/ sizeof(TCHAR)))

    return GetLastError();

  *LocalDllHandle = LoadLibrary(moduleName);

  if(NULL == *LocalDllHandle)

    return GetLastError();

  return NO_ERROR;

}

This function will increment the DLL reference count by calling LoadLibrary on its containing DLL.

hModule is the handle to the containing DLL. We use this to get the full qualified name to the DLL in order to be able to call LoadLibrary.

LocalDllHandle is the variable that will receive the extra handle. This is factually redundant, because it will be the same handle as hModule. We just foresee that this could change in the future.

Since GetModuleFileName is located in kernel32.dll, it is safe to call it from within DllMain.

That’s it. As simple as that, our dll remains in memory even if the calling app tries to unload it, as can be seen by the code of a demo application.

typedef void  (*Function)(void);

int _tmain(int argc, _TCHAR* argv[])

{

  HMODULE module = LoadLibrary(L"Dll1.dll");

  Function function = (Function) GetProcAddress(module, "Function");

  (function)();

  FreeLibrary(module);

  //FreeLibrary(module);

  __try

  {

    (function)();

  }

  __except(EXCEPTION_EXECUTE_HANDLER)

  {

    MessageBox(GetTopWindow(NULL), L"access violation", L"", MB_OK);

  }

  return 0;

}

The app loads the DLL and gets function pointer to an exported function. It calls the function simply to demonstrate that it works. Then it calls FreeLibrary.

If our DLL would not have prevented itself from being unloaded, using the function pointer again would cause an access violation. This does not happen. Instead, you will get a successful function call.

The right way: Conclusion

This method solves the original stated problem, and does not have any nasty side effects. A DLL can safely load itself again, because it is already thereJ. The only net action is that the ref count increments.

It is now possible to read and write global variables inside the DLL – either directly or through accessor functions – and the data will remain in memory, even if the calling application thinks it has unloaded that DLL. Hip hip hooray.

But maybe it has already occurred to you: if the extra handle is the same one that the Calling application gets, then what prevents the application from calling FreeLibrary twice?

Nothing. In fact, you can test this for yourself. Call FreeLibrary twice, the DLL gets unloaded and reusing the function pointer causes an access violation.

There is nothing the DLL itself can do about it. But remember: we are trying to change the behavior of our DLL in the context of an existing application. It does not know what we are trying to do, nor can it notice it without some real black hacking magic.

And it is not like we are subverting the application for malicious reasons. We are changing behavior for legitimate design purposes.

But like all non-obvious behavior, the cardinal rule of programming applies: if you do things like this, document it thoroughly. Document why you have to do it, how it works exactly, and whether there are possible complications or not. Failure to do so can cause loads of problems later on.

Update from Mike Grier:

I got a reaction from Mike Grier himself. The right way that I described is indeed the right way on pre-XP systems. On XP or later you can use GetModuleHandleEx with the GET_MODULE_HANDLE_EX_FLAG_PIN flag to prevent unloading of the DLL.

The advantage is that the calling app cannot unload the DLL by calling FreeLibrary twice. The disadvantage is that you cannot unload the DLL anymore, even if you should want to.

The windows hook hack

This is what triggered this article in the first place.

As mentioned earlier, a correctly installed hook will cause its containing DLL to be fixed into the memory of its containing process.

LRESULT CALLBACK CallWndProc(

    int nCode,

    WPARAM wParam,

    LPARAM lParam

)

{

  return CallNextHookEx(NULL, nCode, wParam, lParam);

}

BOOL APIENTRY DllMain( HMODULE hModule,

                       DWORD  ul_reason_for_call,

                       LPVOID lpReserved)

{

  switch(ul_reason_for_call)

  {

  case DLL_PROCESS_ATTACH:

    g_Hook = SetWindowsHookEx(

WH_CALLWNDPROC,

CallWndProc,

hModule, 0);

    break;

  case DLL_PROCESS_DETACH:

    if(NULL != g_Hook)

      UnhookWindowsHookEx(g_Hook);

    break;

  }

  return TRUE;

}

The hook function is a dummy that does nothing else except calling the next hook. It has to do nothing except to exist.

DllMain will only be called with DLL_PROCESS_DETACH when the application exits. At that point the DLL has to unhook the hook.

The hack: conclusion

Once the hook is installed, the DLL will be locked into the process memory of its calling application. However, this method has its problems.

Most importantly: It is only safe if the calling application has already loaded User32.dll in memory. That is the DLL in which the hook functions reside. If it is not already in memory, there is NO guarantee that User32.dll has already been initialized at this point.

In that case. It will most likely not cause any problems until something subtle changes, at which point there will be all sorts of weird and difficult to diagnose bugs.

But if there is a guarantee that User32.dll is already there, this will not be an issue.

The calling application can still remove the DLL by calling FreeLibrary twice, but this will cause an access violation sometime after doing so when the system tries to use the pointer to the hook function that is not resident anymore.

This method really should not be used if you can use ‘the right way’ (and you can) but I only wanted to show the trick that got this whole article started.

You can – of course – download the demo project and see for yourself. Have fun.

11 Comments

  1. John Lim

    Hi,

    I am not sure if this question is in the same vein as what you have blogged on “Preventing a DLL from being unloaded by the app that uses it “. But essentially, i have implemented a hook similar to the code snipped you provided on “The windows hook hack” article. The problem is when I run(debug) my application that uses this dll, I am unable to recompile my dll after I have closed the application that was using this dll. The error message received was “error PRJ0008 : Could not delete file ‘c:\…\mydll.dll’.
    Somehow the dll is still being used and it cannot be deleted.
    Can you please help? Thanks.

  2. vanDooren

    Hi, yes.
    This is caused by the fact that the hook as I show it is a global hook (because thread id is 0). This means it is hooked into every thread in the same desktop.

    This also means that your DLL stays loaded, even if your app exits.
    To solve this, use UnhookWindowsHookEx to unhook the DLL before the calling application exits. that way the hook is released, and your DLL can be unloaded

  3. John Lim

    Hi vanDooren,

    Thanks. Currently i call UnhookWindowsHookEx in dllmain when DLL_PROCESS_DETACH is detected. I suppose thats not the right way?

    Thanks,
    John

  4. vanDooren

    As you already discovered, that won’t work.
    DLL_PROCESS_DETACH notifications are only sent when the DLL is unloaded. But the DLL won’t be unloaded because there is a global hook active. 🙂

    The existence of your hook prevents the unload which would remove the hook.

    There are a couple of things you can do.
    1) get the application that loaded the DLL to call a function in your DLL that would unregister the DLL.

    2) use GetModuleHandleEx with the GET_MODULE_HANDLE_EX_FLAG_PIN, or my LockLibraryIntoProcessMem fucntion to lock your DLL into memory, instead of using the global hook hack. that way it will be unloaded when the calling process ends.

    3) inject a thread in the process that create the hook (CreateRemoteThread api) and have that thread call a function in your DLL which unhooks the windows hook

    4) store the hook handle in a global data section, export a fucntion from your DLL that unhooks that handle, and have a separate application call that function. The handle needs to be in a global section because otherwise, the second application will look at a different variable (normally, each process has its own copy of the DLL globals). You also need to make sure that no 2 hooks are installed (so you have to use something to signal existence of the hook) and it could still fail if the hook handle is thread local.

    I’ve put them in order of ease. I am relatively sure that 3 should work with minimal effort. If you have to go for 4, remember that I didn’t try any of this, so the end result would probably need tweaking to make it work in your usage scenarios.

    Kind regards,
    Bruno.

  5. John Lim

    Hi Bruno,

    Thank you so much. It definitely helped. 🙂

    My original intention was to have my dll create a hook so that I could retrieve WM_GETMESSAGE (DBT_DEVICEARRIVAL and DBT_DEVICEREMOVECOMPLETE to be precise) so that the dll would know when my generic usb HID device was attached/removed. Suffice to say my dll never got that far. I know this is probably off topic but would you kindly point me to the right way of doing that as well?

    Thanks,
    John 🙂

  6. vanDooren

    What you can do (I think I did this once to see how it worked) is to have your DLL create an invisible window, and a message pump that handles those messages. Whenever a message arrives that you are interested it, execute a callback function or set a status flag or something like that.

    This is much simpler, and saves you the problems associated with hook functions that you already experienced. It is also much easier to debug.

  7. Frank Young

    Hi vanDooren,

    Your description here is valuable to me since I’m having trouble to unload the dll that contains some Detours code. And Your solution 3) on Dec. 17 exactly matches my requirement.

    However, I don’t know how to obtain the function pointer in my DLL module which unhooks the system calls.
    Existing sample codes I found leverage GetProcAddress() to obtain a well-known function pointer (which is assumed to be the same across different processes) in Kernel32.dll (like LoadLibrary). But the function in my DLL is not shared across process. Even it’s shared, different process may load it onto different memory address.

    Are you aware of any solution to this?

    Thank you very much!

    Frank

  8. vanDooren

    what you could do is to export a function in your dll that will perform the unhook function. the hook handle will be stored in a global variable in your dll.

    then a second application will inject a thread in the target process that will call that dll function. this should unhook the hook.

    to be honest, I never tried this, so there might be a caveat or 2. one thing that could be necessary is to store the hook handle in a shared data segment (using declspec). But if the unhook function executes in the context of the target process, this shouldn’t be necessary.

    Another thing that could be necessary is to have your thread function call LoadLibary after it is injected to load the dll, and then use getprocaddress to load the function pointer for the DLL function that will perform the unhook.

  9. Frank Young

    Hi vanDooren,

    Thanks.

    It turns out I have to inject the binary executable code to the remote thread and have that code to call the GetProcAddress(“MyExportedFunc”) to obtain the function pointer.

    A weird thing is: when I WriteProcessMemory to copy the &LocalThread to RemoteAddress, the copied binary isn’t exactly the same as the original code. There are 4bit error in the 5th byte
    I verify this by reading the runtime memory address in the processes, respectively. And I also memcpy the binary code to a local buffer and the same error happens.

    I guess this may be due to Windows Vista has some protection on reading data from the code segment.

    I work around this by declaring a const char[] to maintain the machine code I want to inject and now everything sees ok.

    Thank you for your help and hope my experience can be of any use.

    Frank

  10. Frank Young

    By the way, I believe your idea of injecting second dll to the remote process and having that dll unhook the first one would definitely work.

    And it seems easier. I try my current method is just out of curiosity.

  11. bullet

    From what I read your “right way” shouldn’t be done this way. The MSDN says calling LoadLibrary() in a DLLMain() should not be done which you do. I don’t even understand why this doesn’t get a lock-up, you call LoadLibrary() on a DLL that hasn’t finished loading. From my understanding this should go into an infinite loop. I didn’t read the referenced article on the NT Loader though. I assume it tags loaded-but-not-intialized DLLs?

Leave a Reply

Your email address will not be published. Required fields are marked *