AdSync validated for the plant network

Last week my AdSync application was validated for running on the plant network.


The plant network exists for running the DCS application that controls everything and the kitchen sink. The application runs on a different desktop as some sort of kiosk application. This means that the users have no access to the windows desktop.


The user management of this network is completely implemented within the DCS software. This means that all groups and privileges are application defined. I can create new groups and modify permissions, but all of that stays within the application. Of course the user accounts themselves exist in Active Directory, but everything that is DCS related (groups and permissions) is configured in an application database.


The main reason that this is done (I think) is that the software can also run in a workgroup environment where no Active Directory exists. Imo they should deprecate this and switch to AD only as soon as possible. It’s not like there’s any reason not to run a domain, given the size of the networks for which this software is intended.


Anyway, this is a bit of a problem because people get assigned to groups, based on the department they are working for, and the things they need to do. E.g. people are in the QA group, or Automation, or Technology…


So far so good, but unfortunately, the department someone is in also has implications for Active Directory. Automation people need to be able to reboot certain servers, including the domain controller. The technology group needs a protected folder on the fileserver for storing the reports they generate for the FDA filing of our product…


Before now, there was no way to handle all these things, other than manually creating groups and assigning people to it on an as-needed basis. This is tedious for one thing, but it people change groups, their other permissions would not be changed accordingly.


To solve this problem is a scalable way, I’ve programmed an application called ‘AdSync’.


AdSync will be executed every night after the daily backup scripts have taken an export of the DCS configuration database. Another of my applications will parse this database file and generate an XML file containing just the user account configuration, including users, groups and permissions.


AdSync will first check the groups that exist in the XML file, and verify if they exist in AD. If they don’t exist, they will be created, but with a ‘DCS ‘ prefix. This way there is a clear distinction between the existing groups, and those that are managed by my application.


When that is done, the group membership of the user in the XML file is compared with the Active Directory group membership. If they are different, the necessary changes will be made to the AD group. This way, the AD membership is always a mirror of the DCS group membership.


The synchronization is done only once per day, because it is based on the nightly backup of the configuration database. Exporting that database takes about an hour, and renders the system unusable for engineering work. This means we can only do it at night.


Every change that is made by my application, as well as the execution timestamp and any errors that may occur are logged in a custom Windows Event Log for auditing.


Since this is a production environment of pharmaceutical products, installing this on the system takes a good amount of paperwork. The change has to be approved by QA and other relevant departments; I have to do an impact assessment; I have to write an installation manual; I have to set up a validation protocol, and execute it with a witness from validation to prove that the application works as intended; …


So all in all, it takes quite a lot of work, but now everything is ready. Yaay me. J


My application was written with VC# 2008, but built to run on the .NET 2.0 framework, which is qualified by the DCS vendor for use alongside the DCS software. The Active Directory interface was made using the System.DirectoryServices namespace. I used the .NET Developer’s Guide to Directory Services as a reference, because the System.DirectoryServices namespace is horrible underdocumented in MSDN.


I have to admit it is nice to keep on developing software. Officially I am a system administrator, but my boss agrees with me that making custom tools like these is something of real value to the company, and something that cannot be outsourced without spending a significant amount of money.

Planning the plant shutdown, part 2

The preparation of the plant shutdown continues unabated. More and more I can understand why this needs months of intense planning.


There are lots of dependencies between everything that needs doing. For example: the floors of the cold room need new coating. For this the entire room needs to be brought to near environmental temperature, but this cannot happen too suddenly or the floor will crack.


Sanding away the old coating can be done while the temperature is still low. And during the first day, another team has access to the ceiling panels, but not during the final sanding or coating stages. Other work needs to wait until the coatings are dry. The other teams can then work in the cold room as the temperature is dropping again.


But as the temperature is re-established, it is important that the controller loops can run uninterrupted. This means from that time onwards I cannot disrupt the DCS network anymore because that would cause problems for the people who rely for their work on the correct functioning of the DCS network.


After a lot of discussion back and forth, I’ve gotten 1 weekend (from Friday evening to Sunday evening) where I can do with the system as I please. During the same weekend, the utilities will all be down, as well as mains electrical distribution.


During the shutdown I need to do quite some things. But as luck has it, we have bought additional server racks and servers. Several servers need replacing anyway, so this is a rough sequence of how we will do things:


  1. Create an additional Domain Controller, put it in the new rack, disconnect it from the network, and connect it to a new switch. That switch will be the backbone of a ‘clone’ of the DCS domain.
  2. Rename it to the same name as the old master server which has to be replaced.
  3. Issue a development freeze so that we can restore the databases to the new server without risking the loss of work.
  4. Restore the master databases and perform the software upgrade.
  5. Add new servers in the rack to replace the other machines that needed replacing. All new servers will be connected to the cloned domain.
  6. Move the engineering workstation to the cloned domain.
  7. Wait for official signoff of the DCS network. At this moment we have the major infrastructure running on a cloned domain.
  8. As soon as we have ownership of the network we shutdown the live domain controller and all other servers that have a replacement servers running in the cloned domain.
  9. As soon as the domain controllers are down, the backbone switch of the cloned domain is connected to the live network so that the new domain controller can take over.
  10. The automation engineers can start their work as soon as that happens, because the engineering workstation is already back online.
  11. While the automation engineers do their thing, we can move the remaining servers into their new slots in the new server racks and reconfigure them.
  12. The old servers which were replaced can be taken out of the existing racks and put aside for repurposing.
  13. The additional servers which were bought for new functionality can be moved into the existing racks and configured.
  14. While all this is going on, the operator terminals can all be reconfigured 1 by 1 after their software upgrade.

This is a rough first draft of how what I will do during shutdown. I’ll need to make a more detailed planning and see what can be done in parallel. The brunt of the work needs to be done beforehand so that the cloned domain can be prepared well before the shutdown. We only have 2 days of shutdown to do what we need, so everything needs to be done as efficiently as possible.

My latest article in NTInsider

A bit late perhaps, but my latest article on API development got published in the NTInsider (owned by OSR Online).


The online version can be found here. It requires free registration.


I know that at least 2 people have read it completely, because that is the number of mails I got to tell me I missed something.


  1. In the code example at the bottom, I acidentally switched the names of fooA and fooB in the declaration. Doh!
  2. That same code, compiled for a 64 bit platform, gives a linker warning. I don’t know how I missed that. This code was made using VS2005, so one of two things could have happened: VC2005 didn’t report the warning (unlikely), or I only checked the compilation in detail, and then only checked if the function was exported without checking if there were linker warnings. It’s probably the latter.

I’ll see if I can get these errata published. But at least now I know that people actually read my articles and use the code. Well, at least one person is using the code :) I’ll have to make an extra effort next time to verify that a) I didn’t make a stupid type, and b) There are no warnings anywhere.

Configuring an application through a batch file: what domain am I in?

I recently had to make an install script for one of my applications which manipulates the Active Directory. The app itself is executed via a scheduled batch file, and it expects the domain name as a command line parameter. I wanted the batch file to figure out on its own what the domain was of the computer on which it is executed. That turned out to be harder than I thought, but in the end I found an elegant, if rather hairy solution.


FOR /F “tokens=1* delims=REG_SZ ” %%A IN (‘REG QUERY HKLM\System\CurrentControlSet\Services\Tcpip\Parameters /v Domain’) DO (
SET CURR_DOMAIN=%%B
)


This code snippet will query the registry to get the Domain the computer belongs to. Of course, the REG QUERY command returns not just the name, but a tab separated table with the key name, the key type and the key value. That table is then munged by the ‘FOR’ loop which uses ‘REG_SZ ‘ (the whitespace is a tab character) as a delimiter. The actual domain name itself is then stored in the variable ‘CURR_DOMAIN’


It’s crude, but it works rather well. This trick saves me the necessity of having someone configure this by hand when the application is installed. Apart from the fact that this is error prone, it would make the installation procedure more verbose.


I haven’t tried this on computer that is not part of a domain.  Luckily, that is not a situation that will ever occurr on the network that I have to maintain.

C++ keyword of the day: export

The export keyword is a bit like the Higgs boson of C++. Theoretically it exists, it is described by the standard, and noone has seen it in the wild. :)


Before I get flamed to hell and back: that last part is not entirely true. There is 1 C++ compiler front-end in the world which actually supports it. That is the one made by EDG (Edison Design Group). This is used by Comeau which claim to have the only compiler that is 100% standards compliant and Borland (for which the support of export is a bit vague). I heard that the Intel compiler shipped with it, but I don’t know if this is true or not.


What’s all the fuss about?


It seems a bit strange that a language should contain a keyword that all compiler makers refuse to implement. Comeau has some interesting content here as does Wikipedia.


The reason that I believe ‘export’ has never really taken off is that is seems to be an significant effort to implement it, combined with the fact that is doesn’t achieve all that much. The export keyword allows a C++ programmer to declare a template class in a header, and then provide the implementation separately in a cpp file. This is the normal C++ way of doing things. Unfortunately, there’s a snag.


Template class implementations cannot be compiled on their own. Template code itself is meaningless without specification of template arguments. Thus it is that a compiler doesn’t know what to do with an implementation until the template class is used somewhere. At that point the template arguments will be known, and the compiler can compile the template class.


Since the template is only compiled then, the easiest thing to do would be to do what most compilers do now: demand that the implementation is in scope when it is used somewhere. I.e. the implementation has to be put in a header file, and that header file has to be included by the source file that is being compiled. the reason for this is that the compiler would simply not know where to look for it if the implementation was in a cpp file somewhere.


The export keyword would solve this by telling the compiler ‘Look somewhere else for the implementation’. The compiler would then have to compile the implementation for that class, and save the object code somewhere for that combination of template arguments.


The export keyword sounds like a great thing, but it doesn’t solve that many problems.


Separation of interface and implementation


The separation of interface and implementation is one of the cornerstones of C++ philosophy. The traditional way is to have the declaration in a header file, and then put the implementation in a cpp file.


Since a template implementation cannot be compiled anyway, we could just as well put it in a header file.


And then we could also make 1 header file with just the declarations, and underneath that include the header file(s) containing the implementation. The nice thing is that you can then put specializations in different headers to keep the code comprehensible.


This way we keep true to the principle of separation without needing the ‘export’ keyword


Distribution


You can’t distribute template classes in a binary form, no matter whether export is supported or not.


Because templates cannot be compiled on their own, you cannot link them into a static or dynamic library. The only thing you can do is to distribute the source code.


Templates are distributed as source code. The export keyword cannot do anything about that. And if you want to distribute templates, what are you going to do? You can develop your classes using ‘export’, but then they will be usable only by a very limited subset of compilers.


If you develop your classes as if ‘export’ doesn’t exist, then they can be used by all compilers.


Of course you could do both, and use ugly macro magic to construct your code in such a way that would leave the choice to the client programmer. Some people do this. But that is kind of  ugly and time consuming. Furthermore, it’s not like it gets you any more clients, because they will be able to use your code in any case.


Compiler performance


It is true that our scheme of using header files is less efficient than using the export keyword. Without ‘export’, a template is compiled whenever it is used. You can mitigate this somewhat by including commonly used templates in a precompiled header file. But the impact will only be somewhat lessened.


And whenever an implementation detail changes, every source file which includes the template has to be recompiled. This can be a real pain in large projects.


Conclusion


‘export’ is facing the chicken or the egg dilemma.


‘export’ is neglected by most compiler vendors because it is not an important language feature -> You cannot use it in code that has to be portable (which is ironic, since it is a standard keyword) -> it is not used in many scenarios -> there is no large codebase depending on support for ‘export’ -> there is little incentive for compiler vendors to start supporting it.


Make no mistake: I would appreciate it if I could use ‘export’, if only because it allows me to organize my template implementations in a standard manner, but I don’t think it’ll happen soon.

Smoke and mirrors in the world of finance

The recent nosedive of Lehman Brothers brought back some memories of a company that I once worked for as a contractor.


That company was a NASDAQ darling for a long time, but the financial situation was not great when the stock market for high tech stuff collapsed. Our site had to work hard to save enough money to be spared the axe. Unfortunately, in the year leading up to the stock collapse, a lot of equipment had been ordered for the purpose of increasing our ouput with 100%. They had to do this, because a lot of equipment had lead times up to a year.


Thus it was that while everyone started to wonder about the reality of getting virtually no orders at all, new equipment was pouring in that we already didn’t need anymore. And in the semiconductor high speed optics world, there is no such thing as cheap equipment. The cheapest thing was probably a power meter at 30 KEuro. The most expensive thing was a 500KEuro 10 GGHz bit error rate tester.


I should note that the CFO of the site was very good at what he did, which allowed the site to exist for as long as it did. In order to make the papers look better, he invoked a special rule over the new equipment: noone could use it.


Apparently, if you don’t start using equipment, it is not subject to the write off procedures, and it hasn’t actually cost any money in the books. The hardware still represents the original sum of money and as a result, doesn’t show up as a cost. Financially, it was just very bulky money. Note that this is completely legal.


Unfortunately it is also as realistic as the tooth fairy, because in reality, the money is already gone. So you get situations like I had, where you have a power meter shaped hole in a measurement system and a brand new power meter waiting to be used. I wan’t allowed to put 1 and 1 together because that would cost money. I argued that the money was already gone, and we might as well do something useful with the equipment. And while the director agree with me, he also said that it didn’t work that way on paper.


It was then that I realized the world of finances is all smoke and mirrors, and it only works as long as the people believe the dream. As soon as everybody discovers that the gold is just gilt, you get what happened to Lehman Brothers. Financial institutions are like hot air balloons. If the hot air gets out, all that is left is the skin and the dead weight, and it will plummet to its rocky end.

References in C++ are not necessarily safe

People who are new to C++ sometimes have the mistaken idea that using references instead of pointers makes your code safe. People who have been programming a bit longer know this is anything but the case. References are just semantic sugar coated pointers.


I’ll explain in more detail with a couple of examples. They use this pretty simple class A.


struct A
{
 
int &I;
 
A(int &i) : I(i)
 
{
 
}
 
void print(void)
 
{
   
cout << I << endl;
 
}
};


When A is instantiated, the constructor takes a reference to an integer and uses it to initialize an internal integer reference. The print method simply prints the value of that integer.


Case 1


int * i= new int(32);
A a(*i);
*i = 43;
a.print();
delete i;
a.print();


I create a new integer value on the heap and I initialize a pointer with its address. I then create a new instance of A, and pass the newly created integer by reference. a initializes its own internal reference with the address of that integer. And indeed, if I assign a value to that integer and then print a’s internal reference, the values match.


But what happens if that integer is deleted from the heap? It depends. The behavior is undefined. The next read from a.I could result in a bad value or an application crash. But in any case, it is not safe.


You might argue that I am purposely using pointers to cause problems, but in a large application, you won’t know how and where your objects will be created. This could happen.


Case 2


A second case is much more innocuous. It uses no visible pointers.


vector<int> v;
v.push_back(42);
A b(v[0]);
v[0] = 43;
b.print();
v.pop_back(); //remove that element from the container
b.print();
v.push_back(2); //put something else in the same location
b.print();


Instead of dynamically allocation memory myself, I put an integer in a vector and then feed a reference to that element into an instance of A. This new instance (b) is now using the location of an element in a vector as an internal reference.


The thing with vectors is that they normally have more capacity than elements. This is because you don’t want the vector to allocate more space for every element that gets inserted. Likewise, you are not going to do a deallocation with each element that gets removed.


Because of this, when that element is removed, the physical space is still there. A read from that location is not immediately going to trigger a crash. Instead, the program will keep on running, but with bad data instead. Or maybe not, if that element was at the threshold of triggering a deallocation. With bugs like this, you never know what will happen.


Case 3


There is an even more interesting option. I am not going to write a repro case for it, but consider the following: I create multiple instances of A, giving them each a reference to an integer that is located on the stack, and I pass those instances of A as pointers to newly created threads, to be used there.


Meanwhile, the stack on which the integers are located is unwound. What happens next?


Again, it depends. Probably the stack gets stomped. Or perhaps only the data will be corrupted. Again, what happens is unknown. And I don’t say that this design pattern is a good one. It isn’t. But given certain constraints, it might just be an approach for solving a specific problem (though perhaps not the best one).


Conclusion


By now, anyone should be convinced that using references do not protect your application against design mistakes.


The only time your references are really safe is if:
a) your application is single threaded, and there is no asynchronous code executed anywhere, and
b) your classes don’t store references anywhere.


Outside of those constraints, references don’t guarantee anything more than pointers would. The only guarantee you have with references is that at some point in time, there was an object of the correct type at the specified location, and it was not NULL. Probably.


If someone is actively trying to subvert your code, you can’t even be sure of that. But since there is nothing you can do about that, you might as well use references because they make your life easier, and are better than raw pointers when it comes to preventing honest mistakes.

Please use a ‘real’ email address

Yesterday I tried to register on the opcfoundation forum so that I could find out why the opc SDK comes without documentation or code samples.


I know that corporate members (i.e. the ones who pay $$$$) can download code samples and specifications, but a publicly available SDK is not going to do anyone much good without access to at least the documentation.


Anyway, the message board required me to specify my email address so I used my gmail address. I also made a mistake while typing in the captcha. After looking again after the first 2 failures, I noticed that it had to be typed in from right to left, but by then my email adress had been banned.


I sent a mail to the forum admin, asking for the ban to be undone. This is the complete message I got back.


All gmail accounts are banned.  Please use a “real” email address.


Apart from the total lack of style which I would not expect from an admin of a forum for professionals (notice the lack of addressing), it amazes me that people still think that webmail addresses are unclean (notice the sarcasm). Most people I know on most forums (incl the forum for which I am a moderator) use webmail. And this specific forum was very low traffic so it’s not like they are battling spammers by the hundreds.


My gut tells me that the problem is that webmail addresses are primarily used by non-paying members. Foundations like these are interested in big bucks. If you can’t even drop 1000$ for the membership, then you are obviously to poor to count for anything, and you can subsequently be dismissed without much thought.

How DLL exports really work

I found this list of article on Raymond’s blog. Raymond’s blog is one of the more interesting for programmers who use native APIs because he often touchs on things that are not documented, but interesting to know if you care about how things really work under the hood.


These links all point to information that goes into DLL importing and exporting, and how it is implemented under the hood. Very interesting stuff.


  • How were DLL functions exported in 16-bit Windows?

  • How were DLL functions imported in 16-bit Windows?

  • How are DLL functions exported in 32-bit Windows?

  • Exported functions that are really forwarders

  • Rethinking the way DLL exports are resolved for 32-bit Windows

  • Calling an imported function, the naive way

  • How a less naive compiler calls an imported function

  • Issues related to forcing a stub to be created for an imported function

  • What happens when you get dllimport wrong?

  • Names in the import library are decorated for a reason

  • The dangers of dllexport

  • Another thing which may be useful if you haven’t read it yet are Michael Grier’s articles on the NT DLL loader over here. I’ve blogged about those articles before. Michael is one of the project leaders who maintained the NT DLL loader for some time.


     

    Stuck on stupid

    It happens to every programmer. It doesn’t happen often, but it does happen to everyone. You are looking at a piece of code or some debugging output, scratching your head and thinking ‘This is impossible’. It’s probably only 10 or 20 lines of code or text, and you’ve looked carefully at every line at least a dozen times over the last hour.


    Every line makes sense and seems to mean exactly what you think it means, every word is in place, every semicolon is accounted for, and yet the sum of those lines is something completely unlike what you think it should be.


    What makes matters even worse is that you just know that you are missing something obvious; something that anyone else would see after looking at the issue for less than 5 seconds. Maybe someone is even explaining the issue to you and you still don’t get it.


    Well, I had such a moment just this weekend.


    I was working on some C++ code that involved friend functions and streams. I’ve never done much with streams in C++, and never done anything with friends. Basically, friend declarations allow e.g. class A to specify that class B is allowed to access its private variables. Hence the joke: in C++, your friends can see your privates. I’ve never used it in a design, because if other classes need access to private data, the design is probably wrong.


    There are some exceptions, where it can be extremely useful. One of these cases is if you have to create a serialization function that can serialize and deserialize an object. That function would need access to the private data to efficiently reconstruct the object, but you don’t want any other functions or classes to have the same privilege.


    In my case, I had a template class that I wanted to reconstruct from a generic iostream. I implemented a global function that performed the reconstruction, but despite my friend declaration that gave istream access, the compiler didn’t agree. This is what my code boiled down to:


    class A
    {
    friend istream;
    int i;
    };

    istream& operator >>(istream& is, A &a)
    {
    is >> a.i; //error C2248: ‘A::i’ :
    //cannot access private member declared in class ‘A’
    return is;
    }


     


    I looked at it for a long time, read the ‘friend’ documentation in MSDN, consulted my copy of ‘The C++ programming language, 3ed’, asked a fellow MVP to explain it to me, and I still didn’t get it. I reasoned: ‘I just gave istream access to A::I, so WTF is the problem’.


    In this case, the solution was really a ‘duh’ moment. Yes, istreams have access to A. But – and this is the whole issue – it is not the istream doing the accessing. The istream accesses an integer by reference; an integer that was retrieved by the global operator >> function. My code was functionally equivalent to:


    istream& operator >>(istream& is, A &a)
    {
    int &temp = a.i; //error C2248
    is >> temp;
    return is;
    }


    And that is pretty obviously wrong. By writing is >> a.i, I lulled myself into believing it was ‘is’ who did the access, while in reality it was someone else.


    As I said already: this was one of those ‘Aaargh’ moments where I just seemed to be stuck on ‘stupid’. Ah well. Live, learn, and have more coffee next time J Glad this doesn’t happen every day, week, or month.