Analysis of an opinion piece



A couple of days ago I was idly browsing slashdot in a quite moment. I was reading an article about efforst of Novell to provide a stable kernel level API for device drivers.




The discussion quickly devolved into the usual closed source vs open source drivers flamewar, but then I stumbled upon the following 'pearl of wisdom'.




Let's get philosophical. Why does this problem exist? Has it ever occured to any of you that device drivers are themselves a complete throwback and an obomination in the 21st Century? We take them for granted, but there is no good reason whatsoever for a computer peripheral to use "device drivers". None.

If you can't design a piece of hardware that works through an existing standardised interface you're no kind of engineer at all. And take your pick.. firewire, USB, RS232, SCSI…

Do you suppose every video display, digital camera, audio converter and so on is somehow uniquely special, that it is so ground breaking in its design that it needs custom crafted code just to make it work?

We are so entrenched in our legacy thinking that nobody, not even smart developers ever ask themselves the obvious paradigm breaking question, why the hell should you need a device driver? The reason is no more than a gross failure of modern computing, a failure of standardisation, a failure of coordination and regulation. It is a failure of ourselves as users and customers to demand a higher standard of compatibility. It is a failure of us as developers and coders to solve a simple problem once and move on.

Before you answer with some circular reasoning that merely begs the question take five and think it through. I speak as a software and hardware engineer who has designed and built entire computer systems and written an operating system.




This idea is naive and extremely shortsighted to say the least. The only way this would work is if every conceivable device is accounted for in the hardware bus protocol specifications.




For some devices this is already done. USB mass storage for example is a good example of OS transparent device communications. As far as simple storage is concerned, no device drivers are needed.




But that is as far as it goes. Consider the next piece of hardware a simple USB to serial port converter. These devices are standard today, but the no-device-driver approach would only work if this device was foreseen in the USB protocol specifications.




But even if it was, some of these devices can double as a 8 bit digital IO port. This can be really convenient. However, what would be needed to make them work? Either a device driver or an extension to the USB protocol that would already be bloated.




And even if that could be done, suppose my company wanted to add a feature, like runtime switching between different serial modes (RS232 vs RS485). What would I have to do? Beg the USB consortium to add that feature to the protocol spec? Bribe them? And they would have to get a committee agreement by the time my company wants to release the new product.




But even then, how would the software know about the exact device capabilities? Remember, there would be no device drivers. So the OS would have to provide numerous abstraction layers to account for all options.




But hey, the OS vendor has to put out new version for each new feature I want to add. So they had better have a daily release cycle to put out new versions for all the new features that all manufacturers think of. That would be so impossible it would not happen at all. Or very slowly and limited at best.




Back to the glorious days of DOS. Where all device support was done by the applications themselves. That has to be an improvement over the abstract device interfaces that are only made possible by device drivers.




NOT.




You would think that a system engineer with hardware and software experience would have a feeling for these issues, or at least a sense of nuance before making statements like this. Especially if he had written an entire operating system by himself, don't you think?




Fun with template classes: Manipulating LabVIEW arrays in C++, part 2

This article is a follow-up on a previous article: 'Manipulating LabVIEW arrays in C++' in which I’ve explained how you could use C++ template classes to work with LabVIEW arrays. There was one annoying issue left to solve: you couldn’t use the standard syntax for indexing in a multidimensional array.

I.e. the following code would not work:

  for(int row=0; row<requestCountRows; row++)

    for(int column=0; column<requestCountColumns; column++)

      Array[row][column] = i++;

After some thinking I was able to redesign the CArrayHandlePtr class to be able to be used like that.

The general idea

The idea behind my solution is to let the [x] operator for an n dimensional array return a new instance of the CArrayHandlePtr class that represents a class with n-1 dimensions of which the first element points to the first element of the subset pointed to by the [x] operator.

In that case, the operation Array[row][column] = 5 is equivalent to ((Array[row]) [column] = 5;

Array[row] returns a temporary 1D array temp. temp is then indexed like this: temp[column]. Since temp is a 1D array, it returns a reference to a specific element.

This approach is scalable to as many dimensions as you want, because a multidimensional index statement is simply broken down into n -1 temporary elements, each of which has a number of dimensional which has 1 dimension les than the previous one.

The implementation problem

The problem with this approach is that LabVIEW arrays do no lend themselves directly to it. The data block is supposed to begin directly after the dimension sizes. To return an array class of a lesser number of dimensions would mean that the selected subset would have to be preceded by the dimension sizes. This is clearly not an option.

To solve this problem we need array helper classes.

The helper classes

The point of a helper class is to be able to target a subset of a multidimensional array. To do this they should have a independent dimension size specifiers and data pointer.

Apart from that, the helper class has 1 simple requirement: the [] operator has to return a new helper of 1 dimension less than itself. If its number of dimensions is 1, it has to return a reference to the specified element.

The helper classes look like this:

template <class T, int Q=1> class CArrayHelper

{

private:

  int32* m_Sizes;

  T* m_Data;

public:

  CArrayHelper(T* Data, int32 *Sizes)

  {

    m_Data = Data;

    m_Sizes = Sizes;

  }

  CArrayHelper<T, Q-1> operator[](int Index)

  {

    int subsetSize = 1;

    /*calculate the size of a complete subset*/

    for(int i=1; i<Q; i++)

      subsetSize *= m_Sizes[ i];

    /*return an array helper that represents a subset of the original array*/

    return CArrayHelper<T, Q-1>(m_Data + subsetSize * Index, m_Sizes +1);

  }

};

template <class T> class CArrayHelper<T, 1>

{

private:

  int32* m_Sizes;

  T* m_Data;

public:

  CArrayHelper(T* Data, int32 *Sizes)

  {

    m_Data = Data;

    m_Sizes = Sizes;

  }

  T& operator[](int Index)

  {

    return m_Data[Index];

  }

};

As you can see, these classes are very simple. Their only purpose is to be able to provide a way to specify a subset of a multidimensional array. That way the [] operators can be concatenated to read 1 specific element.

The actual array classes

There are 2 array classes: one general one and one that is specialized for 1 dimensional arrays. The reason that the specialization is needed is that the [] operator for a multidimensional array should return an array helper class of 1 dimension less, while the [] operator for a 1 dimensional array should return a reference to an element in the array.

template <class T, int Q=1> class CArrayHandlePtr

{

public:

  /*type definition for LabVIEW array handle*/

#pragma pack(push)

#pragma pack(1)

  typedef struct

  {

    int dimSize[Q];

    T data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

#pragma pack(pop)

  h_LvArray* m_HandlePtr;

  /*resize the array*/

  int Resize(int DimSize[Q])

  {

    /*implementation omitted*/

  }

  T& operator[](int Index[Q])

  {

    /*implementation omitted*/

  }

  CArrayHelper<T, Q-1> operator[](int Index)

  {

    int subsetSize = 1;

    /*calculate the size of a complete subset*/

    for(int i=1; i<Q; i++)

      subsetSize *= (**m_HandlePtr)->dimSize[ i];

    /*return an array helper that represents a subset of the original array*/

    return CArrayHelper<T, Q-1>

      ((T*)(**m_HandlePtr)->data + subsetSize * Index,

      (int32*)(**m_HandlePtr)->dimSize +1);

  }

};

template <class T> class CArrayHandlePtr<T, 1>

{

public:

  /*type definition for LabVIEW array handle*/

  typedef struct

  {

    int dimSize;

    T data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

  h_LvArray* m_HandlePtr;

  /*resize the array*/

  void Resize(int DimSize)

  {

    /*implementation omitted*/

  }

  T& operator[](int Index)

  {

    return (**m_HandlePtr)->data[Index];

  }

};

As you can see, the generic array class allow you to use either a complete element specification via the [](int[Q]) operator, or a partial specification via the [](int) operator. The partial specification returns an array helper. One way or another, you will reach a single array element.

Using the arrays

You now have both syntax options for addressing elements inside the array. You can do it like this:

  for(int page=0; page<requestCountPages; page++)

    for(int row=0; row<requestCountRows; row++)

      for(int column=0; column<requestCountColumns; column++)

        Array[page][row][column] = i++;

Or like this:

  int index[3];

  for(int page=0; page<requestCountPages; page++)

  {

    index[0] = page;

    for(int row=0; row<requestCountRows; row++)

    {

      index[1] = row;

      for(int column=0; column<requestCountColumns; column++)

      {

        index[2] = column;

        Array[index] = i++;

      }

    }

  }

Both have the same effect.

Afterthoughts

This solution is easier to use than my first LabVIEW array implementation, because you now have both syntax options for indexing in the array.

On top of that, I now know more about partial template specialization than before. It has been a lot of fun already. For my definition of fun at least.

I am still not satisfied with this solution though. We are now using a total of 4 classes, which have between them 5 [] operators and 2 implementations of the Resize function. Any method we want to add to the array class has to be implemented in both the generic and the specific class. There is also no compile time option to enable range checking. Nor has any attention been given to indexing performance optimization.

I think I will write another follow-up article to at least solve the multiple implementations problem.

As always, the complete dll and LabVIEW code can be downloaded used under the MIT license. Enjoy. [;-)]

The MIT source code license

One thing that might not be obvious to everybody is why I choose the MIT license for the code I put up for download.

To make life easy for everyone, I decided to use a known license, so that it would be easy to use, and require too much effort of users to understand the issues. Licenses like the GPL and MIT license are well documented.

The GPL license basically says ‘You are free to do what you want with this code, as long as you distribute your code with your binaries’. The GPL guarantees freedom of the code, not of the programmer.

The MIT license basically says ‘You are free to do what you want with this code. We believe in open software, but understand that this is not always feasible’. The MIT license guarantees freedom of the programmer, not of the code.

For a professional programmer, it is not always possible to open up the code. It depends on the situation and the customer. I have been in this situation myself, and I was lucky enough to find an algorithm I needed with an MIT license.

That is why I choose freedom of the programmer over freedom of the code.

Writing articles and releasing code to allow others to learn from your experience is one way to give something back to the community as payment for the help you were given when you were still a newbie.

As such, I hope that the people who use my code in one way or another will do something back for the community when they become experienced programmers themselves.

But I am not going to tag all my code with the GPL license and force the issue. The only result of that would be that A) it would not be used by the people who really want to use it, but are not allowed to. B) people would use it, possible breaking the law while doing so, causing all sorts of legal problems.

Live and let live.

Fun with template classes: Manipulating LabVIEW arrays in C++, part 1

One of the best things about LabVIEW is its ability to call external code. This code can be used in the form of a dll, a CIN (custom compiled C code) an ActiveX object or a .NET class.

I have often created dlls (on windows) or shared object libraries (linux) to create an interface between LabVIEW and external native code.

This blog entry is about working with LabVIEW arrays in external code. This is simple as long as you create an interface that takes a pointer to the array data.

For data input this poses no problem. For data output this is also simple as long as you follow the C style: supply a valid data pointer, a buffer size and a pointer to a variable that will receive the number of elements written.

Of course, if you don’t know the number of elements in advance, your code becomes more complex, and will likely result in more external function calls and over-allocation.

To overcome this problem we need to dynamically allocate LabVIEW arrays in our external code that can be passed to LabVIEW. This is much more complex than working with a simple data pointer.

What are LabVIEW arrays?

A LabVIEW array is in fact a structure that contains a length specifier and the actual data elements. The C declaration is like this:

typedef struct

  {

    int dimSize;

    int data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

The array contains a dimension size specifier that indicates the size of the data part, and the data part itself.

The data part declaration is specified as an array with size 1. the reason for this is that the size specifier and the data are part of the same block of data, so data cannot be a pointer.

In order to be able to dereference data, it is declared as an array. The actual size is not 1, but whatever is mentioned in dimSize. It is only declared as an array of 1 byte because 0 sized arrays cannot exist in the C language.

Whenever LabVIEW has to pass a LabVIEW style array to external code, it does this by supplying a relocate able handle.

In case of multidimensional arrays the declaration is somewhat different:

typedef struct

  {

    int dimSize[NUM_DIMENSIONS];

    int data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

Each dimension has an element in the dimSize array to indicate the size of that dimension. For a 3 dimensional array, this array has 3 elements.

The data block is still declared as a 1D array.

It is also worth to note that LabVIEW structures and arrays are packed with 1-byte alignment. I.e. there is no padding between structure fields to align variables on address boundaries. This means that declarations should be done like this to insure proper packing:

#pragma pack(push)

#pragma pack(1)

  typedef struct

  {

    int dimSize[Q];

    T data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

#pragma pack(pop)

The syntax will be different for different compilers. This syntax is used with Microsoft Visual C++.

What is the problem with LabVIEW arrays?

On first glance the handles seem easy to use, but there are 2 distinct problems when you want to use them.

The first is simple you need a unique declaration for each data type and each number of dimensions for which you want to create an array.

typedef struct

  {

    int dimSize

    int data[1];

  } t_LvArrayInt1D, *p_LvArrayInt1D, **h_LvArrayInt1D;

typedef struct

  {

    int dimSize;

    double data[1];

  } t_LvArrayDbl1D, *p_LvArrayDbl1D, **h_LvArrayDbl1D;

typedef struct

  {

    int dimSize[2];

    double data[1];

  } t_LvArrayDbl2D, *p_LvArrayDbl2D, **h_LvArrayDbl2D;

This becomes tedious after a while, and it is cumbersome to maintain a whole header file of declarations for all possible types.

The second problem is that you have to manually do all the arithmetic for calculating the position in the data block. You have to have different calculations for each unique combination of data type and number of dimension size. This is very prone to errors, and even more tedious to maintain than the array declarations themselves. The standard indexing notation for multi dimensional array cannot be used because the array size can change at runtime.

The third problem is that it is impossible to represent a zero length array. Zero length C style arrays do not exist. Of course the LabVIEW memory manager programmers could have implemented a hack for this, but they didn’t.

This means that if you want to resize arrays for output, you need to create an interface that takes a pointer to a handle. This way you can allocate a new array if the handle is NULL, or you can resize it if it isn’t. As a result, you are now working with a pointer to a pointer to a pointer to a data block. That is a lot of dereferencing to take care of.

The fourth problem is that the code for resizing and allocating, like the code for indexing in the arrays is unique per data type and per dimension size.

C++ template classes to the rescue

Luckily for us, there is a way around all these issues: C++ template classes.

C++ template classes are the most complex but also most powerful feature of the C++ language. It allows you to define templates for classes and functions that are only specialized for specific types at compile time.

LabVIEW arrays of any type and number of dimensions can be implemented using the following class:

template <class T, int Q=1> class CArrayHandlePtr

{

public:

  /*type definition for LabVIEW array handle*/

  #pragma pack(push)

  #pragma pack(1)

  typedef struct

  {

    int dimSize[Q];

    T data[1];

  } t_LvArray, *p_LvArray, **h_LvArray;

  #pragma pack(pop)

  h_LvArray* m_HandlePtr;

  /*resize the array*/

  void Resize(int DimSize[Q])

  {

    /*implementation omitted*/

  }

  T& operator[](int Index[Q])

  {

    /*implementation omitted*/

  }

};

T and Q have to be supplied at compile time. T is the data type of the elements in the array, and Q is the number of dimensions. The compiler will create a correct implementation of t_LvArray that represents the LabVIEW style array.

The class has 2 other functions: a resize functions that allows you to resize the array (or to allocate it if the array handle was NULL), and a [] operator that returns a reference to the element specified in the Index array.

The constructor trick

The template class does not have a constructor. The reason for this is that there is no point in using true LabVIEW arrays except as function arguments to external code.

This means that the array handle pointer will be initialized by LabVIEW. The class has only 1 member variable: the handle pointer itself. If that is supplied by LabVIEW, we change its meaning from a simple address value to a templacized class instance.

This might seem a bit dodgy, but there is no constructor code that needs to be run. If the address member is initialized, the object is considered to be properly constructed. This is entirely legal C++ code.

The only downside is that we have to be sure that the definition of the data type in LabVIEW has to match what we think it to be. There is no way for the external code to know if the array is 2D or 3D for example. It is the responsibility of the programmer to insure this.

Of course, if you decide you want to use true LabVIEW arrays within your C++ code for internal purposes, you can add a constructor that will allocate memory through the LabVIEW memory manager. Just be sure to dispose of it afterwards, and to provide proper copy constructors to prevent heap errors.

Resizing the array

The implementation for the resize function is this:

  int Resize(int DimSize[Q])

  {

    int totalSize = 1;

    int errorCode = 0;

    /*calculate number of elements*/

    for(int i=0; i<Q; i++)

      totalSize *= DimSize[ i];

    /*convert to number of bytes*/

    totalSize *= sizeof(T);

    /*add size of the dimension array*/

    totalSize += Q * sizeof(int32);

    if(0 >= totalSize)

      return mgArgErr;

    if(NULL != *m_HandlePtr)

    {

      errorCode = DSSetHSzClr(*m_HandlePtr, totalSize);

      if(noErr != errorCode)

        return errorCode;

    }

    else

    {

      *m_HandlePtr = (h_LvArray)DSNewHClr(totalSize);

      if(NULL == *m_HandlePtr)

        return mZoneErr;

    }

    for(int i=0; i<Q; i++)

      (**m_HandlePtr)->dimSize[ i] = DimSize[ i];

    return noErr;

  }

TotalSize is the size in bytes of the complete memory block, including the dimension size specifiers. Some extra checking is done to insure that the total size is a positive number.

Then the handle is resized if it was a valid handle, or a new handle is allocated if it was a NULL handle. Finally, the dimension sizes have to be filled in manually because the memory manager does not know this information.

Indexing in the array

The final functionality that has to be implemented is the [] operator that will allow the programmer to read and write values to and from a position in the array.

T& operator[](int Index[Q])

  {

    int index = 0;//sizeof(t_LvArray);

    for(int i=0; i<Q-1; i++)

    {

      int subSize = 1;

      for(int j=1; j<Q; j++)

        subSize *= Index[ i] * (**m_HandlePtr)->dimSize[ j];

      index += subSize;

    }

    index += Index[Q-1];

    return (**m_HandlePtr)->data[index];

  }

};

Because the real data block is just a flat 1D array, we have to calculate the position in the array that matches with the specified indices. This calculation is not so hard to understand:

With each dimension that is accounted for, the index in that dimension is multiplied with the total number of elements in a sub-dimension. This value (the subSize value) is calculated for each dimension except for the last dimension (the least significant dimension). That index is just added to the intermediate total because it has no number of elements per sub dimension.

The resulting index is the position in the 1D data block that represents the element at the specified multidimensional indices.

Using the array

Using the array is very simple, as can be seen in this example:

int fun2Ddouble(

                              int requestCountRows,

                              int requestCountColumns,

                              CArrayHandlePtr<double, 2> Array)

{

  int index[2] = {0,0};

  int size[2] = {requestCountRows, requestCountColumns};

  Array.Resize(size);

 

  double i=0;

  for(int row=0; row<requestCountRows; row++)

    for(int column=0; column<requestCountColumns; column++)

    {

      index[0] = row;

      index[1] = column;

      Array[index] = i++;

    }

  return 0;

}

Limitations of this design

Using the magic of C++ templates, we are able to implement 1 class that can represent all LabVIEW arrays of any given size or data type.

That does not mean that we are totally satisfied however. The follow

It is regrettable that we have to use arrays as index to the [] operator. It would be nicer to be able to use true multidimensional array syntax. i.e:

  for(int row=0; row<requestCountRows; row++)

    for(int column=0; column<requestCountColumns; column++)

      Array[row][column] = i++;

See the next chapter for some more info on that.

Another ‘problem’ if you wish, is that the [] operator does not protect against buffer overflow. This is trivial to implement, but I have not done so for performance reasons. And after all, true C style arrays do not protect against it either. It would perhaps be a good idea to implement this using conditional compilation, enabling the programmer to turn this feature on or off at compile time. This is just proof of concept code however, so I am not going to do that for now.

A performance problem is also that the sub dimension sizes are recalculated for all dimensions, each time the [] operator is invoked. It would make more sense to store these values somewhere, but we cannot do that. The requirement that an instance of the template class is 100 % identical with the LabVIEW definition means that we are not allowed to store additional data in the class body.

Afterthoughts

This article explains how you can use C++ template classes to create 1 class that you can use to access LabVIEW arrays of any data type and number of dimension. Bear in mind that the code is just proof of concept code.

The example LabVIEW and C++ code is available for download under the MIT license.

I have already found a solution of the multidimensional syntax problem. With some additional programming it is possible to use the [row][column] notation for 2D arrays, and [page][row][column] notation for 3D arrays.

The solution is not limited by the number of dimensions (you could use 100D arrays for all I care) but it is a bit too complex to include it as part of this article.

I will dedicate my next blog article to the solution for that problem.

Interop between C++ and C or LabVIEW

One of the questions that recur from time to time is: ‘How do I use C++ classes in a language that has only C bindings, like C itself or LabVIEW?'




The short answer: you can’t.




The long answer: you can’t, but you can wrap the C++ code in C functions, and export those in a wrapper dll, thus allowing you to manipulate C++ objects through a C interface.




The simple solution




The general way to do this is to export a function for creating objects that returns the object pointer to the caller. Another function will take that pointer to do something with it, and a third function will take the pointer to delete it.






You have to cast the pointer to something that can be recognized by the caller. You cannot export a class pointer, because C or LabVIEW have no idea what a class is. To be safe, you have to cast it to an INT_PTR to insure that it is always large enough to hold a pointer.




 




#define OBJPTR INT_PTR






The 3 essential functions look like this:














int simple_CreateObject(OBJPTR *ObjPtr)
{
  *ObjPtr = (OBJPTR) new SomeClass();
  if(NULL == *ObjPtr)
    return 1;








  return NO_ERROR;
}
















int simple_UseObject(OBJPTR ObjPtr)
{
  return ((SomeClass*)ObjPtr)->DoSomething();
}



















int simple_DeleteObject(OBJPTR ObjPtr)
{
  delete (SomeClass*)ObjPtr;
  return NO_ERROR;
}







This approach is simple and it works. It has at least one serious problem though: the application will crash if it accidentally asks the wrapper to delete the same pointer twice, or to dereference an already deleted pointer.







Especially if you develop this wrapper for use by LabVIEW, I consider it bad taste if you do not shield the LabVIEW program against crashes caused by trivial mistakes.

A better solution




A better solution is to map all object pointers to a unique integer value. The calling program will only receive the integer values that it can supply to other functions to manipulate the C++ object.







The other wrapper functions have to find the pointer that is mapped to that integer value. If there is no mapping, it can simply return an error code instead of crashing the application.







When the object is deleted it is removed from the map so that the application cannot use it anymore.







The integer reference is incremented with each object that is created so that there will never be a duplicate reference.







map<BETTER_OBJ_REF, SomeClass*> g_PtrMap;




int g_Index = 0;














int
better_CreateObject(BETTER_OBJ_REF *ObjRef)
{
  SomeClass* ptr = NULL;

  ptr = new SomeClass();







  if(NULL == ptr)
    return 1;



















  if(NO_ERROR == errorCode)
  {
    g_PtrMap[++g_Index] = ptr;
    *ObjRef = g_Index;
  }










  return NO_ERROR;
}














int
better_UseObject(BETTER_OBJ_REF ObjRef)
{
  map<BETTER_OBJ_REF, SomeClass*>::iterator iter;
  iter = g_PtrMap.find(ObjRef);








  if(iter != g_PtrMap.end())
    return (iter->second)->DoSomething();










  else
    return 1;
}
















int better_DeleteObject(BETTER_OBJ_REF ObjRef)
{
  map<BETTER_OBJ_REF, SomeClass*>::iterator iter;
  iter = g_PtrMap.find(ObjRef);



















  if(iter != g_PtrMap.end())
  {
    SomeClass* ptr = iter->second;
    g_PtrMap.erase(iter);
    delete ptr;




    return NO_ERROR;




  }







  else
   
return 1;




}




 




Afterthoughts




These 2 examples are proof of concept examples to demonstrate how to use C++ objects in any language that has C bindings.







This is not production quality code though, because of the following reasons:







  • It does not catch C++ exceptions that could be thrown somewhere in the C++ code.
  • It does not properly handle the overflow of the g_Index variable after 2^32-1 object allocations.
  • There is no synchronization of access to the global map. This means the better_xxx functions are not safe for multithreaded usage.
  • There is no NULL pointer checking in the simple_xxx functions yet.






You can download the demo project with C++ and LabVIEW code. It is licensed under the MIT license so you can use it any way you want.