Is Your “C++11 from Scratch” Course Still Valid Today? Yes, Absolutely!

I’m very proud of my “C++11 from Scratch” course published by Pluralsight.

We are in 2018, and there have been C++14 and C++17 in the meantime. So, a legit question is: “Does it make sense for me to watch your C++11 course today for a beginner-oriented introduction to C++?” And the answer is a BIG STRONG YES! 😊

In fact, in that course you will learn modern C++ topics that are valid in both C++11, and also in next versions of the language. For example, what you will learn about the parameter passing rules, like passing by reference vs. passing by value, is perfectly valid in C++11, C++14, and C++17 as well.

Moreover, the practical introductions I gave to standard library’s classes like std::string, or std::vector, and to the std::sort algorithm, just to name a few, are totally valid also in C++14 and C++17.

Similarly, my discussions on defining custom types, constructors, destructor, the RAII pattern and the scope-based lifetime of objects are still valid in C++14 and C++17, as well.

Maybe a better title for that course would be “Modern C++ from Scratch”. Anyway, the content is already there, available for an enjoyable learning experience, with a mix of slides containing interesting visuals, and demo code.

Sample slide: Introducing the std::string class
Sample slide: Introducing the std::string class

And, if you are already familiar with C++11, you may enjoy my follow-up course on “Practical C++14 and C++17 Features”.

Happy learning!

 

ATL::CStringW vs. std::wstring Performance: String Sorting and Comparisons

According to previous measurements, std::wstring performs better than ATL::CStringW:

  1. in all string concatenation tests
  2. when sorting string vectors that are made by small strings, thanks to std::basic_string’s SSO

So, I focused my attention on the string vector sorting scenario, and it seems to me that (at least in the VS2015 implementation), the slowdown of (non-SSO) wstrings is caused by wmemcmp calls, that are used to compare wstrings when sorting the string vectors. On the other hand, CStringW invokes wcscmp, that seems to run faster.

In fact, invoking std::sort with a custom comparator function that calls wcscmp to compare the C-style pointers returned by wstring::c_str, results in faster vector<wstring> sorting times. So, in this case wstring sorting performs better than CStringW even for non-SSO strings.

However, as Stephan T. Lavavej pointed out, std::basic_string supports embedded nulls, so wstring cannot use wcscmp (that works only for C-style null-terminated strings, without embedded nulls).

 

String Performance Tests: ATL vs. STL

I wrote some C++ code to test the performance of the ATL CStringW class vs. the C++ Standard Library’s std::wstring.

There are several aspects that can be considered when comparing string class performance: In the aforementioned code, I tested string vector sorting and string concatenation.

The code is available in this repository on GitHub.

You can take a look at the README for further details (including my test results).

 

Limited-time Discount on Pluralsight Annual Subscriptions

I’d like to give you a heads-up that for a limited time, Pluralsight will be discounting Individual Annual Subscriptions 33% (or $100), making them only $199.

33% Off Pluralsight Annual Subscriptions – Save $100 for a Limited Time!

I encourage you to take this opportunity to save $100 on your annual subscription!

Note that if you are an existing subscriber you can take advantage of this offer as well: In fact, your current subscription will be extended for a year for $199.

New Pluralsight Course: Practical C++14 and C++17 Features

A new course of mine was published in the Pluralsight library: Practical C++14 and C++17 Features.

From the course short description:

C++14 and C++17 added many new features to the C++ language. This course will teach you practical features introduced in C++14 and C++17, that you will be able to use to write clearer, simpler, and higher-quality modern C++ code.

You can take this course to learn about practical features added in C++14 and C++17, ranging from syntactic sugar like digit separators, to more substantial features like polymorphic lambdas (this course will offer an introduction to basic lambdas as well), relaxed constexpr functions, the Chrono library with its standard-defined duration suffixes, and C++17 juice ranging from nested namespaces, variable declarations in if statements, to “constexpr if” and structured bindings, just to name a few.

Building an Italian-to-English dictionary with std::map

I discussed these topics with both slides and demo code, including showing some bugs and how to fix them.

Demo: Sorting by string length using lambdas

You can watch the course trailer and read a more detailed course description and the table of content starting from this course page.

Proper unit conversions are important!

I put the discussed features in proper context for learners who are already familiar with basic elements of C++11. For example, when I introduced C++14 std::make_unique, I also talked about smart pointers and introduced std::unique_ptr as well. If you need an introduction to basic elements of modern C++, you can take my “C++11 from Scratch” course.

Here’s some feedback from my reviewers:

You’ve done an excellent job with the animated shapes/callouts throughout the module. They really help me to follow along with the narrative explanations. [Peer Review]

The content is logically organized and chunked into bite-size clips. I also like your mix of slides and demos. [Peer Review]

This is an excellent challenge to the viewer to spot the bug in the code. [Peer Review]

Overall, a strong module that will be well-received by an intermediate audience. The explanations are clear and the concepts build on each other, making it easy to follow along. Keep up the great work! [Peer Review]

Raw owning pointers are radioactive!

Thank You

Writing and producing this course has been an interesting journey and a rewarding experience for me. There are several people who worked with me during this journey and with their contributions helped me producing this quality course. I’d like to thank my ASM (former Editor) Beth Gerard-Hess, my Production Editor Austin Crawford, my Curriculum Director Tod Gentille, my reviewers (both QA and peer), and all the Pluralsight persons who worked on this course project. Thanks also to Stephan T. Lavavej for interesting e-mail conversations that provided good food for thought.

I hope you will enjoy this new course on Practical C++14 and C++17 Features: Happy learning! 😊

 

Subtle Bug When Converting Strings to Lowercase

Suppose that you want to convert a std::string object to lowercase.

The first thing you would do is probably searching the std::string documentation for a convenient easy simple method named to_lower, or something like that. Unfortunately, there’s nothing like that.

So, you might start developing your own “to_lower” function. A typical implementation I’ve seen of such custom function goes something like this: For each character in the input string, convert it to lowercase invoking std::tolower. In fact, there’s even this sample code on cppreference.com:

// From http://en.cppreference.com/w/cpp/string/byte/tolower

std::string str_tolower(std::string s) {
    std::transform(s.begin(), s.end(), s.begin(), 
                   [](unsigned char c) { return std::tolower(c); }
                  );
    return s;
}

Well, if you try this code with something like str_tolower(“Connie”), everything seems to work fine, and you get “connie” as expected.

Now, since C++ folks like storing UTF-8-encoded text in std::string objects, in some large code base someone happily takes the aforementioned str_tolower function, and invokes it with their lovely UTF-8 strings. Fun ensured! …Well, actually, bugs ensured.

So, the problem is that str_tolower, under the hood, calls std::tolower on each char in the input string. While this works fine for pure ASCII strings like “Connie”, such code is a bug farm for UTF-8 strings. In fact, UTF-8 is a variable-width character encoding. So, there are some Unicode “characters” (code points) that are encoded in UTF-8 using one byte, while other characters are encoded using two bytes, and so on, up to four bytes. The poor std::tolower has no clue of such UTF-8 encoding features, so it innocently spits out wrong results, char by char.

For example, I tried invoking the above function on “PERCHÉ” (the last character is the Unicode U+00C9 LATIN CAPITAL LETTER E WITH ACUTE, encoded in UTF-8 as the two-byte sequence 0xC3 0x89), and the result I got was “perchÉ” instead of the expected “perché” (é is Unicode U+00E9, LATIN SMALL LETTER E WITH ACUTE). So, the pure ASCII characters in the input string were all correctly converted to lowercase, but the final non-ASCII character wasn’t.

Actually, it’s not the std::tolower function: It’s that this function was misused, invoking it in a way that the function was not designed for.

This is one of the perils of taking std::string-based C++ code that initially worked with ASCII strings, and thoughtlessly reuse it for UTF-8-encoded text.

In fact, we saw a very similar bug in a previous blog post.

So, how can you fix that problem? Well, a portable way is using the ICU library with its icu::UnicodeString class and its toLower method.

On the other hand, if you are writing Windows-specific C++ code, you can use the LcMapStringEx API. Note that this function uses the UTF-16 encoding (as almost all Windows Unicode APIs do). So, if you have UTF-8-encoded text stored in std::string objects, you first have to convert it from UTF-8 to UTF-16, then invoke the aforementioned API, and finally convert the UTF-16-encoded result back to UTF-8. For these UTF-8/UTF-16 conversions, you may find my MSDN Magazine article on “Unicode Encoding Conversions with STL Strings and Win32 APIs” interesting.

 

Maps with Case Insensitive String Keys

How to implement a map with case insensitive string keys? If you use the standard std::map associative container with std::string or std::wstring as key types, you get a case sensitive comparison by default.

If you take a look at std::map documentation, you’ll see that in addition to the key type and value type, there’s also a third template parameter that you can plug into std::map: it’s a comparison function object to sort the keys. The default option for this comparison function is std::less<Key>.

So, if you provide a custom comparison object that ignores the key string case, you can have a map with case insensitive keys:

map<string, ValueType, StringIgnoreCaseLess> myMap;

Now, the question is: What would such comparison object look like?

A Failed Approach

An approach I saw sometimes (e.g. on StackOverflow) is to use std::lexicographical_compare, comparing the strings char-by-char, after invoking tolower on each char. Basically, the idea is to compare the lowercase versions of each corresponding characters in the input strings. The code from the aforementioned SO answer follows:

// Code from: https://stackoverflow.com/a/1801913/1629821

struct ci_less
{
  // case-independent (ci) compare_less binary function
  struct nocase_compare
  {
    bool operator() (const unsigned char& c1, const unsigned char& c2) const {
      return tolower (c1) < tolower (c2); 
    }
  };
  
  bool operator() (const std::string & s1, const std::string & s2) const {
    return std::lexicographical_compare 
        (s1.begin (), s1.end (),   // source range
         s2.begin (), s2.end (),   // dest range
         nocase_compare ());       // comparison
  }
};

The problem with this code is that it doesn’t work for international strings. In fact, while for a pure ASCII string like “Connie” you can use this technique to successfully compare “Connie” with “connie” or “CONNIE”, this won’t work for strings containing international characters.

For example, consider the Italian word “perché”. The last character in “perché” is U+00E9, i.e. the ‘LATIN SMALL LETTER E WITH ACUTE’, which is encoded in UTF-8 as the hex byte sequence 0xC3 0xA9. Its uppercase form is É U+00C9 (encoded in UTF-8 as 0xC3 0x89). So, let’s assume you use the UTF-8 encoding to store your international text in std::string objects. Well, invoking tolower char by char, as implemented in the aforementioned SO answer, will fail. In fact, tolower is unable to correctly process UTF-8 sequences (at least in the Microsoft VS2015 CRT implementation I used in my tests).

A Better Approach for International Text

So, how to fix that? Well, on Windows there’s a CompareStringEx API (available since Vista, according to the MSDN documentation) that seems to work, at least in my tests with some Italian text. You can call this API passing the NORM_IGNORECASE comparison flag.

As for most Windows APIs, the Unicode encoding used for text is UTF-16. So, let’s start writing a nice C++ wrapper around this CompareStringEx C-interface API. Let’s assume that we want to compare two UTF-16 wstrings, and that they are “ordinary” strings that don’t contain embedded NULs. A possible implementation for this C++ helper function follows:

// C++ wrapper around the Windows CompareStringEx C API
inline int CompareStringIgnoreCase(const std::wstring& s1, 
                                   const std::wstring& s2)
{
    // According to the MSDN documentation, the CompareStringEx function 
    // is optimized for NORM_IGNORECASE and string lengths specified as -1.

    return ::CompareStringEx(
        LOCALE_NAME_INVARIANT,
        NORM_IGNORECASE,
        s1.c_str(),
        -1,
        s2.c_str(),
        -1,
        nullptr,        // reserved
        nullptr,        // reserved
        0               // reserved
    );
}

Now, you can simply invoke this C++ helper function inside the comparison object that will be used with std::map:

// Comparison object for std::map, ignoring string case
struct StringIgnoreCaseLess
{
    bool operator()(const std::wstring& s1, const std::wstring& s2) const
    {
        // (s1 < s2) ignoring string case
        return CompareStringIgnoreCase(s1, s2) == CSTR_LESS_THAN;
    }
};

This basically implements the condition (s1 < s2) ignoring the string case.

And, finally, you can simply plug this comparison object into a std::map with UTF-16-encoded wstring keys:

map<wstring, ValueType, StringIgnoreCaseLess> myCaseInsensitiveStringMap;

Or, using the nice C++11 alias template feature:

template <typename ValueType>
using CaseInsensitiveStringMap = std::map<std::wstring, ValueType, 
                                          StringIgnoreCaseLess>;

// Simply use CaseInsensitiveStringMap<ValueType>

You can find some compilable C++ sample code on GitHub.

Note on UTF-8 String Keys

If you really want to use UTF-8-encoded std::string keys, then you have to add some code to the comparison object, to first convert the input strings from UTF-8 to UTF-16, and then invoke CompareStringEx for the UTF-16 text comparison.

If you are working on C++ code that is already Windows platform specific, I think that choosing the UTF-16 encoding to represent international text is more “natural” (and more efficient) than converting back and forth between UTF-8 and UTF-16.

 

Detecting Unicode Space Characters

Some programmers love UTF-8 as they believe they can reuse old “ANSI” APIs with UTF-8-encoded text. UTF-8 does have some advantages (like being endian-neutral), but pretending to blindly reuse old “ANSI” APIs for UTF-8 text is not one of them.

For example: There are various space characters defined in Unicode.

You can use the iswspace function to check if a Unicode UTF-16 wide character is a white-space, e.g.:

#include <ctype.h>      // for iswspace
#include <iostream>

int main()
{
    // Let’s do a test with the punctuation space (U+2008)
    const wchar_t wch = 0x2008;

    if (iswspace(wch)) {
        std::cout << "OK.\n";
    }
}

The corresponding old “ANSI” function is isspace: Can you use it with Unicode text encoded in UTF-8? I’m open to be proven wrong, but I think that’s not possible.

 

Printing UTF-8 Text to the Windows Console

Let’s suppose you have a string encoded in UTF-8, and you want to print it to the Windows console.

You might have heard of the _setmode function and the _O_U8TEXT flag.

The MSDN documentation contains this compilable code snippet that you can use as the starting point for your experimentation:

// crt_setmodeunicode.c  
// This program uses _setmode to change  
// stdout to Unicode. Cyrillic and Ideographic  
// characters will appear on the console (if  
// your console font supports those character sets).  
  
#include <fcntl.h>  
#include <io.h>  
#include <stdio.h>  
  
int main(void) {  
    _setmode(_fileno(stdout), _O_U16TEXT);  
    wprintf(L"\x043a\x043e\x0448\x043a\x0430 \x65e5\x672c\x56fd\n");  
    return 0;  
}  

So, to print UTF-8-encoded text, you may think of substituting the _O_U16TEXT flag with _O_U8TEXT, and use printf (or cout) with a byte sequence representing your UTF-8-encoded string.

For example, let’s consider the Japanese name for Japan, written using the kanji 日本.

The first kanji is the Unicode code point U+65E5; the second kanji is U+672C. Their UTF-8 encodings are the 3-byte sequences 0xE6 0x97 0xA5 and 0xE6 0x9C 0xAC, respectively.

So, let’s consider this compilable code snippet that tries to print a UTF-8-encoded string:

#include <fcntl.h>
#include <io.h>
#include <stdint.h>
#include <iostream>

int main()
{
    _setmode(_fileno(stdout), _O_U8TEXT);

    // Japanese name for Japan, 
    // encoded in UTF-8
    uint8_t utf8[] = { 
        0xE6, 0x97, 0xA5, // U+65E5
        0xE6, 0x9C, 0xAC, // U+672C
        0x00 
    };

    std::cout << reinterpret_cast<const char*>(utf8) << '\n';
}

This code compiles fine. However, if you run it, you’ll get this error message:

Error when trying to print UTF-8 text to the console
Error when trying to print UTF-8 text to the console

So, how to print some UTF-8 encoded text to the Windows command prompt?

Well, it seems that you have to first convert from UTF-8 to UTF-16, and then use wprintf or wcout to print the UTF-16-encoded text. This isn’t optimal, but at least it seems to work.

 

What Is the Encoding Used by the error_code message String?

std::system_error is an exception class introduced in C++11, that is thrown by various functions that interact with OS-specific APIs. The platform-dependent error code is represented using the std::error_code class (returned by system_error::code). The error_code::message method returns an explanatory string for the error code. So, what is the encoding used to store the text in the returned std::string object? UTF-8? Some other code-page?

To answer this question, I spelunked inside the MSVC STL implementation code, and found this _Winerror_message helper function that is used to get the description of a Windows error code.

STL _Winerror_message helper function
STL _Winerror_message helper function

This function first calls the FormatMessageW API to get the error message encoded in Unicode UTF-16. Then, the returned wchar_t-string is converted to a char-string, that is written to an output buffer allocated by the caller.

The conversion is done invoking the WideCharToMultiByte API, and the CodePage parameter is set to CP_ACP, meaning “the system default Windows ANSI code page” (copy-and-paste’d from the official MSDN documentation).

I think in modern C++ code, in general it’s a good practice to store UTF-8-encoded text in std::strings. The code pages are a source of subtle bugs, as there are many of them, they can change, and you end up getting garbage characters (mojibake) when different char-strings using different code pages are mixed and appended together (e.g.  when written to UTF-8 log files).

So, I’d have preferred using the CP_UTF8 flag with the WideCharToMultiByte call above, getting a char-string containing the error message encoded as a UTF-8 string.

However, this would cause mojibake bugs for C++/Windows code that uses cout or printf to print message strings, as this code assumes CP_ACP by default.

So, my point is still that char-strings should in general use the UTF-8 encoding; but unless the Windows console and cout/printf move to UTF-8 as their default encoding, it sounds like the current usage of CP_ACP in the error message string is understandable.

Anyway, due to the use of CP_ACP in the wchar_t-to-char string conversion discussed above, you should pay attention when writing error_code::message strings to UTF-8-encoded log files. Maybe the best thing would be writing custom code to get the message string from the error code identifier, and encoding it using UTF-8 (basically invoking FormatMessage followed by WideCharToMultiByte with CP_UTF8).

Thanks to Stephan T. Lavavej, Casey Carter and Billy O’Neal for e-mail communication on this issue.