Miscellany – not security – Tales from the Crypto

Final parts of the Git move – VSTS

I’ve posted before how I’d like to get my source code out of the version control system I used to use, because it was no longer supported by the manufacturer, and into something else.

I chose git, in large part because it uses an open format, and as such isn’t going to suffer the same problem I had with ComponentSoftware’s CS-RCS.

Now that I’ve figured out how to use Bash on Ubuntu on Windows to convert from CS-RCS to git, using the rcs-fast-export.rb script, I’m also looking to protect my source control investment by storing it somewhere off-site.

This has a couple of good benefits – one is that I’ll have access to it when I’m away from my home machine, another is that I’ll be able to handle catastrophic outages, and a third is that I’ll be able to share more easily with co-conspirators.

I’m going to use Visual Studio Team Services (VSTS), formerly known as Visual Studio Online, previous to that, as Team Foundation Services Online. You can install VSTS on your own server, or you can use the online tool at <yourdomain>.visualstudio.com. If your team is smaller than five people, you can do this for free, just like you can use Visual Studio 2015 Community Edition for free. This is a great way in which Microsoft supports hobbyist developers, open source projects, college students, etc.

Where do we start?

After my last post on the topic, you have used git and rcs-fast-export.rb to create a Git repository.

You may even have done a “git checkout” command to get the source code into a place where you can work on it. That’s not necessary for our synchronisation to VSTS, because we’re going to sync the entire repository. This will work whether you are using the Bash shell or the regular Command Prompt, as long as you have git installed and in your PATH.

If you’ve actually made any changes, be sure to add and commit them to the local Git repository. We don’t want to lose those!

I’m also going to assume you have a VSTS account. First, visit the home page.

Under “Recent Projects & Teams”, click “New”.

Give it a name and a description – I suggest leaving the other settings at their default of “Agile” and “Git” unless you have reason to change. The setting of “Git” in particular is required if you’re following along, because that’s how we’re going to synchronise next.

When you click “Create project”, it’ll think for a while…

And then you’ll have the ability to continue on. Not sure my team’s actually “going to love this”, considering it’s just me!

Yes, it’s not just your eyes, the whole dialog moved down the screen, so you can’t hover over the button waiting to hit it.

Click “Navigate to project”, and you’ll discover that there’s a lot waiting for you. Fortunately a quick popup gives you the two most likely choices you’ll have for any new project.

As my team-mates will attest, I don’t do Kanban very well, so we’ll ignore that side of things, I’m mostly using this just to track my source code. So, hit “Add Code”, and you get this:

Some interesting options here

Don’t choose any yet Smile

“Clone to your computer” – an odd choice of the direction to use, since this is an empty source directory. But, since it has a “Clone in Visual Studio” button, this may be an easy way to go if you already have a Visual Studio project working with Git that you want to tie into this. There is a problem with this, however, in that if you’re working with multiple versions of Visual Studio, note that any attempt from VSTS to open Visual Studio will only open the most recently installed version of Visual Studio. I found no way to make Visual Studio 2013 automatically open from the web for Visual Studio 2013 projects, although the Visual Studio Version Selector will make the right choice if you double click the SLN file.

“Push an existing repository from command line” – this is what I used. A simple press of the “Copy to clipboard” button gives me the right commands to feed to my command shell. You should run these commands from somewhere in your workspace, I would suggest from the root of the workspace, so you can check to see that you have a .git folder to import before you run the commands.

BUT – I would strongly recommend not dismissing this screen while you run these commands, you can’t come back to it later, and you’ll want to add a .gitignore file.

The other options are:

“Import a repository” – this is if you’re already hosting your git repository on some other web site (like Github, etc), and want to make a copy here. This isn’t a place for uploading a fast-import file, sadly, or we could shortcut the git process locally. (Hey, Microsoft, you missed a trick!)

“Initialize with a README or gitignore” – a useful couple of things to do. A README.md file is associated with git projects, and instructs newcomers to the project about it – how to build it, what it’s for, where to find documentation, etc, etc – and you can add this at any time. The .gitignore file tells git what file names and extensions to not bother with putting into. Object files, executables, temporary files, machine generated code, PCH & PDB files, etc, etc. You can see the list is long, and there’s no way to add a .gitignore file with a single button click after you’ve left this page. You can steal one from an empty project, by simply copying it – but the button press is easier.

What I’ve found

I’ve found it useful to run the “git remote” and “git push” commands from the command-line (and I choose to run them from the Bash window, because I’m already there after running the RCS export), and then add the .gitignore. So, I copy the commands and send them to the shell window, before I press the “Add a .gitignore” button, choose “Visual Studio” as my gitignore type, and then select “Initialize”:

First, let’s start with a recap of using the rcs-fast-export command to bring the code over from the old RCS to a new Git repository:

Commands in that window:

cd workspace/
mkdir Juggler
cd Juggler
../rcs-fast-export.rb -A ../AuthorsFile /mnt/c/RCS/c/stress/Juggler > Juggler.gx
git init
git fast-import < Juggler.gx

Commands:

git reset

No commands – we’ve imported and are ready to sync up to the VSTS server.

Commands (copied from the “Add Code” window):

git remote add origin .visualstudio.com/DefaultCollection/_git/Juggler">https://<yourdomain>.visualstudio.com/DefaultCollection/_git/Juggler
git push -u origin –all
[Then you have to logon with the credentials for your VSTS account.]
[And finally, select your .gitignore from the “Add Code” window, and hit “Initialize” – this is optional, but will prevent you from syncing object files, etc]

But that’s not quite all…

Your solution still has lines in it dictating what version control you’re using. So you want to unbind that.

[If you don’t unbind existing version control, you won’t be able to use the built-in version control features in Visual Studio, and you’ll keep getting warnings from your old version control software. When you uninstall your old version control software, Visual Studio will refuse to load your projects. So, unbinding your old version control is really important!]

I like to do that in a different directory from the original, for two reasons:

I don’t want to overwrite or delete the working workspace I’ve been working in until the new workspace works. So I still have the old directory to work from if I need to, while I’m moving to the new place.
I want to make sure that a developer (even if it’s just me six months from now, after I’ve wiped everything in a freak electromagnet accident) can connect to this version control source, and build everything.

So, now it’s Command Prompt window time…

Yes, you could do that from Visual Studio, but it’s just as easy from the command line. Note that I didn’t actually enter credentials here – they’re cached by Windows.

Commands entered in that window:

md workspace/Juggler
cd workspace/Juggler
git clone https://<yourdomain>.visualstudio.com/DefaultCollection/_git/Juggler .
Juggler2.sln

Your version control system may complain when opening this project that it’s not in the place it remembers being in… I know mine does. Tell it that’s OK.

[Yes, I’ve changed projects, from Juggler to EFSExt. I suddenly realised that Juggler is for Visual Studio 2010, which is old, and not installed on this system.]

Now that we’ve opened the solution in Visual Studio, it’s time to unbind the old source control. This is done by visiting the File => Source Control => Change Source Control menu option:

You’ll get a dialog that lists every project in this solution. You need to select every project that has a check-mark in the “Connected” column, and click the “Unbind” button.

Luckily, in this case, they’re already selected for me, and I just have to click “Unbind”:

You are warned:

Note that this unbinding happens in the local copy of the SLN and VCPROJ, etc files – it’s not actually going to make any changes to your version control. [But you made a backup anyway, because you’re cautious, right?]

Click “Unbind” and the dialog changes:

Click OK, and we’re nearly there…

Finally, we have to sync this up to the Git server. And to do that, we have to change the Source Control option (which was set when we first loaded the project) to Git.

This is under Tools => Options => Source Control. Select the “Microsoft Git Provider” (or in Visual Studio 2015, simply “Git”):

Press “OK”. You’ll be warned if your solution is still bound in some part to a previous version control system. This can happen in particular if you have a project which didn’t load, but which is part of this solution. I’m not addressing here what you have to do for that, because it involves editing your project files by hand, or removing projects from the solution. You should decide for yourself which of those steps carries the least risk of losing something important. Remember that you still have your files and their history in at least THREE version control systems at this point – your old version control, the VSTS system, and the local Git repository. So even if you screw this up, there’s little real risk.

Now that you have Git selected as your solution provider, you’ll see that the “Changes” option is now available in the Team Explorer window:

Save all the files (but I don’t have any open!) by pressing Ctrl-Shift-S, or selecting File => Save All.

If you skip this step, there will be no changes to commit, and you will be confused.

Select “Changes”, and you’ll see that the SLN files and VCPROJ files have been changed. You can preview these changes, but they basically are to do with removing the old version control from the projects and solution.

It wants a commit message. This should be short and explanatory. I like “Removed references to old version control from solution”. Once you’ve entered a commit message, the Commit button is available. Click it.

It now prompts you to Sync to the server.

So click the highlighted word, “Sync”, to see all the unsynced commits – you should only have one at this point, but as you can imagine, if you make several commits before syncing, these can pile up.

Press the “Sync” button to send the commit up to the server. This is also how you should usually get changes others have made to the code on the server. Note that “others” could simply mean “you, from a different computer or repository”.

Check on the server that the history on the branch now mentions this commit, so that you know your syncing works well.

And you’re done

Sure, it seems like a long-winded process, but most of what I’ve included here is pictures of me doing stuff, and the stuff I’m doing is only done once, when you create the repository and populate it from another. Once it’s in VSTS, I recommend building your solution, to make sure it still builds. Run whatever tests you have to make sure that you didn’t break the build. Make sure that you still have valid history on all your files, especially binary files. If you don’t have valid history on any files in particular, check the original version control, to see if you ever did have. I found that my old CS-RCS implementation was storing .bmp files as text, so the current version was always fine, but the history was corrupted. That’s history I can’t retrieve, even with the new source control.

Now, what about those temporary repositories? Git makes things really easy – the Git repository is in a directory off the root of the workspace, called “.git”. It’s hidden, but if you want to delete the repository, just delete the “.git” folder and its contents. You can delete any temporary workspaces the same way, of course.

I did spend a little time automating the conversion of multiple repositories to Git, but that was rather ad-hoc and wobbly, so I’m not posting it here. I’d love to think that some of the rest of this could be automated, but I have only a few projects, so it was good to do by hand.

Final Statement

No programmer should be running an unsupported, unpatched, unupdated old version control system. That’s risky, not just from a security perspective, but from the perspective that it may screw up your files, as you vary the sort of projects you build.

No programmer should be required to drop their history when moving to a new version control system. There is always a way to move your history. Maybe that way is to hire a grunt developer to fetch versions dated at random/significant dates throughout history out of the old version control system, and check them in to the new version control system. Maybe you can write automation around that. Or maybe you’ll be lucky and find that someone else has already done the automation work for you.

Hopefully I’ve inspired you to take the plunge of moving to a new version control system, and you’ve successfully managed to bring all your precious code history with you. By using Visual Studio Team Services, you’ve also got a place to track features and bugs, and collaborate with other members of a development team, if that’s what you choose to do. Because you’ve chosen Git, you can separate the code and history at any time from the issue tracking systems, should you choose to do so.

Let me know how (if?) it worked for you!

Got on with Git

In which I move my version control from ComponentSoftware’s CS-RCS Pro to Git while preserving commit history.

[If you don’t want the back story, click here for the instructions!]

OK, so having watched the video I linked to earlier, I thought I’d move some of my old projects to Git.

I picked one at random, and went looking for tools.

I’m hampered a little by the fact that all my old projects used ComponentSoftware’s “CS-RCS Pro”.

Why did you choose CS-RCS Pro?

A couple of really good reasons:

It works on Windows
It integrates moderately well with Visual Studio through the VSS functionality
It’s compatible with GNU RCS, which I had some familiarity with
It was free if you’re the only dev on your projects

But you know who doesn’t use CS-RCS Pro any more?

That’s right, ComponentSoftware.

It’s a dead platform, unsupported, unpatched, and belongs off my systems.

So why’s it still there?

One simple reason – if I move off the platform, I face the usual choice when migrating from one version control system to another:

Carry all my history, so that I can review earlier versions of the code (for instance, when someone says they’ve got a new bug that never happened in the old version, or when I find a reversion, or when there’s a fix needed in one area of the code tree that I know I already made in a different area and just need to copy)
Lose all the history by starting fresh with the working copy of the source code

The second option seems a bit of a waste to me.

OK, so yes, technically I could mix the two modes, by using CS-RCS Pro to browse the ancient history when I need to, and Git to browse recent history, after starting Git from a clean working folder. But I could see a couple of problems:

Of course the bug I’m looking through history for is going to be across the two source control packages
It would mean I still have CS-RCS Pro sitting around installed, unpatched and likely vulnerable, on one of my dev systems

So, really, I wanted to make sure that I could move my files, history and all.

What stopped you?

I really didn’t have a good way to do it.

Clearly, any version control system can be moved to any other version control system by the simple expedient of:

For each change X:
- Set the system date to X’s date
- Fetch the old source control’s files from X into the workspace
- Commit changes to the new source control, with any comments from X
- Next change

But, as you can imagine, that’s really long-winded and manual. That should be automatable.

In fact, given the shared APIs of VSS-compatible source control services, I’m truly surprised that nobody has yet written a tool to do basically this task. I’d get on it myself, but I have other things to do. Maybe someone will write a “VSS2Git” or “VSS2VSS” toolkit to do just this.

There is a format for creating a single-file copy of a Git repository, which Git can process using the command “git fast-import”. So all I have to find is a tool that goes from a CS-RCS repository to the fast-import file format.

Nobody uses CS-RCS Pro

So, clearly there’s no tool to go from CS-RCS Pro to Git. There’s a tool to go from CS-RCS Pro to CVS, or there was, but that was on the now-defunct CS-RCS web site.

But… Remember I said that it’s compatible with GNU RCS.

And there’s scripts to go from GNU RCS to Git.

What you waiting for? Do it!

OK, so the script for this is written in Ruby, and as I read it, there seemed to be a few things that made it look like it might be for Linux only.

I really wasn’t interested in making a Linux VM (easy though that may be) just so I could convert my data.

So why are you writing this?

Everything changed with the arrival of the recent Windows 10 Anniversary Update, because along with it came a new component.

Bash on Ubuntu on Windows.

It’s like a Linux VM, without needing a VM, without having to install Linux, and it works really well.

With this, I could get all the tools I needed – GNU RCS, in case I needed it; Ruby; Git command line – and then I could try this out for myself.

Of course, I wouldn’t be publishing this if it wasn’t somewhat successful. But there are some caveats, OK?

Here’s the caveats

I’ve tried this a few times, on ONE of my own projects. This isn’t robustly tested, so if something goes all wrong, please by all means share, and people who are interested (maybe me) will probably offer suggestions, some of them useful. I’m not remotely warrantying this or suggesting it’s perfect. It may wipe your development history out of your one and only copy of version control… so don’t do it on your one and only copy. Make a backup first.

GNU RCS likes to store files in one of two places – either in the same directory as the working files, but with a “,v” pseudo-extension added to the filename, or in a sub-directory off each working folder, called “RCS” and with the same “,v” extension on the files. If you did either of these things, there’s no surprises. But…

CS-RCS Pro doesn’t do this. It has a separate RCS Repository Root. I put mine in C:\RCS, but you may have yours somewhere else. Underneath that RCS Repository Root is a full tree of the drives you’ve used CS-RCS to store (without the “:”), and a tree under that. I really hope you didn’t embed anything too deep, because that might bode ill.

Initially, this seemed like a bad thing, but because you don’t actually need the working files for this task, you can pretend that the RCS Repository is actually your working space.

Maybe this is obvious, but it took me a moment of thinking to decide I didn’t have to move files into RCS sub-folders of my working directories.

Make this a “flag day”. After you do this conversion, never use CS-RCS Pro again. It was good, and it did the job, and it’s now buried in the garden next to Old Yeller. Do not sprinkle the zombification water on that hallowed ground to revive it.

This also means you MUST check in all your code before converting, because checking it in afterwards will be … difficult.

Enough already, how do we do this?

Assumption: You have Windows 10.

Install Windows 10 Anniversary Update – this is really easy, it’s an update, you’ve probably been offered it already, and you may even have installed it. This is how you’ll know you have it:
Install Bash on Ubuntu on Windows – everyone else has written an article on how to do this, so here’s a link (I was going to link to the PC World article, but the full-page ad that popped up and obscured the screen, without letting me click the “no thanks” button persuaded me otherwise).
Run the following commands in the bash shell:
sudo apt-get update
sudo apt-get install git
sudo apt-get install ruby
[Optional] Run “sudo apt-get instal rcs”, if you want to use the GNU RCS toolset to play with your original source control tree. Not sure I’d recommend doing too much of that.
Change directory in the bash shell to a new, blank workspace folder you can afford to mess around in.
Now a long bash command, but this really simply downloads the file containing rcs-fast-export:
curl http://git.oblomov.eu/rcs-fast-export/blob_plain/c8a2bd6edbb21c1bfaf269ad1ec0e82af72c911a:/rcs-fast-export.rb -o rcs-fast-export.rb
Make it executable with the command “chmod +x rcs-fast-export.rb”
Git uses email addresses, rather than owner names, and it insists on them having angle brackets. If your username in CS-RCS Pro was “bob”, and your email address is “kate@example.com”, create an authors file with a bash command like this:
echo “bob=Kate Smith <kate@example.com>” > AuthorsFile
Now do the actual creation of the file to be imported, with this bash command:
./rcs-fast-export.rb -A AuthorsFile /mnt/c/RCS/…path-to-project… > project-name.gitexport
[Note a couple of things here – starting with “./”, because that isn’t automatically in the PATH in Linux. Your Windows files are logically mounted in drives under /mnt, so C:\RCS is in /mnt/c/RCS. Case is important. Your “…path-to-project…” probably starts with “c/”, so that’s going to look like “/mnt/c/RCS/c/…” which might look awkward, but is correct. Use TAB-completion on folder names to help you.]
Read the errors and correct any interesting ones.
Now import the file into Git. We’re going to initialise a Git repository in the “.git” folder under the current folder, import the file, reset the head, and finally checkout all the files into the “master” branch under the current directory “.”. These are the bash commands to do this:
git init
git fast-import < project-name.gitexport
git reset
git checkout master .
Profit!
If you’re using Visual Studio and want to connect to this Git repository, remember that your Linux home directory sits under “%userprofile%\appdata\local\lxss\home”

This might look like a lot of instructions, but I mostly just wanted to be clear. This is really quick work. If you screw up after the “git init” command, simply “rm –rf .git” to remove the new repository.

Capital ‘I’ Internet

The Atlantic today published a reminder that the Associated Press has declared in their style guide as of today that the word “Internet” will be spelt with a lowercase ‘i’ rather than an uppercase ‘I’.

The title is “Elegy for the Capital-I Internet”, but manages to be neither elegy nor eulogy, and misses the mark entirely, focusing as it does on the awe-inspiring size of the Internet being why the upper-case initial was important; then moving to describe how its sheer ubiquity should lead us to associating it with a lower-case i.

Here’s my take

The "Internet", capital I, gives the information that this is the only one of its kind, anywhere, ever. There is only one Internet. A lower-case I would indicate that there are several "internets". And, sure enough, there are several lower-class networks-of-networks (which is the definition of “internet” as a lower-case noun).

I’d like to inform the people who are engaging in this navel-gazing debate over big-I or small-i, that there functionally is only exactly one Internet. When their cable company came to "install the Internet", there was no question on the form to say "which internet do you want to connect to?" and people would have been rightly upset if there had been.

So, from that perspective, very much capital-I is still the right term for the Internet. There’s only one. Those other smaller internets are not comparable to “the Internet”.

From a technical perspective, we’re actually at the time when it’s closest to being true that there’s two internets. We’re in the midst of the long, long switch from IPv4 to IPv6. We’ve never done that before. And, while there are components of each that will talk to the other, it’s possible to describe the IPv6 and IPv4 collections of networks as two different "internets". So, maybe small-i is appropriate, but for none of the reasons this article describes.

Having said that, IPv6 engineers work really really hard to make sure that users just plain don’t notice that there’s a second internet while they’re building it, and it just feels exactly like it would if there was still only one Internet.

Again, you come back to "there is only one Internet", you don’t get to check a box that selects which of several internets you are going to connect to, it’s not like "the cloud", where there are multiple options. You are either connected to the one Internet, or you’re not connected to any internet at all.

The winner is…

Capital I, and bollocks to the argument from the associated press – lower-cased, because it’s not really that big or important, and neither is the atlantic. So, with their own arguments (which I believe are fallacious anyway), I don’t see why they deserve an upper-case initial.

The Atlantic, on the other hand – that’s huge and I wouldn’t want to cross it under my own steam.

And the Internet, different from many other internets, deserves its capital I as a designation of its singular nature. Because it’s a proper noun.

The blame game: It’s always never human error

The Ubuntu “Circle of Friends” logo.

Depending on the kind of company you work at, it’s either:

a group of three friends holding hands and dancing in a merry circle
a group of three colleagues each pointing at the other two to tell you who to blame
three guys tied to a pole desperately trying to escape the rising orange flood waters

If you work at the first place, reach out to me on LinkedIn – I know some people who might want to work with you.

If you’re at the third place, you should probably get out now. Whatever they’re paying you, or however much the stock might be worth come the IPO, it’s not worth the pain and suffering.

If you’re at the second place, congratulations – you’re at a regular, ordinary workplace that could do with a little better management.

What’s this to do with security?

A surprisingly great deal.

Whenever there’s a security incident, there should be an investigation as to its cause.

Clearly the cause is always human error. Machines don’t make mistakes, they act in predictable ways – even when they are acting randomly, they can be stochastically modeled, and errors taken into consideration. Your computer behaves like a predictable machine, but at various levels it actually routinely behaves like it’s rolling dice, and there are mechanisms in place to bias those random results towards the predictable answers you expect from it.

Humans, not so much.

Humans make all the mistakes. They choose to continue using parts that are likely to break, because they are past their supported lifecycle; they choose to implement only part of a security mechanism; they forget to finish implementing functionality; they fail to understand the problem at hand; etc, etc.

It always comes back to human error.

Or so you think

Occasionally I will experience these great flashes of inspiration from observing behaviour, and these flashes dramatically affect my way of doing things.

One such was when I attended the weekly incident review board meetings at my employer of the time – a health insurance company.

Once each incident had been resolved and addressed, they were submitted to the incident review board for discussion, so that the company could learn from the cause of the problem, and make sure similar problems were forestalled in future.

These weren’t just security incidents, they could be system outages, problems with power supplies, really anything that wasn’t quickly fixed as part of normal process.

But the principles I learned there apply just as well to security incident.

Root cause analysis

The biggest principle I learned was “root cause analysis” – that you look beyond the immediate cause of a problem to find what actually caused it in the long view.

At other companies, who can’t bear to think that they didn’t invent absolutely everything, this is termed differently, for instance, “the five whys” (suggesting if you ask “why did that happen?” five times, you’ll get to the root cause). Other names are possible, but the majority of the English-speaking world knows it as ‘root cause analysis’

This is where I learned that if you believe the answer is that a single human’s error caused the problem, you don’t have the root cause.

But!

Whenever I discuss this with friends, they always say “But! What about this example, or that?”

You should always ask those questions.

Here’s some possible individual causes, and some of their associated actual causes:

Bob pulled the wrong lever	Who trained Bob about the levers to pull? Was there documentation? Were the levers labeled? Did anyone assess Bob’s ability to identify the right lever to pull by testing him with scenarios?
Kate was evil and did a bad thing	Why was Kate allowed to have unsupervised access? Where was the monitoring? Did we hire Kate? Why didn’t the background check identify the evil?
Jeremy told everyone the wrong information	Was Jeremy given the right information? Why was Jeremy able to interpret the information from right to wrong? Should this information have been automatically communicated without going through a Jeremy? Was Jeremy trained in how to transmute information? Why did nobody receiving the information verify it?
Grace left her laptop in a taxi	Why does Grace have data that we care about losing – on her laptop? Can we disable the laptop remotely? Why does she even have a laptop? What is our general solution for people, who will be people, leaving laptops in a taxi?
Jane wrote the algorithm with a bug in it	Who reviews Jane’s code? Who tests the code? Is the test automated? Was Jane given adequate training and resources to write the algorithm in the first place? Is this her first time writing an algorithm – did she need help? Who hired Jane for that position – what process did they follow?

I could go on and on, and I usually do, but it’s important to remember that if you ever find yourself blaming an individual and saying “human error caused this fault”, it’s important to remember that humans, just like machines, are random and only stochastically predictable, and if you want to get predictable results, you have to have a framework that brings that randomness and unpredictability into some form of logical operation.

Many of the questions I asked above are also going to end up with the blame apparently being assigned to an individual – that’s just a sign that it needs to keep going until you find an organisational fix. Because if all you do is fix individuals, and you hire new individuals and lose old individuals, your organisation itself will never improve.

[Yes, for the pedants, your organisation is made up of individuals, and any organisational fix is embodied in those individuals – so blog about how the organisation can train individuals to make sure that organisational learning is passed on.]

Finally, if you’d like to not use Ubuntu as my “circle of blame” logo, there’s plenty of others out there – for instance, Microsoft Alumni:

Microsoft Alumni

A quick April Fools’ Day reminder

Tomorrow is April 1, also known as April Fools’ Day.

As a result, you should expect that anything I say on this blog is fabrication, fantasy, foolery and snark.

Apparently, this hasn’t previously been completely stupidly blindly obvious.

Leap Day again

I’ve mentioned before how much I love the vagaries of dates and times in computing, and I’m glad it’s not a part of my regular day-to-day work or hobby coding.

Here’s some of the things I expect to happen this year as a result of the leap year:

Hey, it’s February 29 – some programs, maybe even operating systems, will refuse to recognise the day and think it’s actually March 1. Good luck figuring out how to mesh that with other calendar activities. Or maybe you’ll be particularly unlucky, and the app/OS will break completely.
But the fun’s not over, as every day after February 29, until March 1 NEXT YEAR, you’re a full 366 days ahead of the same date last year. So, did you create a certificate that expires next year, last year? If so, I hope you have a reminder well ahead of time to renew the certificate, because otherwise, your certificate probably expires 365 days ahead, not one year. Or maybe it’ll just create an invalid certificate when you renew one today.
The same is true for calendar reminders – some reminders for “a year ahead” will be 365 days ahead, not one year. Programmers often can’t tell the difference between AddDays(365) and AddYears(1) – and why would they, when the latter is difficult to define unambiguously (add a year to today’s date, what do you get? February 28 or March 1?)
But the fun’s not over yet – we’ve still got December 31 to deal with. Why’s that odd? Normal years have a December 31, so that’s no problem, right? Uh, yeah, except that’s day 366. And that’s been known to cause developers a problem – see what it did to the Zune a few years back.
Finally, please don’t tell me I have an extra day and ask me what I’m going to do with it – the day, unless you got a day off, or are paid hourly, belongs to your employer, not to you – they have an extra day’s work from you this year, without adding to your salary at all.

And then there’s the ordinary issues with dates that programmers can’t understand – like the fact that there are more than 52 weeks in a year. “ASSERT(weeknum>0 && weeknum<53);”, anyone? 52 weeks is 364 days, and every year has more days than that. [Pedantic mathematical note – maybe this somewhat offsets the “employer’s extra day” item above]

Happy Leap Day – and always remember to test your code in your head as well as in real life, to find its extreme input cases and associated behaviours. They’ll get tested anyway, but you don’t want it to be your users who find the bugs.

Get on with git

Out with the old

Version control is one of those vital tools for developers that everyone has to use but very few people actually enjoy or understand.

So, it’s with no surprise that I noted a few months ago that the version control software on which I’ve relied for several years for my personal projects, Component Software’s CS-RCS, has not been built on in years, and cannot now be downloaded from its source site. [Hence no link from this blog]

Not so in with the new

I’ve used git before a few times in professional projects while I was working at Amazon, but relatively reluctantly – it has incredibly baroque and meaningless command-line options, and gives the impression that it was written by people who expected their users to be just as proficient with the ins and outs of version control as they are.

While I think it’s a great idea for developers to build software they would use themselves, I think it’s important to make sure that the software you build is also accessible by people who aren’t the same level of expertise as yourself. After all, if your users were as capable as the developer, they would already have built the solution for themselves, so your greater user-base comes from accommodating novices to experts with simple points of entry and levels of improved mastery.

git, along with many other open source, community-supported tools, doesn’t really accommodate the novice.

As such, it means that most people who use it rely on “cookbooks” of sets of instructions. “If you want to do X, type commands Y and Z” – without an emphasis on understanding why you’re doing this.

This leads inexorably to a feeling that you’re setting yourself up for a later fall, when you decide you want to do an advanced task, but discover that a decision you’ve made early on has prevented you from doing the advanced task in the way you want.

That’s why I’ve been reluctant to switch to git.

So why switch now?

But it’s clear that git is the way forward in the tools I’m most familiar with – Visual Studio and its surrounding set of developer applications.

It’s one of those decisions I’ve made some time ago, but not enacted until now, because I had no idea how to start – properly. Every git repository I’ve worked with so far has either been set up by someone else, or set up by me, based on a cookbook, for a new project, and in a git environment that’s managed by someone else. I don’t even know if those terms, repository and environment, are the right terms for the things I mean.

There are a number of advanced things I want to do from the very first – particularly, I want to bring my code from the old version control system, along with its history where possible, into the new system.

And I have a feeling that this requires I understand the decisions I make when setting this up.

So, it was with much excitement that I saw a link to this arrive in my email:

Next thing is I’m going to watch this, and see how I’m supposed to work with git. I’ll let you know how it goes.

Auto convert inked shapes in PowerPoint–coming to OneNote

I happened upon a blog post by the Office team yesterday which surprised me, because it talked about a feature in PowerPoint that I’ve wanted ever since I first got my Surface 2.

Shape recognition

Here’s a link to documentation on how to use this feature in PowerPoint.

https://support.office.com/en-us/article/use-a-pen-to-draw-write-or-highlight-text-on-a-windows-tablet-6d76c674-7f4b-414d-b67f-b3ffef6ccf53

It seems like the obvious feature a tablet should have.

Here’s a video of me using it to draw a few random shapes:

But not just in PowerPoint – this should be in Word, in OneNote, in Paint, and pretty much any app that accepts ink.

And at last, OneNote

So here’s the blog post from Office noting that this feature will finally be available for OneNote in November.

https://blogs.office.com/2015/10/20/onenote-partners-with-fiftythree-to-support-pencil-and-paper-plus-shape-recognition-coming-soon/

On iPad, iPhone and Windows 10. Which I presume means it’ll only be on the Windows Store / Metro / Modern / Immersive version of OneNote.

That’s disappointing, because it should really be in every Office app. Hell, I’d update from Office 2013 tomorrow if this was a feature in Office 2016!

Let’s not stop there

Please, Microsoft, don’t stop at the Windows Store version of OneNote.

Shape recognition, along with handwriting recognition (which is apparently also hard), should be a natural part of my use of the Surface Pen. It should work the same across multiple apps.

That’s only going to happen if it’s present in multiple apps, and is a documented API which developers – of desktop apps as well as Store apps – can call into.

Well, desktop apps can definitely get that.

How can I put it into my own app?

I’ll admit that I haven’t had the time yet to build my own sample, but I’m hoping that this still works – there’s an API called “Ink Analysis”, which is exactly how you would achieve this in your app:

https://msdn.microsoft.com/en-us/library/ms704040.aspx

It allows you to analyse ink you’ve captured, and decide if it’s text or a drawing, and if it’s a drawing, what kind of drawing it might be.

[I’ve marked this with the tag “Alun’s Code” because I want to write a sample eventually that demonstrates this function.]

NCSAM resume–sorry for the interruption

TL;DR – hardware problems, resuming NCSAM posts when / if I can get time.

Well, that went about as well as can be expected.

I start a month of daily posts, and the first one is all that I achieved.

Perhaps I’ve run out of readers, because nobody asked if I was unwell or had died.

No, I haven’t died, the simple truth is that a combination of hardware failures and beta testing got the better of me.

I’d signed up to the Fast Ring of Windows Insider testing, and had found that Edge and Internet Explorer both seemed to get tired of running Twitter and Facebook, and repeatedly got slower and slower to refresh, until eventually I had to quit and restart them.

Also the SP3 refused to recognise my Microsoft Band as plugged in [actually a hardware failure on the Band, but I’ll come to that another day].

Naturally, I assumed this was all because of the beta build I was using.

So, I did what any good beta tester would do. I filed feedback, and pressed the “Roll back” button.

It didn’t seem to take as long as I expected.

Uh-oh

That’s your first sign that something is seriously wrong, and you should take a backup of whatever data is left.

So I did, which is nice, because the next thing that happened is that I tried to open a Windows Store app.

It opened a window and closed immediately.

Crap.

So did every other Windows Store / Metro / Modern / Immersive app I tried.

Including Windows Store itself.

After a couple of days of dinking around with various ‘solutions’, I decided I’d reached beta death stage, and should FFR (FDISK, FORMAT and Reinstall).

First, make another backup, just because you can’t have too many copies of the data you rely on.

And … we’re good?

That should have been close to the end of the story, with me simply reinstalling all my apps and moving along.

In fact, I started that.

Then my keyboard stopped working. It didn’t even light up.

Plugging the keyboard (it’s a Surface Pro Type Cover) into another Surface (the Surface Pro Type Covers work on, but don’t properly cover, a Surface 2, which we have in my house) demonstrated that the keyboard was just fine on a different system, just not on my main system.

So, I kept a few things running by using my Bluetooth keyboard and mouse, and once I convinced myself it was worth the trip, I took my Surface Pro 3 out to the Microsoft Store in Bellevue for an appointment.

I dream of Jeannie – no, that’s creepy

Jeannie was the tech assigned to help me with my keyboard issue. Helpful and friendly, she didn’t waste time with unnecessary questions or dinking around with stuff that could already be ruled out.

She unplugged the keyboard and tried it on another system. It worked. No need to replace the keyboard.

Can she do a factory reset?

Be my guest – I made another backup before I came out to the store.

So, another quick round of FFR, and the Surface still doesn’t recognise the keyboard.

Definitely a hardware problem, and that’s the advantage of going to the Microsoft Store.

Let me get you a replacement SP3, says Jeannie, and heads out back to the stock room.

Bad news, she says on coming back, We don’t have the exact model you have (an i7 Surface Pro 3 with 256 GB of storage).

Is it OK if we get you the next model up, with twice the storage?

Only if you can’t find any way to upgrade me for free to the shiny Surface Book you have on display up front.

So, now I have a bigger Surface Pro 3

Many thanks to Jeannie for negotiating that upgrade!

But now I have to reinstall all my apps, restore all my data, and get back to functioning before I can engage in fun things like blogging.

I’ll get back into the NCSAM posts, but they’ll be more overview than I was originally planning.

Windows 10 – first impressions

I’ve updated from Windows 8.1 to Windows 10 Enterprise Insider Preview over this weekend, on my Surface Pro 3 and a Lenovo tablet. Both machines are used for software development as well as playing games, so seemed the ideal place to practice.

So here’s some initial impressions:

1. VPN still not working properly

I’ve mentioned before (ranted, perhaps) about how the VPN support in Windows 8.1 is great for desktop apps, but broken for Metro / Modern / Immersive / Windows Store apps.

Still, maybe now I’m able to provide feedback, and Windows is in a beta test phase, perhaps they’ll pay attention and fix the bugs.

2. Stuff crashes

It’s a beta, but just in case you were persuaded to install this on a production system, it’s still not release quality.

Every so often, the Edge browser (currently calling itself “Project Spartan”) will just die on you.

I’ve managed to get the “People Hub” to start exactly twice without crashing immediately.

3. Update after you install

Download the most recent version from the Insider’s page, and you still have to apply an update to the entire system before you’re actually up to date. The update takes essentially as long as the initial install.

4. Update before you install – and make a backup

Hey, it’s a beta – what did you expect?

Things will break, you’ll find yourself missing functionality, so you may need to restore to your original state. Update before you install, and fewer things will be as likely to go wrong in the upgrade.

5. Provide feedback – even about the little things

They won’t fix things you don’t provide feedback about.

OK, so maybe they also won’t fix things that you DO provide feedback on, but that’s how life works. Not everything gets fixed. Ever.

But if you don’t report issues, you won’t ever see them fixed.

6. The new People Hub is awful

The People “Hub” in Windows 10, from the couple of times I’ve managed to execute it, basically has my contacts, and can display what’s new from them in Outlook Mail.

I rather enjoy the Windows 8.1 People Hub, where you can see in one place the most recent interactions in Twitter, Facebook, LinkedIn and Skype. Or at least, that’s what it promises, even if it only actually delivers Facebook and Twitter.

7. Videos can now be deleted

It’s always possible to delete a video file, of course, but in Windows 8.1, after you’ve finished watching a video from the Videos app, you had to go find some other tool in which to do so – and hope that you deleted the right one.

In Windows 10 you can use the context menu (right click, or tap and hold) on a video to delete it from your store.

Still needs some more work – it doesn’t display subtitles / closed-captioning, it only orders alphabetically, and there’s no jumping to the letter “Q” by pressing the “Q” key, but this app is already looking very functional even for those of us who collect MP4 files to watch.

8. No Media Center

I really, really liked the Media Center. More than TiVo. We have several Media Center PCs in our house, and now we have to figure out what we’re going to do. I’m not going back to having a made-for-purpose device that can’t do computing, I want my Media Center. I’ll try some of its competitors, but it’d be really nice if Microsoft relents and puts support back for Media Center.

9. Edge / Spartan browser – awesome

Excellent HTML5 compatibility, reduced chance of being hit by third party vulnerabilities, F12 Developer Tools, and still allows me to test for XSS vulnerabilities if I choose to do so.

Pretty much what I want in a browser, although from a security standpoint, the choice to allow two third party ~~vulnerabilities~~ add-ins into the browser, Flash and Reader, seems to be begging future trouble.

Having said that, you can disable Adobe Flash in the Advanced Settings of your Spartan browser. I’m going to recommend that you do that on all your non-gaming machines. Then find out which of your web sites need it, and either fix them, or decide whether you can balance the threat of Flash with the utility of that service.

The F12 Developer Tools continue to be a very useful set of web site debugging tools, and assist me greatly in discovering and expanding on web site vulnerabilities. I personally find them easier than debugging tools in other browsers, and they have the benefit of being always installed in recent Microsoft browsers.

The “Reader” view is a nice feature, although it was present in Windows 8.1, and should be used any time you want to actually read the contents of a story, rather than wade through adverts and constant resizing of other content around the text you’re actually interested in.

9.1 XSS

Because, you know, I’m all about the XSS.

Internet Explorer has a pretty assertive XSS filter built in, and even when you turn it off in your settings, it still comes back to prevent you. I find this to be tricky, because I sometimes need to convince developers of the vulnerabilities in their apps. Firefox is often helpful here, because it has NO filters, but sometimes the behaviour I’m trying to show is specific to Internet Explorer.

Particularly, if I type a quote character into the URL in Internet Explorer, it sends a quote character. Firefox will send a %22 or %27 (double or single quotes). So, sometimes IE will trigger behaviour that Firefox doesn’t.

Sadly, although Spartan does seem to still be useful for XSS testing, the XSS filter can’t be specifically turned off in settings. I’d love to see if I can find a secret setting for this.

10. Microsoft Print to PDF

Windows has needed a PDF printer since, oh, Windows 3.1. A print driver that prompts you for a file name, and saves whatever you’re printing as a PDF file.

With Office, this kind of existed with Save as PDF. With OneNote, you could Print to OneNote, open the View ribbon, and hide the header, before exporting as a PDF. But that’s the long way around.

With Windows 10, Microsoft installed a new printer driver, “Microsoft Print to PDF”. It does what it says on the tin, allowing you to generate PDFs from anywhere that can print.

11. Tablet Mode / PC mode

I use a Surface Pro 3 as my main system, and I have to say that the reversion to a mainly desktop model of operations is nice to my eyes, but a little confusing to the hands – I don’t quite know how to manage things any more.

Sometimes I like to work without the keyboard, because the tablet works well that way. But now I can’t close apps by sliding from top to bottom, even when I’ve expanded them to full screen. Not sure how I’m supposed to do this.