Miscellany – not security

Extracting data from Word forms with XSL Transforms (XSLT)

“What’s the point,” pondered Alice, “Of getting other people to stuff things in a box, if one cannot ever get them out?”

Ok, she never did say that, but it’s the sort of thing Alice would wonder.

Particularly if she noticed how often modern businesses send around Word forms with input fields designed to be filled out by team members, only to then be manually copied into spreadsheets, databases, or other documents.

I’d put this as the second most irritating waste of document functionality.

And it doesn’t have to be this way.

What is a Word form, anyway?

FIrst, let’s look at what you get with a Word form. There really isn’t anything quite as specific a beast as a Word form. It’s just a Word document. With form fields. Form fields are places into which users can type text, check boxes, select from drop down lists, etc.

Once form fields have been put into a document, the original document author can ‘restrict’ the document such that only editing the form fields is allowed. This is usually done with a password, to make it less likely that others will edit the document beyond the form fields.

The presence of a password should not be taken to indicate that this is a security measure.

Removing the restriction can be done by guessing the password, or accessing the settings.xml inside the docx file, and changing the value of “w:enforcement” from “1” to “0”. Other methods include saving to RTF, then editing the file in a text editor before saving it as docx again.

Restricting the document is done to make it less likely that blithe nonces will return your document to you with changes that are outside of the fields you’ve provided to them, or with fields removed. This is important, because you can’t as easily extract data from a document if you don’t know where it is.

Show me

Here’s what a form looks like when it’s restricted for editing, and has a number of form field elements provided – I’ve given a text field for a person’s name, a drop-down list for their zodiac sign, and a check box for education level. This is the sort of thing you might expect a form to be really useful for collecting.

Now that you’ve sent this out to a hundred recipients, though, you want to extract the data from each form.

Secret knowledge: docx files are ZIP files containing XML files

First we’ve got to get the part of the document containing the data out. Knowing, as we do, that a docx file is just a ZIP file full of XML files, we could unzip it and go searching for the data. I’ve already done that – the data is in the file called “word/document.xml”. You could just rename the docx file to a zip file, open it in Explorer, navigate into the ‘word’ folder, and then drag the document.xml file out for handling, but that’s cumbersome, and we want an eventual automated solution.

Turning to PowerShell

Yeah, you could write this in a batch file using whatever ZIP program you’ve downloaded, it wouldn’t be that difficult, but I’m thinking about PowerShell a lot these days for my automation. Here’s code that will take a docx file and extract just the word/document.xml component into an output file whose name is provided.

# Load up the types required to handle a zip file.
Add-Type -AssemblyName System.IO.Compression.Filesystem

Function Get-DocXDocFile ($infilename, $outfilename){
$infileloc = [System.IO.Path]::Combine($pwd,$infilename)
$zip = [System.IO.Compression.ZipFile]::OpenRead($infileloc)
$zip.Entries | where { $_.FullName -eq “word/document.xml” } | foreach {
$outfileloc = [System.IO.Path]::Combine($pwd,$outfilename)
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, “$outfileloc”,$true)
}
}

By now, if you’re like me, you’ve opened up that XML file and looked into it, and decided you don’t care that much to read its entrails.

That’s OK, I did it for you.

The new-style fields are all in “w:sdt” elements, and can be found by the “w:tag” name under the “w:sdtPr” element.

Old-style fields are all in “w:fldChar” elements, and can be found by the “w:name” value under the “w:ffData” element.

In XPath, a way of describing how you find a specific element / attribute in an XML file, that’s expressed as follows:

//w:sdt/w:sdtPr/w:tag[@w:val=’Txt1Tag’]/../..

//w:fldChar/w:ffData/w:name[@w:val=’Text1′]/../..

This does assume that you gave each of your fields names or tags. But it would be madness to expect data out if you aren’t naming your fields.

If you’re handy with .NET programming, you’re probably half way done writing the code to parse this using XmlDocument.

If only there was an easy – well, easier – way to do this!

If you’re not handy with .NET programming, you might need something a little (but sadly, not a lot) easier.

Remember those XPath elements? Wouldn’t it be really cool if we could embed those into a document, and then have that document automatically expand them into their contents, so we could do that for every form file we’ve got?

Well, we can.

It’s called XSLT

Short for Extensible Stylesheet Language Transformation (which is definitely long enough to need something to be short for it), XSLT, which really has no good pronunciation because I’m never going to say something that sounds like “ex-slut” at work, XSLT is a way to turn one XML-formatted document into some kind of output.

Let’s say we’re working with the document I outlined above (and which I will forget to attach to this blog post until someone points it out). We’ve already extracted document.xml, and with the right XSL file, and a suitable XSLT command (such as the Microsoft msxml tool, or whatever works in your native environment), we can do something like this:

Maybe instead of text, you prefer something more like CSV:

I will probably forget to attach the XSL stylesheets that I used for these two transformations to this blog post.

Maybe next time we can see about building this into a tool…

Here’s the files I forgot to add: ExtractData

Posted in Alun's code, Miscellany - not security | Leave a comment

My contribution to Movember

Whether it’s “No-shave November” or “Movember”, there’s a lot of attention given this time of year to men’s health in general, and cancer in particular.

I don’t take part in either of these events, partly because I don’t like the way a beard / moustache feels, but mostly because I already spend my November extremely aware of men’s cancer issues.

So, let me tell you, on this International Men’s Day, how my right testicle tried to kill me. Warning – rude words ahead, including the dreaded “c-word”.

Here we go gathering nuts in May

A little over fifteen years ago, I was living a fantastic life.

A wife, a six-year-old son, a house in a nice suburb of Austin, working from home on my own projects, and making enough money with those projects to justify doing so.

As anyone who’s ever watched any “funny home video” shows on TV will tell you, the purpose of a six year old is to throw things at your crotch, or to swing things at your crotch, or to hit you in your crotch, or to head-butt you in your crotch.

OK, so that’s maybe not his sole purpose, but that year it seemed like this was happening more often than usual. It wasn’t, of course, but it was noticeable that I was … feeling the impact a little more keenly than usual.

It takes balls to go see a doctor

I checked, my wife checked, and we concurred – something was definitely not as it had been. I mean, everyone knows that a man’s testicles aren’t the same size and shape on each side, and I’d been blessed with a particularly disparate pair from my teenage years.

But this was something new – swelling that just increased gradually, and a firmness that was inappropriately placed.

It was time to see the doctor.

Even knowing this, and reading about how badly – and how quickly – testicular diseases can impact men, it was extraordinarily difficult to face the task of picking up the phone, calling to speak to a doctor’s [female] receptionist, and tell them exactly why I wanted to come and see the doctor. Nonetheless, I girded my loins as much as I could, swallowed hard, and made the call.

The key is to remind yourself that this is probably the fifth call that receptionist has received this week on the same topic, and that she wouldn’t be working in a doctor’s office if she weren’t ready to hear medical terms briefly describing anatomical parts. I’m surprised how quickly I came to this conclusion, given how many decades it took me to learn that when a doctor asks “so, how are you doing today?”, they actually want to hear the details, rather than “oh, fine, thanks, and you?”

“Good luck”

The doctor’s visit was quick and clinical, just what you’d hope for. A flashlight applied to the nether regions, in much the same way you might check a hen’s egg for occupants, a little uncomfortable palpation, and a quick inspection of nearby things while you have your underpants down.

“You’ve got a hydrocele,” he said, doing that thing with the rubber gloves where you snap them off, startling an already nervous patient. “A short surgery should fix that.”

Relief. Nothing quite as horrifying or scary as I had expected.

“I’ll set you up with a urologist, and we’ll get that taken care of in the next couple of weeks. Good luck.”

I’d never had a doctor wish me “good luck” before, and it quite chilled me.

You’re never ready for the C-word

I visited the urologist, got set up for surgery, and discussed plans with my wife.

It was always in the back of my head that this could be something more than merely having a little extra fluid to drain.

So we talked about the C-word. I think of it that way, because on all the forms since, this is the one word the medical establishment goes out of its way to avoid writing in full. There are long words, foreign words, culturally taboo words, and all of them are written in full on some or other medical forms. There are abbreviations, but no word more than this one results in hardened medical professionals ceding to decency and refusing to name it in full:

Cancer

You kind of guessed that was going to be the result, right?

We kind of did, too, and had discussed the idea that if there was any cancerous signs, that quite frankly I preferred being a living eunuch, if that was necessary, to being a dead, but otherwise intact, cancerous corpse. It seems such an obvious decision to make, but it’s still a very hard one to bring to bear.

And my wife did so on her own.

Because the only way to tell if the testicle looked cancerous was while I was under general anaesthetic in the operating room.

And sure enough, the doctor came out mid-surgery, while I’m away with the fairies, to talk to my wife about the situation at hand. I can only imagine how that conversation went, so I shan’t try to replay it here. I can only express how truly grateful I am that my wife gave consent to do what we had already discussed – to remove that cancerous nasty thing and send it to a lab for study.

So I woke up to a woman looking unutterably upset at the prospect that she’d had to make life-altering medical decisions, for which I have always been truly grateful. There literally isn’t a day that goes by that I wish she’d made any other choice.

And yet even to this day, it still bothers her – that’s how upsetting it is to be on the outside of this disease.

It wasn’t much fun on the inside, either, to be honest, and that’s my story which I can tell.

Time to hulk up!

This was all in the week before Thanksgiving, 2002, a year when the first movie featuring an all-CGI Incredible Hulk was being advertised on the TV.

Poor Bruce Banner, strapped to a table, unable to move, while gamma rays coursed through his body under the control of a malfunctioning computer, turning him into the hangriest super-anti-hero ever.

After a trip to San Antonio, during which I felt every pothole on the I-35 from Austin, to have Thanksgiving dinner with my inlaws, we returned home and started observational and preventive treatment as follow up for good ole “testicular C”.

First, the tattoos. I have five tattoos now, each one a single dot, in the shape of a cross.

For targetting.

I wasn’t exactly strapped to a table, but I was unable to move, while gamma rays coursed through my body, laser cross-hairs ensuring that the focused radiation hit only the right parts of my intestines. They call it radiotherapy, and when you go to an oncologist / radiologist to get radiotherapy in Austin in 2002, you sit in a waiting room surrounded by inspirational photos of Lance Armstrong. Whatever you feel about his drug use while winning the Tour de France competing against others who almost certainly used most of the same drugs themselves, he continues to be inspirational to many cancer survivors like myself, simply for having survived enough to be able to ride a bike.

Testicular cancer doesn’t travel across, it goes up – so the process is, remove the testicle, fry the intestines lightly, and monitor the chest with ongoing X-rays just to make sure. Removing the testicle is called an “orchiectomy” – true story, the orchid plant is named after testicles, because that’s what the plant’s bulbs allegedly look like. This is why testicular cancer awareness pins are orchid-coloured.

Insurance companies make me sick

One of the side effects you think of with any cancer treatment is serious nausea, and this is definitely the case with radiotherapy. It makes you feel uncomfortably unwell. American medical care being run by insurance companies, I was given leave to have fifteen anti-nausea pills. For 25 days of treatment. During which I’d need multiple pills per day.

The only thing to do – snack on saltine crackers, and where possible actually cook some meals at least for my son. Bland food was really pretty much all I could manage. To this day, he quite rightly refuses to eat chicken and rice.

Because my wife had to return to work, and was travelling as a result, I drove myself to appointments, and that’s probably my biggest mistake in all of this – the American Cancer Society offers free rides to patients attending hospital and doctor appointments, and has many other services besides. Take advantage of them, I donate to them specifically for you to use their services.

Cat scans and pregnancy tests

After that, every six months to a year, I’d get a CT scan of my abdomen, and a blood test every month. CT scans are not the most comfortable of procedures, particularly with the iodine contrast dyes.

Once in a while, the person administering the blood test would question whether the test was really for me. On my doctor’s advice, I would ask them to re-check the form. It turns out that I was basically being given a monthly pregnancy test, to ensure the cancer wasn’t coming back.

More surgeries, and more insurance

Still more surgeries were in my future over the next year – apparently, skin likes to stick to skin in unusual situations and in uncomfortable ways.

The insurance company raised our rates – presumably in line with regular price rises, but to the point where it was difficult to afford. After all, even back before the ACA, it wasn’t right to raise insurance rates just because someone got sick. However, what WAS legal back then was the ability of other insurance providers to call the cancer a pre-existing condition, and to use that as reason to either refuse to sell me a policy, or to jack up the rates. Personal insurance policies are expensive to begin with, but when you can’t shop around (or threaten to do so), you’re really out of luck.

And that’s why I took the Microsoft job, and jacked in my personal business for the most part. Because American health insurance kills the American dream more often than it deserves to.

And now, the lesson

So, the final lesson – and there always is one – is that if you are a man, aged between twenty and thirty-five, or you know someone who fits, or will fit, that description, know that it’s important to check your health – actually touch and feel your body, particularly your ‘man parts’ – on a regular basis. When things change in a way that isn’t expected, it’s really important to give your doctor a call. That week. Perhaps even that day that you notice it. The person who takes your call has heard it all before – and if you aren’t comfortable talking to them, you can actually ask to speak to a nurse, a physician’s assistant, and even specifically to a man, if that’s what you need to feel comfortable to cover this.

Your doctor will tell you if it’s important, or something not to worry about. They’ll give you advice on what to watch for in future, and wish you good luck if you need it.

Above all, don’t literally die of embarrassment.

Posted in Miscellany - not security, What my wife knows | Leave a comment

Final parts of the Git move – VSTS

I’ve posted before how I’d like to get my source code out of the version control system I used to use, because it was no longer supported by the manufacturer, and into something else.

I chose git, in large part because it uses an open format, and as such isn’t going to suffer the same problem I had with ComponentSoftware’s CS-RCS.

Now that I’ve figured out how to use Bash on Ubuntu on Windows to convert from CS-RCS to git, using the rcs-fast-export.rb script, I’m also looking to protect my source control investment by storing it somewhere off-site.

This has a couple of good benefits – one is that I’ll have access to it when I’m away from my home machine, another is that I’ll be able to handle catastrophic outages, and a third is that I’ll be able to share more easily with co-conspirators.

I’m going to use Visual Studio Team Services (VSTS), formerly known as Visual Studio Online, previous to that, as Team Foundation Services Online. You can install VSTS on your own server, or you can use the online tool at <yourdomain>.visualstudio.com. If your team is smaller than five people, you can do this for free, just like you can use Visual Studio 2015 Community Edition for free. This is a great way in which Microsoft supports hobbyist developers, open source projects, college students, etc.

Where do we start?

After my last post on the topic, you have used git and rcs-fast-export.rb to create a Git repository.

You may even have done a “git checkout” command to get the source code into a place where you can work on it. That’s not necessary for our synchronisation to VSTS, because we’re going to sync the entire repository. This will work whether you are using the Bash shell or the regular Command Prompt, as long as you have git installed and in your PATH.

If you’ve actually made any changes, be sure to add and commit them to the local Git repository. We don’t want to lose those!

I’m also going to assume you have a VSTS account. First, visit the home page.

Under “Recent Projects & Teams”, click “New”.

Give it a name and a description – I suggest leaving the other settings at their default of “Agile” and “Git” unless you have reason to change. The setting of “Git” in particular is required if you’re following along, because that’s how we’re going to synchronise next.

When you click “Create project”, it’ll think for a while…

And then you’ll have the ability to continue on. Not sure my team’s actually “going to love this”, considering it’s just me!

Yes, it’s not just your eyes, the whole dialog moved down the screen, so you can’t hover over the button waiting to hit it.

Click “Navigate to project”, and you’ll discover that there’s a lot waiting for you. Fortunately a quick popup gives you the two most likely choices you’ll have for any new project.

As my team-mates will attest, I don’t do Kanban very well, so we’ll ignore that side of things, I’m mostly using this just to track my source code. So, hit “Add Code”, and you get this:

Some interesting options here

Don’t choose any yet Smile

“Clone to your computer” – an odd choice of the direction to use, since this is an empty source directory. But, since it has a “Clone in Visual Studio” button, this may be an easy way to go if you already have a Visual Studio project working with Git that you want to tie into this. There is a problem with this, however, in that if you’re working with multiple versions of Visual Studio, note that any attempt from VSTS to open Visual Studio will only open the most recently installed version of Visual Studio. I found no way to make Visual Studio 2013 automatically open from the web for Visual Studio 2013 projects, although the Visual Studio Version Selector will make the right choice if you double click the SLN file.

“Push an existing repository from command line” – this is what I used. A simple press of the “Copy to clipboard” button gives me the right commands to feed to my command shell. You should run these commands from somewhere in your workspace, I would suggest from the root of the workspace, so you can check to see that you have a .git folder to import before you run the commands.

BUT – I would strongly recommend not dismissing this screen while you run these commands, you can’t come back to it later, and you’ll want to add a .gitignore file.

The other options are:

“Import a repository” – this is if you’re already hosting your git repository on some other web site (like Github, etc), and want to make a copy here. This isn’t a place for uploading a fast-import file, sadly, or we could shortcut the git process locally. (Hey, Microsoft, you missed a trick!)

“Initialize with a README or gitignore” – a useful couple of things to do. A README.md file is associated with git projects, and instructs newcomers to the project about it – how to build it, what it’s for, where to find documentation, etc, etc – and you can add this at any time. The .gitignore file tells git what file names and extensions to not bother with putting into. Object files, executables, temporary files, machine generated code, PCH & PDB files, etc, etc. You can see the list is long, and there’s no way to add a .gitignore file with a single button click after you’ve left this page. You can steal one from an empty project, by simply copying it – but the button press is easier.

What I’ve found

I’ve found it useful to run the “git remote” and “git push” commands from the command-line (and I choose to run them from the Bash window, because I’m already there after running the RCS export), and then add the .gitignore. So, I copy the commands and send them to the shell window, before I press the “Add a .gitignore” button, choose “Visual Studio” as my gitignore type, and then select “Initialize”:

First, let’s start with a recap of using the rcs-fast-export command to bring the code over from the old RCS to a new Git repository:

Commands in that window:

cd workspace/
mkdir Juggler
cd Juggler
../rcs-fast-export.rb -A ../AuthorsFile /mnt/c/RCS/c/stress/Juggler > Juggler.gx
git init
git fast-import < Juggler.gx

Commands:

git reset

No commands – we’ve imported and are ready to sync up to the VSTS server.

Commands (copied from the “Add Code” window):

git remote add origin .visualstudio.com/DefaultCollection/_git/Juggler">https://<yourdomain>.visualstudio.com/DefaultCollection/_git/Juggler
git push -u origin –all
[Then you have to logon with the credentials for your VSTS account.]
[And finally, select your .gitignore from the “Add Code” window, and hit “Initialize” – this is optional, but will prevent you from syncing object files, etc]

But that’s not quite all…

Your solution still has lines in it dictating what version control you’re using. So you want to unbind that.

[If you don’t unbind existing version control, you won’t be able to use the built-in version control features in Visual Studio, and you’ll keep getting warnings from your old version control software. When you uninstall your old version control software, Visual Studio will refuse to load your projects. So, unbinding your old version control is really important!]

I like to do that in a different directory from the original, for two reasons:

I don’t want to overwrite or delete the working workspace I’ve been working in until the new workspace works. So I still have the old directory to work from if I need to, while I’m moving to the new place.
I want to make sure that a developer (even if it’s just me six months from now, after I’ve wiped everything in a freak electromagnet accident) can connect to this version control source, and build everything.

So, now it’s Command Prompt window time…

Yes, you could do that from Visual Studio, but it’s just as easy from the command line. Note that I didn’t actually enter credentials here – they’re cached by Windows.

Commands entered in that window:

md workspace/Juggler
cd workspace/Juggler
git clone https://<yourdomain>.visualstudio.com/DefaultCollection/_git/Juggler .
Juggler2.sln

Your version control system may complain when opening this project that it’s not in the place it remembers being in… I know mine does. Tell it that’s OK.

[Yes, I’ve changed projects, from Juggler to EFSExt. I suddenly realised that Juggler is for Visual Studio 2010, which is old, and not installed on this system.]

Now that we’ve opened the solution in Visual Studio, it’s time to unbind the old source control. This is done by visiting the File => Source Control => Change Source Control menu option:

You’ll get a dialog that lists every project in this solution. You need to select every project that has a check-mark in the “Connected” column, and click the “Unbind” button.

Luckily, in this case, they’re already selected for me, and I just have to click “Unbind”:

You are warned:

Note that this unbinding happens in the local copy of the SLN and VCPROJ, etc files – it’s not actually going to make any changes to your version control. [But you made a backup anyway, because you’re cautious, right?]

Click “Unbind” and the dialog changes:

Click OK, and we’re nearly there…

Finally, we have to sync this up to the Git server. And to do that, we have to change the Source Control option (which was set when we first loaded the project) to Git.

This is under Tools => Options => Source Control. Select the “Microsoft Git Provider” (or in Visual Studio 2015, simply “Git”):

Press “OK”. You’ll be warned if your solution is still bound in some part to a previous version control system. This can happen in particular if you have a project which didn’t load, but which is part of this solution. I’m not addressing here what you have to do for that, because it involves editing your project files by hand, or removing projects from the solution. You should decide for yourself which of those steps carries the least risk of losing something important. Remember that you still have your files and their history in at least THREE version control systems at this point – your old version control, the VSTS system, and the local Git repository. So even if you screw this up, there’s little real risk.

Now that you have Git selected as your solution provider, you’ll see that the “Changes” option is now available in the Team Explorer window:

Save all the files (but I don’t have any open!) by pressing Ctrl-Shift-S, or selecting File => Save All.

If you skip this step, there will be no changes to commit, and you will be confused.

Select “Changes”, and you’ll see that the SLN files and VCPROJ files have been changed. You can preview these changes, but they basically are to do with removing the old version control from the projects and solution.

It wants a commit message. This should be short and explanatory. I like “Removed references to old version control from solution”. Once you’ve entered a commit message, the Commit button is available. Click it.

It now prompts you to Sync to the server.

So click the highlighted word, “Sync”, to see all the unsynced commits – you should only have one at this point, but as you can imagine, if you make several commits before syncing, these can pile up.

Press the “Sync” button to send the commit up to the server. This is also how you should usually get changes others have made to the code on the server. Note that “others” could simply mean “you, from a different computer or repository”.

Check on the server that the history on the branch now mentions this commit, so that you know your syncing works well.

And you’re done

Sure, it seems like a long-winded process, but most of what I’ve included here is pictures of me doing stuff, and the stuff I’m doing is only done once, when you create the repository and populate it from another. Once it’s in VSTS, I recommend building your solution, to make sure it still builds. Run whatever tests you have to make sure that you didn’t break the build. Make sure that you still have valid history on all your files, especially binary files. If you don’t have valid history on any files in particular, check the original version control, to see if you ever did have. I found that my old CS-RCS implementation was storing .bmp files as text, so the current version was always fine, but the history was corrupted. That’s history I can’t retrieve, even with the new source control.

Now, what about those temporary repositories? Git makes things really easy – the Git repository is in a directory off the root of the workspace, called “.git”. It’s hidden, but if you want to delete the repository, just delete the “.git” folder and its contents. You can delete any temporary workspaces the same way, of course.

I did spend a little time automating the conversion of multiple repositories to Git, but that was rather ad-hoc and wobbly, so I’m not posting it here. I’d love to think that some of the rest of this could be automated, but I have only a few projects, so it was good to do by hand.

Final Statement

No programmer should be running an unsupported, unpatched, unupdated old version control system. That’s risky, not just from a security perspective, but from the perspective that it may screw up your files, as you vary the sort of projects you build.

No programmer should be required to drop their history when moving to a new version control system. There is always a way to move your history. Maybe that way is to hire a grunt developer to fetch versions dated at random/significant dates throughout history out of the old version control system, and check them in to the new version control system. Maybe you can write automation around that. Or maybe you’ll be lucky and find that someone else has already done the automation work for you.

Hopefully I’ve inspired you to take the plunge of moving to a new version control system, and you’ve successfully managed to bring all your precious code history with you. By using Visual Studio Team Services, you’ve also got a place to track features and bugs, and collaborate with other members of a development team, if that’s what you choose to do. Because you’ve chosen Git, you can separate the code and history at any time from the issue tracking systems, should you choose to do so.

Let me know how (if?) it worked for you!

Posted in Alun's code, Miscellany - not security | Leave a comment

Got on with Git

In which I move my version control from ComponentSoftware’s CS-RCS Pro to Git while preserving commit history.

[If you don’t want the back story, click here for the instructions!]

OK, so having watched the video I linked to earlier, I thought I’d move some of my old projects to Git.

I picked one at random, and went looking for tools.

I’m hampered a little by the fact that all my old projects used ComponentSoftware’s “CS-RCS Pro”.

Why did you choose CS-RCS Pro?

A couple of really good reasons:

It works on Windows
It integrates moderately well with Visual Studio through the VSS functionality
It’s compatible with GNU RCS, which I had some familiarity with
It was free if you’re the only dev on your projects

But you know who doesn’t use CS-RCS Pro any more?

That’s right, ComponentSoftware.

It’s a dead platform, unsupported, unpatched, and belongs off my systems.

So why’s it still there?

One simple reason – if I move off the platform, I face the usual choice when migrating from one version control system to another:

Carry all my history, so that I can review earlier versions of the code (for instance, when someone says they’ve got a new bug that never happened in the old version, or when I find a reversion, or when there’s a fix needed in one area of the code tree that I know I already made in a different area and just need to copy)
Lose all the history by starting fresh with the working copy of the source code

The second option seems a bit of a waste to me.

OK, so yes, technically I could mix the two modes, by using CS-RCS Pro to browse the ancient history when I need to, and Git to browse recent history, after starting Git from a clean working folder. But I could see a couple of problems:

Of course the bug I’m looking through history for is going to be across the two source control packages
It would mean I still have CS-RCS Pro sitting around installed, unpatched and likely vulnerable, on one of my dev systems

So, really, I wanted to make sure that I could move my files, history and all.

What stopped you?

I really didn’t have a good way to do it.

Clearly, any version control system can be moved to any other version control system by the simple expedient of:

For each change X:
- Set the system date to X’s date
- Fetch the old source control’s files from X into the workspace
- Commit changes to the new source control, with any comments from X
- Next change

But, as you can imagine, that’s really long-winded and manual. That should be automatable.

In fact, given the shared APIs of VSS-compatible source control services, I’m truly surprised that nobody has yet written a tool to do basically this task. I’d get on it myself, but I have other things to do. Maybe someone will write a “VSS2Git” or “VSS2VSS” toolkit to do just this.

There is a format for creating a single-file copy of a Git repository, which Git can process using the command “git fast-import”. So all I have to find is a tool that goes from a CS-RCS repository to the fast-import file format.

Nobody uses CS-RCS Pro

So, clearly there’s no tool to go from CS-RCS Pro to Git. There’s a tool to go from CS-RCS Pro to CVS, or there was, but that was on the now-defunct CS-RCS web site.

But… Remember I said that it’s compatible with GNU RCS.

And there’s scripts to go from GNU RCS to Git.

What you waiting for? Do it!

OK, so the script for this is written in Ruby, and as I read it, there seemed to be a few things that made it look like it might be for Linux only.

I really wasn’t interested in making a Linux VM (easy though that may be) just so I could convert my data.

So why are you writing this?

Everything changed with the arrival of the recent Windows 10 Anniversary Update, because along with it came a new component.

Bash on Ubuntu on Windows.

It’s like a Linux VM, without needing a VM, without having to install Linux, and it works really well.

With this, I could get all the tools I needed – GNU RCS, in case I needed it; Ruby; Git command line – and then I could try this out for myself.

Of course, I wouldn’t be publishing this if it wasn’t somewhat successful. But there are some caveats, OK?

Here’s the caveats

I’ve tried this a few times, on ONE of my own projects. This isn’t robustly tested, so if something goes all wrong, please by all means share, and people who are interested (maybe me) will probably offer suggestions, some of them useful. I’m not remotely warrantying this or suggesting it’s perfect. It may wipe your development history out of your one and only copy of version control… so don’t do it on your one and only copy. Make a backup first.

GNU RCS likes to store files in one of two places – either in the same directory as the working files, but with a “,v” pseudo-extension added to the filename, or in a sub-directory off each working folder, called “RCS” and with the same “,v” extension on the files. If you did either of these things, there’s no surprises. But…

CS-RCS Pro doesn’t do this. It has a separate RCS Repository Root. I put mine in C:\RCS, but you may have yours somewhere else. Underneath that RCS Repository Root is a full tree of the drives you’ve used CS-RCS to store (without the “:”), and a tree under that. I really hope you didn’t embed anything too deep, because that might bode ill.

Initially, this seemed like a bad thing, but because you don’t actually need the working files for this task, you can pretend that the RCS Repository is actually your working space.

Maybe this is obvious, but it took me a moment of thinking to decide I didn’t have to move files into RCS sub-folders of my working directories.

Make this a “flag day”. After you do this conversion, never use CS-RCS Pro again. It was good, and it did the job, and it’s now buried in the garden next to Old Yeller. Do not sprinkle the zombification water on that hallowed ground to revive it.

This also means you MUST check in all your code before converting, because checking it in afterwards will be … difficult.

Enough already, how do we do this?

Assumption: You have Windows 10.

Install Windows 10 Anniversary Update – this is really easy, it’s an update, you’ve probably been offered it already, and you may even have installed it. This is how you’ll know you have it:
Install Bash on Ubuntu on Windows – everyone else has written an article on how to do this, so here’s a link (I was going to link to the PC World article, but the full-page ad that popped up and obscured the screen, without letting me click the “no thanks” button persuaded me otherwise).
Run the following commands in the bash shell:
sudo apt-get update
sudo apt-get install git
sudo apt-get install ruby
[Optional] Run “sudo apt-get instal rcs”, if you want to use the GNU RCS toolset to play with your original source control tree. Not sure I’d recommend doing too much of that.
Change directory in the bash shell to a new, blank workspace folder you can afford to mess around in.
Now a long bash command, but this really simply downloads the file containing rcs-fast-export:
curl http://git.oblomov.eu/rcs-fast-export/blob_plain/c8a2bd6edbb21c1bfaf269ad1ec0e82af72c911a:/rcs-fast-export.rb -o rcs-fast-export.rb
Make it executable with the command “chmod +x rcs-fast-export.rb”
Git uses email addresses, rather than owner names, and it insists on them having angle brackets. If your username in CS-RCS Pro was “bob”, and your email address is “kate@example.com”, create an authors file with a bash command like this:
echo “bob=Kate Smith <kate@example.com>” > AuthorsFile
Now do the actual creation of the file to be imported, with this bash command:
./rcs-fast-export.rb -A AuthorsFile /mnt/c/RCS/…path-to-project… > project-name.gitexport
[Note a couple of things here – starting with “./”, because that isn’t automatically in the PATH in Linux. Your Windows files are logically mounted in drives under /mnt, so C:\RCS is in /mnt/c/RCS. Case is important. Your “…path-to-project…” probably starts with “c/”, so that’s going to look like “/mnt/c/RCS/c/…” which might look awkward, but is correct. Use TAB-completion on folder names to help you.]
Read the errors and correct any interesting ones.
Now import the file into Git. We’re going to initialise a Git repository in the “.git” folder under the current folder, import the file, reset the head, and finally checkout all the files into the “master” branch under the current directory “.”. These are the bash commands to do this:
git init
git fast-import < project-name.gitexport
git reset
git checkout master .
Profit!
If you’re using Visual Studio and want to connect to this Git repository, remember that your Linux home directory sits under “%userprofile%\appdata\local\lxss\home”

This might look like a lot of instructions, but I mostly just wanted to be clear. This is really quick work. If you screw up after the “git init” command, simply “rm –rf .git” to remove the new repository.

Posted in Alun's code, Miscellany - not security, Programmer Hubris, Windows 10 | 1 Comment

Capital ‘I’ Internet

The Atlantic today published a reminder that the Associated Press has declared in their style guide as of today that the word “Internet” will be spelt with a lowercase ‘i’ rather than an uppercase ‘I’.

The title is “Elegy for the Capital-I Internet”, but manages to be neither elegy nor eulogy, and misses the mark entirely, focusing as it does on the awe-inspiring size of the Internet being why the upper-case initial was important; then moving to describe how its sheer ubiquity should lead us to associating it with a lower-case i.

Here’s my take

The "Internet", capital I, gives the information that this is the only one of its kind, anywhere, ever. There is only one Internet. A lower-case I would indicate that there are several "internets". And, sure enough, there are several lower-class networks-of-networks (which is the definition of “internet” as a lower-case noun).

I’d like to inform the people who are engaging in this navel-gazing debate over big-I or small-i, that there functionally is only exactly one Internet. When their cable company came to "install the Internet", there was no question on the form to say "which internet do you want to connect to?" and people would have been rightly upset if there had been.

So, from that perspective, very much capital-I is still the right term for the Internet. There’s only one. Those other smaller internets are not comparable to “the Internet”.

From a technical perspective, we’re actually at the time when it’s closest to being true that there’s two internets. We’re in the midst of the long, long switch from IPv4 to IPv6. We’ve never done that before. And, while there are components of each that will talk to the other, it’s possible to describe the IPv6 and IPv4 collections of networks as two different "internets". So, maybe small-i is appropriate, but for none of the reasons this article describes.

Having said that, IPv6 engineers work really really hard to make sure that users just plain don’t notice that there’s a second internet while they’re building it, and it just feels exactly like it would if there was still only one Internet.

Again, you come back to "there is only one Internet", you don’t get to check a box that selects which of several internets you are going to connect to, it’s not like "the cloud", where there are multiple options. You are either connected to the one Internet, or you’re not connected to any internet at all.

The winner is…

Capital I, and bollocks to the argument from the associated press – lower-cased, because it’s not really that big or important, and neither is the atlantic. So, with their own arguments (which I believe are fallacious anyway), I don’t see why they deserve an upper-case initial.

The Atlantic, on the other hand – that’s huge and I wouldn’t want to cross it under my own steam.

And the Internet, different from many other internets, deserves its capital I as a designation of its singular nature. Because it’s a proper noun.

Posted in Bad Names, IPv6, Miscellany - not security | Leave a comment

The blame game: It’s always never human error

The Ubuntu “Circle of Friends” logo.

Depending on the kind of company you work at, it’s either:

a group of three friends holding hands and dancing in a merry circle
a group of three colleagues each pointing at the other two to tell you who to blame
three guys tied to a pole desperately trying to escape the rising orange flood waters

If you work at the first place, reach out to me on LinkedIn – I know some people who might want to work with you.

If you’re at the third place, you should probably get out now. Whatever they’re paying you, or however much the stock might be worth come the IPO, it’s not worth the pain and suffering.

If you’re at the second place, congratulations – you’re at a regular, ordinary workplace that could do with a little better management.

What’s this to do with security?

A surprisingly great deal.

Whenever there’s a security incident, there should be an investigation as to its cause.

Clearly the cause is always human error. Machines don’t make mistakes, they act in predictable ways – even when they are acting randomly, they can be stochastically modeled, and errors taken into consideration. Your computer behaves like a predictable machine, but at various levels it actually routinely behaves like it’s rolling dice, and there are mechanisms in place to bias those random results towards the predictable answers you expect from it.

Humans, not so much.

Humans make all the mistakes. They choose to continue using parts that are likely to break, because they are past their supported lifecycle; they choose to implement only part of a security mechanism; they forget to finish implementing functionality; they fail to understand the problem at hand; etc, etc.

It always comes back to human error.

Or so you think

Occasionally I will experience these great flashes of inspiration from observing behaviour, and these flashes dramatically affect my way of doing things.

One such was when I attended the weekly incident review board meetings at my employer of the time – a health insurance company.

Once each incident had been resolved and addressed, they were submitted to the incident review board for discussion, so that the company could learn from the cause of the problem, and make sure similar problems were forestalled in future.

These weren’t just security incidents, they could be system outages, problems with power supplies, really anything that wasn’t quickly fixed as part of normal process.

But the principles I learned there apply just as well to security incident.

Root cause analysis

The biggest principle I learned was “root cause analysis” – that you look beyond the immediate cause of a problem to find what actually caused it in the long view.

At other companies, who can’t bear to think that they didn’t invent absolutely everything, this is termed differently, for instance, “the five whys” (suggesting if you ask “why did that happen?” five times, you’ll get to the root cause). Other names are possible, but the majority of the English-speaking world knows it as ‘root cause analysis’

This is where I learned that if you believe the answer is that a single human’s error caused the problem, you don’t have the root cause.

But!

Whenever I discuss this with friends, they always say “But! What about this example, or that?”

You should always ask those questions.

Here’s some possible individual causes, and some of their associated actual causes:

Bob pulled the wrong lever	Who trained Bob about the levers to pull? Was there documentation? Were the levers labeled? Did anyone assess Bob’s ability to identify the right lever to pull by testing him with scenarios?
Kate was evil and did a bad thing	Why was Kate allowed to have unsupervised access? Where was the monitoring? Did we hire Kate? Why didn’t the background check identify the evil?
Jeremy told everyone the wrong information	Was Jeremy given the right information? Why was Jeremy able to interpret the information from right to wrong? Should this information have been automatically communicated without going through a Jeremy? Was Jeremy trained in how to transmute information? Why did nobody receiving the information verify it?
Grace left her laptop in a taxi	Why does Grace have data that we care about losing – on her laptop? Can we disable the laptop remotely? Why does she even have a laptop? What is our general solution for people, who will be people, leaving laptops in a taxi?
Jane wrote the algorithm with a bug in it	Who reviews Jane’s code? Who tests the code? Is the test automated? Was Jane given adequate training and resources to write the algorithm in the first place? Is this her first time writing an algorithm – did she need help? Who hired Jane for that position – what process did they follow?

I could go on and on, and I usually do, but it’s important to remember that if you ever find yourself blaming an individual and saying “human error caused this fault”, it’s important to remember that humans, just like machines, are random and only stochastically predictable, and if you want to get predictable results, you have to have a framework that brings that randomness and unpredictability into some form of logical operation.

Many of the questions I asked above are also going to end up with the blame apparently being assigned to an individual – that’s just a sign that it needs to keep going until you find an organisational fix. Because if all you do is fix individuals, and you hire new individuals and lose old individuals, your organisation itself will never improve.

[Yes, for the pedants, your organisation is made up of individuals, and any organisational fix is embodied in those individuals – so blog about how the organisation can train individuals to make sure that organisational learning is passed on.]

Finally, if you’d like to not use Ubuntu as my “circle of blame” logo, there’s plenty of others out there – for instance, Microsoft Alumni:

Microsoft Alumni

Posted in Miscellany - not security, Programmer Hubris | Leave a comment

A quick April Fools’ Day reminder

Tomorrow is April 1, also known as April Fools’ Day.

As a result, you should expect that anything I say on this blog is fabrication, fantasy, foolery and snark.

Apparently, this hasn’t previously been completely stupidly blindly obvious.

Posted in Miscellany - not security | Leave a comment

Leap Day again

I’ve mentioned before how much I love the vagaries of dates and times in computing, and I’m glad it’s not a part of my regular day-to-day work or hobby coding.

Here’s some of the things I expect to happen this year as a result of the leap year:

Hey, it’s February 29 – some programs, maybe even operating systems, will refuse to recognise the day and think it’s actually March 1. Good luck figuring out how to mesh that with other calendar activities. Or maybe you’ll be particularly unlucky, and the app/OS will break completely.
But the fun’s not over, as every day after February 29, until March 1 NEXT YEAR, you’re a full 366 days ahead of the same date last year. So, did you create a certificate that expires next year, last year? If so, I hope you have a reminder well ahead of time to renew the certificate, because otherwise, your certificate probably expires 365 days ahead, not one year. Or maybe it’ll just create an invalid certificate when you renew one today.
The same is true for calendar reminders – some reminders for “a year ahead” will be 365 days ahead, not one year. Programmers often can’t tell the difference between AddDays(365) and AddYears(1) – and why would they, when the latter is difficult to define unambiguously (add a year to today’s date, what do you get? February 28 or March 1?)
But the fun’s not over yet – we’ve still got December 31 to deal with. Why’s that odd? Normal years have a December 31, so that’s no problem, right? Uh, yeah, except that’s day 366. And that’s been known to cause developers a problem – see what it did to the Zune a few years back.
Finally, please don’t tell me I have an extra day and ask me what I’m going to do with it – the day, unless you got a day off, or are paid hourly, belongs to your employer, not to you – they have an extra day’s work from you this year, without adding to your salary at all.

And then there’s the ordinary issues with dates that programmers can’t understand – like the fact that there are more than 52 weeks in a year. “ASSERT(weeknum>0 && weeknum<53);”, anyone? 52 weeks is 364 days, and every year has more days than that. [Pedantic mathematical note – maybe this somewhat offsets the “employer’s extra day” item above]

Happy Leap Day – and always remember to test your code in your head as well as in real life, to find its extreme input cases and associated behaviours. They’ll get tested anyway, but you don’t want it to be your users who find the bugs.

Posted in Miscellany - not security, Programmer Hubris, Zune | Leave a comment

Get on with git

Out with the old

Version control is one of those vital tools for developers that everyone has to use but very few people actually enjoy or understand.

So, it’s with no surprise that I noted a few months ago that the version control software on which I’ve relied for several years for my personal projects, Component Software’s CS-RCS, has not been built on in years, and cannot now be downloaded from its source site. [Hence no link from this blog]

Not so in with the new

I’ve used git before a few times in professional projects while I was working at Amazon, but relatively reluctantly – it has incredibly baroque and meaningless command-line options, and gives the impression that it was written by people who expected their users to be just as proficient with the ins and outs of version control as they are.

While I think it’s a great idea for developers to build software they would use themselves, I think it’s important to make sure that the software you build is also accessible by people who aren’t the same level of expertise as yourself. After all, if your users were as capable as the developer, they would already have built the solution for themselves, so your greater user-base comes from accommodating novices to experts with simple points of entry and levels of improved mastery.

git, along with many other open source, community-supported tools, doesn’t really accommodate the novice.

As such, it means that most people who use it rely on “cookbooks” of sets of instructions. “If you want to do X, type commands Y and Z” – without an emphasis on understanding why you’re doing this.

This leads inexorably to a feeling that you’re setting yourself up for a later fall, when you decide you want to do an advanced task, but discover that a decision you’ve made early on has prevented you from doing the advanced task in the way you want.

That’s why I’ve been reluctant to switch to git.

So why switch now?

But it’s clear that git is the way forward in the tools I’m most familiar with – Visual Studio and its surrounding set of developer applications.

It’s one of those decisions I’ve made some time ago, but not enacted until now, because I had no idea how to start – properly. Every git repository I’ve worked with so far has either been set up by someone else, or set up by me, based on a cookbook, for a new project, and in a git environment that’s managed by someone else. I don’t even know if those terms, repository and environment, are the right terms for the things I mean.

There are a number of advanced things I want to do from the very first – particularly, I want to bring my code from the old version control system, along with its history where possible, into the new system.

And I have a feeling that this requires I understand the decisions I make when setting this up.

So, it was with much excitement that I saw a link to this arrive in my email:

Next thing is I’m going to watch this, and see how I’m supposed to work with git. I’ll let you know how it goes.

Posted in Alun's code, Miscellany - not security, Programmer Hubris | Leave a comment

Auto convert inked shapes in PowerPoint–coming to OneNote

I happened upon a blog post by the Office team yesterday which surprised me, because it talked about a feature in PowerPoint that I’ve wanted ever since I first got my Surface 2.

Shape recognition

Here’s a link to documentation on how to use this feature in PowerPoint.

https://support.office.com/en-us/article/use-a-pen-to-draw-write-or-highlight-text-on-a-windows-tablet-6d76c674-7f4b-414d-b67f-b3ffef6ccf53

It seems like the obvious feature a tablet should have.

Here’s a video of me using it to draw a few random shapes:

But not just in PowerPoint – this should be in Word, in OneNote, in Paint, and pretty much any app that accepts ink.

And at last, OneNote

So here’s the blog post from Office noting that this feature will finally be available for OneNote in November.

https://blogs.office.com/2015/10/20/onenote-partners-with-fiftythree-to-support-pencil-and-paper-plus-shape-recognition-coming-soon/

On iPad, iPhone and Windows 10. Which I presume means it’ll only be on the Windows Store / Metro / Modern / Immersive version of OneNote.

That’s disappointing, because it should really be in every Office app. Hell, I’d update from Office 2013 tomorrow if this was a feature in Office 2016!

Let’s not stop there

Please, Microsoft, don’t stop at the Windows Store version of OneNote.

Shape recognition, along with handwriting recognition (which is apparently also hard), should be a natural part of my use of the Surface Pen. It should work the same across multiple apps.

That’s only going to happen if it’s present in multiple apps, and is a documented API which developers – of desktop apps as well as Store apps – can call into.

Well, desktop apps can definitely get that.

How can I put it into my own app?

I’ll admit that I haven’t had the time yet to build my own sample, but I’m hoping that this still works – there’s an API called “Ink Analysis”, which is exactly how you would achieve this in your app:

https://msdn.microsoft.com/en-us/library/ms704040.aspx

It allows you to analyse ink you’ve captured, and decide if it’s text or a drawing, and if it’s a drawing, what kind of drawing it might be.

[I’ve marked this with the tag “Alun’s Code” because I want to write a sample eventually that demonstrates this function.]

Posted in Alun's code, Miscellany - not security, Surface, Windows 10 | Leave a comment