alunj – Tales from the Crypto

alunj

1 2 3 50

A late contribution to the expired root in certificate chain issue…

In case you missed it, on May 30th, a root certificate expired.

This made a lot of applications very unreliable, and been widely regarded as a bad move.

Well, alright, what was regarded as a bad move is that applications should become unreliable in the specific circumstances involved here.

As short a TL;DR as I can manage

When you connect to a  server(web site or application) over SSL/TLS, the server has to send your client (browser or application) its Certificate.

In modern code, this Certificate is used by the client to trace back to a signing authority that is trusted by the client or its operating system.

Some servers like to help this process out, by sending a chain along with the Certificate for a couple of reasons:

  1. The client might not have the ability or time to go building and checking its Certificate chain, and choose to trust only servers that send it an entire chain they can trust
  2. Older clients might not be aware of the newer root certificates up to which the supplied Certificate chain connects.

This second situation is what we’re interested in here. A new root appears, new certificates are issued, and old clients refuse to honour them because they don’t have the new root in their trust store.

This is fixed with “cross-signing”, which allows an older, trusted root, to sign the new untrusted root, so that the older client sees a chain that includes the older root at the top, and is therefore trusted.

Older root certificates expire. It takes 20 years, but it finally happened at the end of May, to this one root certificate, “AddTrust External CA Root”

When that happens, a client who builds the certificate chain and uses this to trust the root certificate is happy, because it sees only certificates that it trusts.

A client who takes the certificate chain as supplied by the server, without building its own, will see that the chain ends in an expired certificate, and refuse to connect, because the entire chain cannot be trusted.

What do I bring to the party?

The two links I provided earlier are well worth a read if you’re interested in solving this problem, and really, I’ve got nothing to add to how this issue occurred, why it’s a problem, how to address it at your server, or any of those fun things.

What I do offer is a tool for .NET (Windows and Linux, Mac, etc) that lets you compare the certificate chain as presented by the server against the certificate chain built by a client. It will report if a certificate in either chain has expired. It’s written in C#, and built with Visual Studio, and takes one parameter – the site to which it will connect on port 443 to query for the certificate and chain.

It’s not a very smart tool, and it makes a few assumptions (though it’s relatively easy to fix if those assumptions turn out to be false).

But it has source code, and it runs on Windows, Linux and (presumably – haven’t tested) Mac.

How does it look?

Working against the sites listed at http://testsites.test.certificatetest.com/, we get the following results:

First: https://aaacertificateservices.test.certificatetest.com/ – Certificate issued from a CA signed by AAA Certificate Services root.

SNAGHTML1d6d6a33

Interestingly, note that the certificate chain in the stream from the server doesn’t include the root certificate at all, but it’s present in the code where we ask the client code what certificates are in the chain for this server.

Second: https://addtrustexternalcaroot.test.certificatetest.com/ – Certificate issued from a CA signed by AddTrust External CA Root.

image

The certificates here expired on 5/30/2020, and it’s no surprise that we see this result in both the chain provided by the server and the chain provided by the client. Again, the root certificate isn’t actually in the chain from the server provided in the stream.

Third: https://addtrustaia.test.certificatetest.com/ – Certificate issued from a CA signed by USERTrust RSA Certification Authority with a cross cert via AIA from AddTrust External CA Root.

image

Nothing noteworthy here, but it’s included here for completeness. I don’t do anything in this code for an AIA cross cert.

Fourth, and most importantly: https://addtrustchain.test.certificatetest.com/ – Certificate issued from a CA signed by USERTrust RSA Certification Authority with a cross cert via server chain from AddTrust External CA Root.

image

Here’s the point of the tool – it’s able to tell you that there’s a certificate in the chain from the server that has expired, and may potentially be causing problems to visitors using an older browser or client library.

Enough waffle, where’s the chicken?

By now, you’ve had enough of reading and you want to see the code – or just run it. I’ve attached two files – one for the source code, the other for the executable content. I leave it up to others to tell you how to install dotnet core on your platform.

Here’s the source code

And here’s the binary

Let me know if, and how, you use this tool, and whether it achieves whatever goal you want from it.

Revisiting NTFS alternate data-streams

I can’t believe it’s been over thirteen years since I last wrote about NTFS Alternate Data-Streams.

A lot has changed since then, including the fact that I’ve taken down the site where my download for “sdir” was listed. But that’s an old tool, and I don’t think we need it any more.

What else has changed is that my wife is studying a number of security courses with the SANS Women’s Academy, which is an excellent initiative to bring more women into the world of information security, where they, along with the rest of humanity (for whom SANS has other programs), are sorely needed. One of the classes she was studying included a piece on NTFS Alternate Data Streams, or ADS.

For the new visitors – what’s an ADS?

An Alternate Data Stream, or ADS, is a parallel stream of data, as the name implies, to the default data stream of a particular file. This default data stream is what most users have spent their lives thinking of as “the file”.

The file is more than just the bytes it contains, in this case. You can go a long way without realising this.

Alternate Data Streams were originally created to support Apple Mac Resource Forks, in files copied from Apple to NTFS and back. I’m not sure Apple even bothers with them any more, now that they’ve moved to something akin to Linux as their OS.

Created as part of the original NTFS in 1993, these Alternate Data Streams shouldn’t be confused with:

  • In-stream metadata, such as EXIF data in JPGs, or ID3 tags in MP3s, M4As, MP4s, etc, or Author information in DOCX files – all of these are inside the default data stream
  • File metadata, such as owner, security, EFS encryption keys – all of these are in the attributes of the file, in the file table (think of this as being in the directory list, not with the files themselves)

Can I see these streams?

Not really easily – at the command prompt, you can use “dir /r” to view the files in your current directory along with all their attendant streams – but you can’t combine the “/r” and “/b” options, so you can’t get a really succinct list of all the streams in your system. Here’s an example listing of a download directory:

image

In PowerShell, you have more control, and you can even call in to .NET, but you don’t need to in order to see file streams. Here’s a simple command to display just the non-default data streams on files in a particular directory:

Get-ChildItem | Get-Item -Stream * | Where-Object Stream -ne ':$DATA' | Format-Table filename,stream,length

The output this produces looks like this:

image

Left as an exercise for the reader – how you do this recursively through subdirectories to find all the streams.

How are ADSs used?

The most common ADS on your directory is almost certainly the stream named “Zone.Identifier”, as this is created on every file you download from the web using Internet Explorer, Edge, Chrome, Outlook, or any application that cooperates with Microsoft’s idea of marking files that have been downloaded. If you open Explorer and view properties on a file that’s been downloaded, you’ll see there’s a checkbox allowing you to “Unblock” this file, along with a note that it came from another computer. Checking the “Unblock” box and clicking OK or Apply will remove this Zone.Identifier stream.

image

This stream is known as the “Mark Of The Web” or “MOTW” in some documentation, so that’s another term to use if you’re searching for this stream.

Other stream names I find on my hard drive:

uidStream – I found this on some eBooks in my “My Kindle Books” folder, but whether they’re specific to the Kindle app, or some other e-reader I’ve used, I can’t be certain.

SmartScreen – these are on some downloaded .EXE files, so from the name and file type, I’ll assume this is from virus scanning downloaded EXEs. Since the stream contains just the word “Anaheim”, I’m not sure how useful this is.

ms-properties – a binary stream on a few of the JPG images I have on my computer, all of which are photos I took on my Surface Pro.

And some very oddly-named streams on some scanned files, because there’s just as much of a standard for stream names as there are for file names, and it’s completely a Wild West out there, so the best way to make sure you’re not going to be overwritten by someone else’s stream is to pick a completely weird and off the wall stream name.

image

Joking aside, the second of those shows that choosing a GUID is actually a good way to name a stream so it doesn’t collide with others – it’s random, and you can make it searchable on the web by documenting it.

Sure enough, if we search for that GUID, there’s some interesting information to be found at https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/4f3837c4-2f96-40d7-b0bf-80dd1d0b0da0, among other places. This particular GUID is used to include some kind of summary information.

I’ve also read in a couple of places that the Windows File Classification Infrastructure uses ADS to carry information.

Other possible uses of ADS

It doesn’t take much thinking to come up with other uses for alternate data streams. Really any time you might want to associate data with a file, or several files, without bothering the application that might want to read the file. Here’s some suggestions:

  • Metadata for files that don’t support embedding ID3 or similar tags
  • Preview clips, sample images
  • Maintaining hashes of file contents so as to be aware of file changes (like the “TripWire” application which detects changes to files on servers that shouldn’t change)
  • Document tracking information

Thinking on this, there’s a couple of ideas I already have – if I can extract ID3 tags from files and put them into an ADS, it’s going to be quicker and easier to find that information than parsing the entire MP4/MP3/M4A file each time I want to look at the data.

Is ADS a security threat?

I’ve commented on this before, and I’ve read a lot about how “obviously” viruses will use ADS to hide, or that exfiltration will use ADS to avoid detection, and while there’s some truth to this idea, I think both threats are overblown.

For exfiltration, the problem is essentially the same as that with using EFS to encrypt a file that’s being exfiltrated – in order for the data to leave the system, it has to pass through the usual file APIs that your DLP solution is hooked into, and unless your DLP solution is being too smart for its britches, the data will be noticed and blocked. Copying a file from NTFS to FAT or exFAT will destroy the associated ADS data as if it was never there, just as it will destroy EFS encryption.

For virus hiding, while it’s not impossible to execute from an ADS, it’s not particularly easy, and the methods used themselves can trigger your antivirus. To load and execute the data in the ADS, you have to use normal means to load and execute code in a default data stream. And those normal means can be detected by the virus scanner just as easily as they detect any other executable content. If your virus scanner hooks normal load/execute APIs, it’ll also intercept the loading and execution of the ADS.

This is probably why there’s only one virus I found significant information on that uses ADS to hide parts of itself – Backdoor:Win32/Rustock.A – which copies itself into streams off the system32 folder. From the technical description of this virus, it’s also clear that the virus has a fail-back facility for when it’s trying to install itself on a system with no ADS support (really, who installs Windows on a FAT partition? Maybe they mean ReFS, which didn’t initially support ADS).

Your most likely ADS security threat

The most likely ADS security threat is still the one for which it’s best known – that of accessing the default data stream of a file by appending “:$DATA” to the requested filename, and getting around restrictions that an application might have in place.

Years and years ago (1998), this was a trick you could use against IIS to fetch the source code of an ASP page – instead of fetching “pagename.asp” (which gave you the output of executing the code), you’d fetch “pagename.asp:$DATA”.

Obviously, IIS fixed this years and years ago, and yet the problem comes up over and over again, in other applications which map incoming requests to files through a simple mapping (file name requested ≊ file name fetched), and which aren’t aware of this kind of issue. (On Windows, you can open a file “for information only” and then query the handle for its canonical name, if you need to write code to get around this – see Writing Secure Code 2nd Edition for details)

So, every now and again, if you’re a hacker and you can’t get a file, try getting it with “:$DATA” at the end of its name.

Command line ADS handling

Command Prompt

The command prompt has a few ways to handle Alternate Data Streams with files.

  1. Pass the stream name to an executable – maybe it’ll handle it.
    Notepad is a good example of this – you can directly edit streams by running “notepad file:stream”.
    Exercise for the reader – create a file called “C”. Open a stream called “foo.txt” on file “C” using notepad.
  2. Redirection – you can always use redirection to and from streams
    ”dir > foo.txt:bar.dat” will create a stream on file foo.txt
    ”more < foo.txt:bar.dat” will display the stream back to you.
  3. Some built-in commands may have stream support built in – but this is very limited.
    As an example, you’ve seen the “dir /r” command I used above, which lists streams.

Very limited, as you can tell – you can’t do “dir /s/r/b” to get a list of all the streams, because the /b parameter ignores the /r parameter. You can’t directly load an executable from a stream, but you can use another EXE to load it for you (there are examples available online of using WMIC and PSEXEC to do this)

If you absolutely have to remove an alternate data stream from an NTFS file with only Explorer or the Command Prompt, moving it to and from a FAT or exFAT formatted drive (such as a USB stick) will do that, but will also kill any other NTFS properties, such as ownerships, permissions, EFS encryption, etc, as well as killing any audit continuity on the file. I don’t recommend this, particularly for those files that you really don’t want your name associated with the creation of.

PowerShell

The news is supposedly a little better in PowerShell, which is meant to have built-in support for ADS.

In PowerShell, we use Get-ChildItem to navigate through folders, and Get-Item to look at individual files. Remove-Item is what we use to delete files. Each of these commands has a “-Stream” parameter, so it seems we are set for our alternate data stream handling.

We can delete a stream from a file as easily(!) as this:

Remove-Item <file> -Stream <stream>

image

It feels a little weird, but it only deletes the stream, not the file itself.

Seems like this should work to list all streams from our current directory going down, right?

Get-ChildItem -Recurse | Get-Item -Stream * | Where-Object Stream -ne ‘:$DATA’ | Format-Table FileName,Stream,Length

Well, it does most of what we’re looking for.

What it specifically misses is the directories.

Wait, what, directories? Folders?

Yeah, you can put an alternate data stream on a folder. You can’t put a default data stream on a directory, but you can put any number of alternate data streams there.

image

My PowerShell script won’t find that ‘whatnot.txt’ stream. Curiously enough, this is documented at https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-item?view=powershell-5.1 even though it’s clearly an oversight. “This parameter isn’t valid on folders” – well, it should be.

image

Can we use the Remove-Item -Stream parameter to delete streams from directory, even if we can’t actually find them using PowerShell?

Sure, but it’s even more scary:

image

Contrary to what the warning says, the directory and all its children were not deleted, just the stream. Everything is safe and well.

Summarising the bugs

So, what needs fixing?

  1. The command prompt needs a DEL command that deletes a stream. Because, really, functionality.
  2. While the command prompt can do a DIR/R, it needs to work with the /B parameter, so that DIR/S/R/B|FINDSTR “:” can be used to find all the streams.
  3. Windows Explorer should have a way to find streams,  browse them, open them in an application, and delete them.
  4. PowerShell’s Get-Item needs to support the -Stream parameter on folders, because NTFS allows streams on folders.
  5. PowerShell’s Remove-Item needs to just delete a stream when you specify -Stream and not scare the living daylights out of you.

One final bug

Oh, yeah, and what on earth is with this lack of support for actual, real, file system features in PowerShell?

image

And yes, I’m kind of cheating here, but not much!

Oh, and this folder confuses the command prompt’s “dir/r/s” as well. Note that the directory “AUX” doesn’t have a stream, but when listing the contents of that directory, the directory “.” DOES.

image

Attribution

The words, the exploration and examples, the concepts and the thinking, are all shared work between Debbie Lester-Jones and myself.

At some point, when she’s done with her classes, one of you could be lucky enough to employ her. Or any of the other awesome students of the SANS Women’s Academy.

Extracting data from Word forms with XSL Transforms (XSLT)

“What’s the point,” pondered Alice, “Of getting other people to stuff things in a box, if one cannot ever get them out?”

Ok, she never did say that, but it’s the sort of thing Alice would wonder.

Particularly if she noticed how often modern businesses send around Word forms with input fields designed to be filled out by team members, only to then be manually copied into spreadsheets, databases, or other documents.

I’d put this as the second most irritating waste of document functionality.

And it doesn’t have to be this way.

What is a Word form, anyway?

FIrst, let’s look at what you get with a Word form. There really isn’t anything quite as specific a beast as a Word form. It’s just a Word document. With form fields. Form fields are places into which users can type text, check boxes, select from drop down lists, etc.

Once form fields have been put into a document, the original document author can ‘restrict’ the document such that only editing the form fields is allowed. This is usually done with a password, to make it less likely that others will edit the document beyond the form fields.

The presence of a password should not be taken to indicate that this is a security measure.

Removing the restriction can be done by guessing the password, or accessing the settings.xml inside the docx file, and changing the value of “w:enforcement” from “1” to “0”. Other methods include saving to RTF, then editing the file in a text editor before saving it as docx again.

Restricting the document is done to make it less likely that blithe nonces will return your document to you with changes that are outside of the fields you’ve provided to them, or with fields removed. This is important, because you can’t as easily extract data from a document if you don’t know where it is.

Show me

Annotation 2020-05-16 165908

Here’s what a form looks like when it’s restricted for editing, and has a number of form field elements provided – I’ve given a text field for a person’s name, a drop-down list for their zodiac sign, and a check box for education level. This is the sort of thing you might expect a form to be really useful for collecting.

Now that you’ve sent this out to a hundred recipients, though, you want to extract the data from each form.

Secret knowledge: docx files are ZIP files containing XML files

First we’ve got to get the part of the document containing the data out. Knowing, as we do, that a docx file is just a ZIP file full of XML files, we could unzip it and go searching for the data. I’ve already done that – the data is in the file called “word/document.xml”. You could just rename the docx file to a zip file, open it in Explorer, navigate into the ‘word’ folder, and then drag the document.xml file out for handling, but that’s cumbersome, and we want an eventual automated solution.

Turning to PowerShell

Yeah, you could write this in a batch file using whatever ZIP program you’ve downloaded, it wouldn’t be that difficult, but I’m thinking about PowerShell a lot these days for my automation. Here’s code that will take a docx file and extract just the word/document.xml component into an output file whose name is provided.

# Load up the types required to handle a zip file.
Add-Type -AssemblyName System.IO.Compression.Filesystem

Function Get-DocXDocFile ($infilename, $outfilename){
$infileloc = [System.IO.Path]::Combine($pwd,$infilename)
$zip = [System.IO.Compression.ZipFile]::OpenRead($infileloc)
$zip.Entries | where { $_.FullName -eq “word/document.xml” } | foreach {
$outfileloc = [System.IO.Path]::Combine($pwd,$outfilename)
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, “$outfileloc”,$true)
}
}

 

By now, if you’re like me, you’ve opened up that XML file and looked into it, and decided you don’t care that much to read its entrails.

That’s OK, I did it for you.

The new-style fields are all in “w:sdt” elements, and can be found by the “w:tag” name under the “w:sdtPr” element.

Old-style fields are all in “w:fldChar” elements, and can be found by the “w:name” value under the “w:ffData” element.

In XPath, a way of describing how you find a specific element / attribute in an XML file, that’s expressed as follows:

//w:sdt/w:sdtPr/w:tag[@w:val=’Txt1Tag’]/../..

//w:fldChar/w:ffData/w:name[@w:val=’Text1′]/../..

This does assume that you gave each of your fields names or tags. But it would be madness to expect data out if you aren’t naming your fields.

If you’re handy with .NET programming, you’re probably half way done writing the code to parse this using XmlDocument.

If only there was an easy – well, easier – way to do this!

If you’re not handy with .NET programming, you might need something a little (but sadly, not a lot) easier.

Remember those XPath elements? Wouldn’t it be really cool if we could embed those into a document, and then have that document automatically expand them into their contents, so we could do that for every form file we’ve got?

Well, we can.

It’s called XSLT

Short for Extensible Stylesheet Language Transformation (which is definitely long enough to need something to be short for it), XSLT, which really has no good pronunciation because I’m never going to say something that sounds like “ex-slut” at work, XSLT is a way to turn one XML-formatted document into some kind of output.

Let’s say we’re working with the document I outlined above (and which I will forget to attach to this blog post until someone points it out). We’ve already extracted document.xml, and with the right XSL file, and a suitable XSLT command (such as the Microsoft msxml tool, or whatever works in your native environment), we can do something like this:

Annotation 2020-05-18 172303

Maybe instead of text, you prefer something more like CSV:

Annotation 2020-05-18 191107

I will probably forget to attach the XSL stylesheets that I used for these two transformations to this blog post.

Maybe next time we can see about building this into a tool…

Here’s the files I forgot to add: ExtractData

What does “input validation” even mean any more?

Information Security is full of terminology.

Sometimes we even understand what we mean. I’ve yet to come across a truly awesome, yet brief, definition of “threat”, for instance.

But one that bugs me, because it shouldn’t be that hard to get right, and because I hear it from people I otherwise respect greatly, is that of “input validation”.

What is “validation”?

Fight me on this, but I think that validation is essentially a yes/no decision on a set of input, whether it’s textual, binary, or whatever other format you care to define.

Exactly what you are validating is up for debate, whether you’re looking at syntax or semantics – is it formatted correctly, versus does it actually make sense?

Syntax versus semantics

“Green ideas sleep furiously” is a famous example of a sentence that is syntactically correct – it follows a standard “Adjective noun verb adverb” pattern that is common in English – but semantically, it makes no sense: ideas can’t be green, and they can’t sleep, and nothing can sleep furiously (although my son used to sleep with his fists clenched really tight when he was a little baby).

“0 / 0” is a syntactically correct mathematical expression, but you can argue if it’s semantically correct.

“Sell 1000 shares” might be a syntactically correct instruction, but semantically, it could be you don’t have 1000 shares, or there’s a business logic limit, which says such a transaction requires extra authentication.

So there’s a difference between syntactical validation and semantic validation, but…

What’s that got to do with injection exploits?

Injection attacks occur when an input data – a string of characters – is semantically valid in the language of the enclosing code, as code itself, and not just as data. Sometimes (but not always) this means the data contains a character or character sequence that allows the data to “escape” from its data context to a code context.

Can validation stop injection exploits?

This is a question I ask, in various round-about ways, in a lot of job interviews, so it’s quite an important question.

The answer is really simple.

Yes. And no.

If you can validate your input, such that it is always syntactically and semantically correct, you can absolutely prevent injection exploits.

But this is really only possible for relatively simple sets of inputs, and where the processing is safe for that set of inputs.

How about an example?

An example – suppose I’ve got a product ordering site, and I’m selling books.

You can order an integer number of books. Strictly speaking, positive integers, and 0 makes no sense, so start at 1. You probably want to put a maximum limit on that field, perhaps restricting people to buying no more than a hundred of that book. If they’re buying more, they’ll want to go wholesale anyway.

So, your validation is really simple – “is the field an integer, and is the integer value between 1 and 100?”

What about a counter-example?

Having said “yes, and no”, I have to show you an example of the “no”, right?

OK, let’s say you’re asking for validation of names of people – what’s your validation rules?

Let’s assume you’re expecting everyone to have ‘latinised’ their name, to make it easy. All the letters are in the range a-z, or A-Z if there’s a capital letter.

Great, so there’s a rule – only match “[A-Za-z]”

Unless, you know, Leonardo da Vinci. Or di Caprio. So you need spaces.

Or Daniel Day-Lewis. So there’s also hyphens to add.

And if you have an O’Reilly, an O’Brian, or a D’Artagnan, or a N’Dour – yes, you’re going to add apostrophes.

Now your validation rule is letting in a far broader range of characters than you start out with, and there’s enough there to allow for SQL injection to happen.

Input can now be syntactically correct by your validation rule, and yet semantically equivalent to data plus SQL code.

Validation alone is insufficient to block injection attacks.

Why do people say validation is sufficient, then?

I have a working hypothesis. It goes like this.

As a neophyte in information security, you learn a trick.

That trick is validation, and it’s a great thing to share with developers.

They don’t need to be clever or worry hard about the input that comes in, they simply need to validate it.

It actually feels good to reject incorrect input, because you know you’re keeping the bad guys out, and the good guys in.

Then you find an input field where validation alone isn’t sufficient.

Something else must be done

But you’ve told everyone – and had other security folk agree with you – that validation is the way to solve injection attacks.

So you learn a new trick – a new way of protecting inputs.

And you call this ‘validation’, too

After all, it … uhh, kind of does the same thing. It stops injection attacks, so it must be validation.

What is this ‘new trick’?

This new trick is encoding, quoting, or in some way transforming the data, so the newly transformed data is safe to accept.

Every one of those apostrophes? Turn them into the sequence “'” if they’re going into HTML, or double them if they’re in a SQL string, or – and this is FAR better – use parameterised queries so you don’t have to even know how the input string is being encoded on its way into the SQL command.

Now your input can be validated – and injection attacks are stopped.

But it’s not validation any more

In fact, once you’ve encoded your inputs properly, your validation can be entirely open and empty! At least from the security standpoint, because you’ve made the string semantically entirely meaningless to the code in which it is to be embedded as data. There are no escape characters or sequences, because they, too, have been encoded or transformed into semantically safe data.

It’s encoding… or transformation.

And I happen to think it’s important to separate the two concepts of validation and encoding.

Validation is saying “yes” or “no” to the question “is this string ‘good’ data?” You can validate in a number of different ways, and with good defence in depth, you’ll validate at different locations, based on different knowledge about what is “good”. This matches very strongly with the primary dictionary definition of “validation” – it’s awesome when a technical term matches very closely with a common language term, because teaching it to others becomes easier.

Encoding doesn’t say “yes” or “no”, encoding simply takes whatever input it’s given, and makes it safe for the next layer to which the data will be handed.

Stop calling encoding “validation”

It’s not.

Padding Oracle 3–making it usable

Just a quick note, because I’ve been sick this week, but last weekend, I put a little more work into my Padding Oracle exploit tool.

You can find the new code up at https://github.com/alunmj/PaddingOracle, and because of all the refactoring, it’s going to look like a completely new batch of code. But I promise that most of it is just moving code from Program.cs into classes, and adding parsing of command-line arguments.

I don’t pretend to be the world’s greatest programmer by any stretch, so if you can tell me a better way to do what I’ve done here, do let me know, and I’ll make changes and post something about them here.

Also, please let me know if you use the tool, and how well it worked (or didn’t!) for you.

Arguments

The arguments currently supported are:

URL

The only parameter unadorned with an option letter – this is the URL for the resource the Padding Oracle code will be pounding to test guesses at the encrypted code.

-c ciphertext

Also, –cipher. This provides a .NET regular expression which matches the ciphertext in the URL.

-t encoding:b64|b64URL|hex|HEX

Also, –textencoding, –encoding. This sets the encoding that’s used to specify the ciphertext (and IV) in the URL. The default is b64

  • b64 – standard base64, URL encoded (so ‘=’ is ‘%3d’, ‘+’ is ‘%2b’, and ‘/’ is ‘%2f’)
  • b64URL – “URL safe” base64, which uses ‘!’, ‘-‘ and ‘~’ instead of the base64 characters that would be URL encoded.
  • hex – hexadecimal encoding with lower case alphabetic characters a-f.
  • HEX – hexadecimal encoding with upper case alphabetic characters A-F.

-i iv

Also, –iv. This provides a .NET regular expression which matches the IV in the URL if it’s not part of the ciphertext.

-b blocksize

Also, –blocksize. This sets the block size in bytes for the encryption algorithm. It defaults to 16, but should work for values up to 32.

-v

Also, –verbose. Verbose – output information about the packets we’re decrypting, and statistics on speed at the end.

-h

Also, –help. Outputs a brief help message

-p parallelism:-1|1|#

Also –parallelism. Dictates how much to parallelise. Specifying ‘1’ means to use one thread, which can be useful to see what’s going on. –1 means “maximum parallelisation” – as many threads as possible. Any other integer is roughly akin to saying “no more than this number of threads”, but may be overridden by other aspects of the Windows OS. The default is –1.

-e encryptiontext

Instead of decrypting, this will encrypt the provided text, and provide a URL in return that will be decrypted by the endpoint to match your provided text.

Examples

These examples are run against the WebAPI project that’s included in the PadOracle solution.

Example 1

Let’s say you’ve got an example URL like this:

http://localhost:31140/api/encrypted/submit?iv=WnfvRLbKsbYufMWXnOXy2Q%3d%3d&ciphertext=087gbLKbFeRcyPUR2tCTajMQAeVp0r50g07%2bLKh7zSyt%2fs3mHO96JYTlgCWsEjutmrexAV5HFyontkMcbNLciPr51LYPY%2f%2bfhB9TghbR9kZQ2nQBmnStr%2bhI32tPpaT6Jl9IHjOtVwI18riyRuWMLDn6sBPWMAoxQi6vKcnrFNLkuIPLe0RU63vd6Up9XlozU529v5Z8Kqdz2NPBvfYfCQ%3d%3d

This strongly suggests (because who would use “iv” and “ciphertext” to mean anything other than the initialisation vector and cipher text?) that you have an IV and a ciphertext, separate from one another. We have the IV, so let’s use it – here’s the command line I’d try:

PadOracle "http://localhost:31140/api/encrypted/submit?iv=WnfvRLbKsbYufMWXnOXy2Q%3d%3d&ciphertext=087gbLKbFeRcyPUR2tCTajMQAeVp0r50g07%2bLKh7zSyt%2fs3mHO96JYTlgCWsEjutmrexAV5HFyontkMcbNLciPr51LYPY%2f%2bfhB9TghbR9kZQ2nQBmnStr%2bhI32tPpaT6Jl9IHjOtVwI18riyRuWMLDn6sBPWMAoxQi6vKcnrFNLkuIPLe0RU63vd6Up9XlozU529v5Z8Kqdz2NPBvfYfCQ%3d%3d" -c "087gb.*%3d%3d" –i "WnfvRL.*2Q%3d%3d"

This is the result of running that command:

capture20181111175736366

Notes:

  • The IV and the Ciphertext both end in Q==, which means we have to specify the regular expressions carefully to avoid the expression being greedy enough to catch the whole query string.
  • I didn’t use the “-v” output to watch it run and to get statistics.
  • That “12345678” at the end of the decrypted string is actually there – it’s me trying to push the functionality – in this case, to have an entirely padding last block. [I should have used the letter “e” over and over – it’d be faster.]

Example 2

Same URL, but this time I want to encrypt some text.

Our command line this time is:

PadOracle "http://localhost:31140/api/encrypted/submit?iv=WnfvRLbKsbYufMWXnOXy2Q%3d%3d&ciphertext=087gbLKbFeRcyPUR2tCTajMQAeVp0r50g07%2bLKh7zSyt%2fs3mHO96JYTlgCWsEjutmrexAV5HFyontkMcbNLciPr51LYPY%2f%2bfhB9TghbR9kZQ2nQBmnStr%2bhI32tPpaT6Jl9IHjOtVwI18riyRuWMLDn6sBPWMAoxQi6vKcnrFNLkuIPLe0RU63vd6Up9XlozU529v5Z8Kqdz2NPBvfYfCQ%3d%3d" -c "087gb.*%3d%3d" –i "WnfvRL.*2Q%3d%3d" –e "Here’s some text I want to encrypt"

When we run this, it warns us it’s going to take a very long time, and boy it’s not kidding – we don’t get any benefit from the frequency table, and we can’t parallelise the work.

capture20181111215602359

And you can see it took about two hours.

Padding Oracle 2: Speeding things up

Last time, I wrote about how I’d decided to write a padding oracle exploit tool from scratch, as part of a CTF, and so that I could learn a thing or two. I promised I’d tell you how I made it faster… but first, a question.

Why build, when you can borrow?

One question I’ve had from colleagues is “why didn’t you just run PadBuster?”

It’s a great question, and in general, you should always think first about whether there’s an existing tool that will get the job done quickly and easily.

Time

Having said that, it took me longer to install PadBuster and the various language components it required than it did to open Visual Studio and write the couple of hundred lines of C# that I used to solve this challenge.

So, from a time perspective, at least, I saved time by doing it myself – and this came as something of a surprise to me.

The time it used up was my normally non-productive time, while I’m riding the bus into Seattle with spotty-to-zero network connectivity (there’s none on the bus, and my T-Mobile hot-spot is useful, but neither fast nor reliable down the I-5 corridor). This is time I generally use to tweet, or to listen to the BBC.

Interest

I just plain found it interesting to take what I thought I knew about padding oracles, and demonstrate that I had it solidly in my head.

That’s a benefit that really can’t be effectively priced.

Plus, I learned a few things doing it myself:

  • Parallelisation in C# is easier than it used to be.
  • There’s not much getting around string conversions in trying to speed up the construction of a base64-encoded URL, but then again, when executing against a crypto back-end, that’s not your bottleneck.
  • Comments and blank lines are still important, especially if you’re going to explain the code to someone else.

Performance

The other thing that comes with writing your own code is that it’s easier to adjust it for performance – you know where the bottlenecks might lie, and you can dive in and change them without as much of a worry that you’re going to kill the function of the code. Because you know at a slightly more intuitive level how it all works.

You can obviously achieve that intuitive level over time with other people’s code, but I wasn’t really going to enjoy that.

Looking at some of the chat comments directed at the PadBuster author, it’s clear that other people have tried to suggest optimisations to him, but he believes them not to be possible.

Guessing

Specifically, he doesn’t see that it’s possible to use guesses as to the plaintext’s likely contents to figure out what values should be in the ciphertext. You just plug the values 0..255 into the N-1 ciphertext block until your padding error from the N block goes away, and then that value can be XORed with the padding value to get the intermediate value from the N block. Then the intermediate value gets XORed with the original ciphertext value from the N-1 block to give the original plaintext.

Let’s see how that works in the case of the last block – where we’re expecting to see some padding anyway. Let’s say our block size is 4. Here’s what two of our ciphertext blocks might look like:

CN-1 CN
0xbe 0x48 0x45 0x30 0x71 0x4f 0xcc 0x63

Pretty random, right? Yeah, those are actually random numbers, but they’ll work to illustrate how we work here.

We iterate through values of CN-1[3] from 0..255, until we get a response that indicates no padding errors.

0x30 comes back without any padding errors. That’s convenient. So, we’ve sent “be484530714fcc63”, and we know now that we’ve got a padding byte correct. Buuut that isn’t the only right padding byte, because this is the last block, which also has a valid padding byte.

In fact, we can see that 0x30 matches the original value of the CN-1 block’s last byte, so that’s not terribly useful. Our padding count has a good chance of not being 1, and we’re trying to find the value that will set it to 1.

Keep iterating, and we get 0x32, giving us a request that doesn’t contain a padding exception. Two values. Which one made our padding byte 0x1, so we can use it to determine the intermediate value?

The only way we get two matches will be because the real plaintext ends in a padding count that isn’t 0x1. One of those values corresponds to 0x1, the other corresponds to the padding count, which could be 0x2..0x4. [Because we’re using four byte blocks as an example – a real-life example might have a 16-byte block size, so the padding count could be up to 0x10]

The clue is in the original plaintext – 0x30 MUST be the value that corresponds to the original padding count, so 0x32 MUST correspond to 0x1.

[If the original padding count was 0x1, we would only find one value that matched, and that would be the original value in CN-1]

That means the Intermediate value is 0x32 XOR 0x1 = 0x33 – which means the plaintext value is 0x3 – there’s three bytes of padding at the end of this block.

We can actually write down the values of the last three plaintext and intermediate blocks now:

CN-1 CN
0xbe 0x48 0x45 0x30 0x71 0x4f 0xcc 0x63
IN
?? 0x4b 0x46 0x33
C’N-1 PN
?? 0x4f 0x42 0x37 ?? 0x3 0x3 0x3

Wow – that’s easy! How’d we do that? Really simple. We know the last padding must be three bytes of 0x3, so we write those down. Then the intermediate bytes must be the XOR of 0x3 with the value in the CN-1 block.

[I chose in the code, instead of just “writing down” the values for each of those bytes, to check each one as I did so, to make sure that things were working. This adds one round-trip for each byte of padding, which is a relatively low cost, compared to the rest of the process.]

Now, if we want to detect the next byte, we want to change the last three bytes of CN-1, so they’ll set the PN values to 0x4, and then iterate through the target byte until we get a lack of padding errors.

So, each new value of the last few bytes of CN-1 will be C’[i] = C[i] XOR 0x3 XOR 0x4 – taking the value in the original, XORing it with the original plaintext, and then with the desired plaintext to get a new value for the ciphertext.

I’ve put those values of C’N-1 in the table above.

This trick doesn’t just stop with the padding bytes, though. I’m going to guess this is a JSON object, so it’s going to end with a ‘}’ character (close-brace), which is 0x7d.

So, C’ = C XOR 0x7d XOR 0x4 = 0xbe XOR 0x7d XOR 0x4 = 0xc7.

Let’s try that – we now send “c74f4237” – no padding error!

A successful guess for the last four bytes of PN. Now we can fill in more of the table:

CN-1 CN
0xbe 0x48 0x45 0x30 0x71 0x4f 0xcc 0x63
IN
0xba 0x4b 0x46 0x33
C’N-1 PN
0xc7 0x4f 0x42 0x37 0x7d 0x3 0x3 0x3

Awesome.

That does require me making the right guess, surely, though?

Yes, but it’s amazing how easy it is to either make completely correct guesses, or just pick a set of values that are more likely to be good guesses, and start by trying those, failing back to the “plod through the rest of the bytes” approach when you need to.

I’ve coded an English-language frequency table into my padding oracle code, because that was appropriate for the challenge I was working on.

This code is available for you to review and use at https://github.com/alunmj/PaddingOracle/blob/master/PadOracle/Program.cs

You can imagine all kinds of ways to improve your guesses – when proceeding backward through a JSON object, for instance, a ‘}’ character will be at the end; it’ll be preceded by white space, double quotes, or brackets/braces, or maybe numerics. A 0x0a character will be preceded by a 0x0d (mostly), etc.

Parallelisation

The other big performance improvement I made was to parallelise the search. You can work on one block entirely independently from another.

I chose to let the Parallel.For() function from C# decide exactly how it was going to split up work between different blocks, and the result is a whole lot faster. There are some wrinkles to manage when parallelising an algorithm, but I’m not going to get into that here. This is not a programming blog, really!

15x performance improvement

I figured I’d put that in big letters, because it’s worth calling out – the parallelisation alone obviously multiplies your performance by the number of cores you’ve got (or the number of cores the web server has, if it’s underpowered), and the predictive work on the text does the rest. Obviously, the predictive approach only works if you can separate between “likely” and “unlikely” characters – if the plaintext consists of random binary data, you’re not going to get much of a benefit. But most data is formatted, and/or is related to English/Latin text.

Bonus stage – use a decryptor to encrypt!

I haven’t published the code for this part yet, but you can use this same breach to encrypt data without knowing the key.

This is really fun and simple once you get all the previous stuff. Here goes.

Let’s encrypt a block.

Encrypting a block requires the generation of two ciphertext blocks from one plaintext block. What the second block is, actually doesn’t matter. We can literally set it to random data, or (which is important) specific data of our choosing.

The first block of the pair, acting like an IV, we can set to 0. There’s a reason for this which we’ll come to in a minute.

With these two initial blocks, we run the decrypter. This will give us a ‘plaintext’ block as output. Remember how the intermediate block is the plaintext block XORed with the first of the pair of blocks? Well, because we set that first block to all zeroes, that means the plaintext block IS the same as the intermediate block. And that intermediate block was generated by decrypting the second block of the pair. In order for that decryption to result in the plaintext we want instead, we can simply take the intermediate block, XOR it with the plaintext block we want, and then put that into the first ciphertext block. [We’re actually XORing this with the first ciphertext block, but that’s a straight copy in this case, because the first ciphertext block is zeroes.]

Now, draw the rest of the owl

Do the same thing for each of the rest of the blocks.

Sadly, there’s no parallelising this approach, and the guessing doesn’t help you either. You have to start with CN (randomly generated) and CN-1 (deduced with the approach above), then when you’ve established what CN-1 is, you can use the same approach to get CN-2, and so on back to the IV (C0). So this process is just plain slow. But it allows you to encrypt an arbitrary set of data.

Padding Oracles for a thousand, please

We did a CTF at work.

I have to say it was loads of fun – I’ve never really participated in a CTF before, so it was really good to test my hacking skills against my new colleagues.

We had one instruction from my manager – “don’t let the interns beat you”. I was determined to make sure that didn’t happen, but I was also determined to share as much knowledge and excitement for InfoSec as possible. This meant that once or twice, I may have egged an intern on in the face of the fact that they were about to discover it anyway, and it just seemed like a really good way to keep them interested.

This is not that story.

This is about me turning the corner from knowing about a security failure, to understanding how it works. Let’s see if I can help you guys understand, too.

Tales from the Crypto

That’s the title of my blog, and there’s not a whole lot of cryptography here. It’s just a play on words, which was a little more relevant when I first started the blog back in 2005. So here’s some crypto, at last.

There’s several aspects to cryptography that you have to get right as a developer:

  • Identify whether you’re doing hashing, signing, encryption, encoding, etc.
  • If you have a key, create and store it securely
  • Pick correct algorithms – modern algorithms with few known issues
  • Use the algorithms in a way that doesn’t weaken them

Having tried to teach all of these to developers in various forms, I can tell you that the first one, which should be the simplest, is still surprisingly hard for some developers to master. Harder still for managers – the number of breach notifications that talk about passwords being “encrypted” is a clear sign of this – encrypted passwords mean either your developers don’t understand and implemented the wrong thing, or your manager doesn’t understand what the developer implemented and thinks “encrypted sounds better than hashed”, and puts that down without checking that it’s still technically accurate.

Key creation (so it’s not predictable), and storage (so it can’t be found by an attacker) is one of those issues that seems to go perennially unsolved – I’m not happy with many of the solutions I’ve seen, especially for self-hosted services where you can’t just appeal to a central key vault such as is currently available in all good cloud platforms.

Picking correct algorithms is a moving target. Algorithms that were considered perfectly sound ten or twenty years ago are now much weaker, and can result in applications being much weaker if they aren’t updated to match new understanding of cryptography, and processor and memory speed and quantity improvements. You can store rainbow tables in memory now that were unthinkable on disk just a decade or two ago.

Finally, of course, if all that wasn’t enough to make cryptography sound really difficult (spoiler: it is, which is why you get someone else to do it for you), there are a number of ways in which you can mess up the way in which you use the algorithm.

Modes, block-sizes, and padding

There are a large number of parameters to set even when you’ve picked which algorithms you’re using. Key sizes, block sizes, are fairly obvious – larger is (generally) better for a particular algorithm. [There are exceptions, but it’s a good rule of thumb to start from.]

There are a number of different modes available, generally abbreviated to puzzling TLAs – ECB, CFB, OFB, CBC, GCM, CTR, and so on and so forth. It’s bewildering. Each of these modes just defines a different order in which to apply various operations to do things like propagating entropy, so that it’s not possible to infer anything about the original plaintext from the ciphertext. That’s the idea, at least. ECB, for instance, fails on this because any two blocks of plaintext that are the same will result in two blocks of ciphertext that are the same.

And if you’re encrypting using a block cipher, you have to think about what to do with the last block – which may not be a complete block. This requires that the block be filled out with “padding” to make a full block. Even if you’re just filling it out with zeroes, you’re still padding – and those zeroes are the padding. (And you have to then answer the question “what if the last block ended with a zero before you padded it?”)

There’s a number of different padding schemes to choose from, too, such as “bit padding”, where after the last bit, you set the next bit to 1, and the remaining bits in the block to 0. Or there’s padding where the last byte is set to the count of how many padding bytes there are, and the remaining bytes are set to 0 – or a set of random bytes – or the count repeated over and over. It’s this latter that is embodied as PKCS#5 or PKCS#7 padding. For the purposes of this discussion, PKCS#7 padding is a generalised version of PKCS#5 padding. PKCS#5 padding works on eight-byte blocks, and PKCS#7 padding works on any size blocks (up to 256 bytes, presumably).

So, if you have a three-byte last block, and the block size is 16 bytes, the last block is ** ** ** 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d 0x0d (where “**” represents the last three bytes of data, and 0x0d represents the hexadecimal value for 13, the number of bytes in the padding). If your last block is full, PKCS#7 covers this by making you create an extra 16-byte block, with the value 0x10 (decimal 16) in every byte.

Tying this in to the CTF

It’s not at all unlikely that you wind up with the scenario with which we were presented in the CTF – a service that communicated with AES encryption, in CBC mode, using PKCS#7 padding. The fact that this was described as such was what tipped me off in the first place. This is the perfect setup for a Padding Oracle attack.

An Oracle is simply a device/function/machine/system/person that you send a request to, and get a response back, and which gives you some information as a result. The classical Oracles of Ancient Greece & Roman times were confusing and unhelpful at best, and that’s really something we want from any cryptographic oracle. The term “Random Oracle” refers to a hypothetical system which returns random information to every query. A good cryptographic system is one that is indistinguishable from a Random Oracle.

Sadly, CBC with PKCS#7 padding is generally very different from a Random Oracle. It is a Padding Oracle, because it will tell us when the padding is correct or incorrect. And that’s our vulnerability.

At this point, I could have done what one of my colleagues did, and download PadBuster, choosing parameters and/or modifying code, to crack the encryption.

But… I’ve been attacking this CTF somewhat … non-traditionally, using tools other than the normal ones, and so I thought I’d try and understand the algorithm’s weaknesses, and implement my own attack. I wrote it on the bus on my way into work, and was pleased to see when I got in that it worked – albeit slowly – first time.

How CBC and PKCS#5/7 is vulnerable

When decrypting each block using CBC, we say that PN = DK(CN)⊕CN-1 – which is just a symbolic way of saying that the recipient Decrypts (with key “K”) the current Ciphertext block (block N), and then XORs the result with the previous Ciphertext block (the N-1th block). Let’s also assume that we’re only decrypting those two blocks, N-1 and N, with N being the last block provided to the recipient.

In other modes, the padding check may not deliver the helpful information we’re looking for, but CBC is special. The way CBC decrypts data is to decrypt the current block of ciphertext (CN), which creates an intermediate block DK(CN). That intermediate block is combined with the previous ciphertext block, CN-1, to give the plaintext block, PN. This combining of blocks is done using the XOR (exclusive-or) operation, which has interesting properties any developer should be familiar with. Particularly, it’s important to note that XOR (represented here as “⊕”) is reversible. If X⊕Y=Z, you know also that Z⊕Y=X and Z⊕X=Y. This is one of the reasons the XOR operation is used in a lot of cryptographic algorithms.

If we want to change things in the inputs to produce a different output, we can really only change two things – the current and the previous block of Ciphertext – CN and CN-1. We should really only alter one input at a time. If we alter CN, that’s going to be decrypted, and a small change will be magnified into a big difference to the DK(CN) value – all the bytes will have changed. But if we alter CN-1, just a bit, what we wind up with is a change in the plaintext value PN which matches that change. If we alter the 23rd bit of CN-1, it will alter the 23rd bit of PN, and only that one bit. Now if we can find what we’ve changed that bit to, we can then figure out what that means we must have changed it from.

If we change the last byte of CN-1, to create C’N-1 (pronounced “C prime of N minus 1”) and cycle it through all the possible values it can take, the decryption will occur, and the recipient will reject our new plain text, P’N (“P prime of N”) because it is poorly formed – it will have a bad padding. With one (two, but I’ll come to that in a minute) notable exception. If the last byte of the plaintext decrypted is the value 0x01, it’s a single byte of padding – and it’s correct padding. For that value of the last byte of C’N-1, we know that the last byte of P’N is 1. We can rewrite PN = DK(CN)⊕CN-1 as DK(CN) = CN-1⊕PN – and then we can put the values in for the last byte: DK(CN)[15] = C’N-1[15]⊕0x01.

Let’s say, for illustration’s sake, that the value we put in that last byte of C’N-1 was 0xa5, when our padding was accepted. That means DK(CN)[15] = 0xa5 ⊕ 0x01 = 0xa4. Note the lack of any “prime” marks there – we’ve figured out what the original value of the decrypted last byte was. Note that this isn’t the same as the last byte of the plain text. No, we get that by taking this new value and XORing it with the original last byte of the previous block of ciphertext – that’s CN-1[15]. For illustration, let’s say that value is 0xc5. We calculate PN[15] = DK(CN)[15]⊕CN-1[15] = 0xa4⊕0xc5 = 0x61. That’s the lower case letter ‘a’.

OK, so we got the first piece of plaintext out – the last byte.

[Remember that I said I’d touch on another case? If CN is the original last block of ciphertext, it already contains valid padding! But not necessarily the 0x01 we’re trying to force into place.]

Let’s get the next byte!

Almost the same process is used to get the next byte, with a couple of wrinkles. First, obviously, we’re altering the second-to-last byte through all possible values. Second, and not quite so obvious, we have to tweak the last byte once as well, because we’re looking to get the sequence 0x02 0x02 (two twos) to happen at the end of P’N. The last byte of C’N-1 to achieve this is simply the last byte of C’N-1 that we used to get 0x01, XORed by 0x03 (because that’s 0x02 ⊕ 0x01). In our illustrative example, that’s 0xa6.

And the next, and the next…

Each time, you have to set the end values of the ciphertext block, so that the end of P’N will look like 0x03 0x03 0x03, 0x04 0x04 0x04 0x04, etc, all the way up to 0x10 … 0x10 (sixteen 16s).

Code, or it didn’t happen

So here’s the 200 lines or so that I wrote on the bus. I also wrote a test harness so that this would work even after the CTF finished and got shut down. You’ll find that in the same repo.

I’ve massaged the code so it’s easier to understand, or to use as an explainer for what’s going on.

I plan on expanding this in a couple of ways – first, to make it essentially command-line compatible with ‘PadBuster’, and second, to produce a graphical demo of how the cracking happens.

And in the next post, I’m going to talk a little about how I optimised this code, so that it was nearly 15x faster than PadBuster.

One Simple Thing

I’ve been a little absent from this blog for a while, mostly because I’ve been settling in to a new job where I’ve briefly changed my focus almost completely from application security to being a software developer.

The blog absence is going to change now, and I’d like to start that with a renewed effort to write something every week. In addition to whatever grabs my attention from the security news feeds I still suck up, I want to get across some of knowledge and approaches I’ve used while working as an application security guy. I’ll likely be an application security guy in my next job, whenever that is, so it’ll stand me in good stead to write what I think.

“One Simple Thing”

The phrase “One Simple Thing” underscores what I try to return to repeatedly in my work – that if you can get to the heart of what you’re working on, everything else flows easily and smoothly.

This does not mean that there’s only one thing to think about with regards to security, but that when you start asking clarifying questions about the “one simple thing” that drives – or stops – a project in the moment, it’s a great way to make tremendous progress.

The Simplest Security Thing

I’ll start by discussing the One Simple Thing I pick up by default whenever I’m given a security challenge.

What are we protecting?

This is the first question I ask on joining a new security team – often as early as the first interviews. Everyone has a different answer, and it’s a great way to find out what approaches you’re likely to encounter. The question also has several cling-on questions that it demands be asked and answered at the same time:

Why are we protecting it?

Who are we protecting it from?

Why do they want it?

Why shouldn’t they get it?

What are our resources?

These come very quickly out of the One Simple Thing of “what are we protecting?”.

Here’s some typical answers:

  • Our systems need to continue running and be accessible at all times
  • We need to keep making money
  • We need to stop [the risk of] losing money
  • Our customers trust us, and we need that to continue
  • We operate in a regulatory compliance climate, and need to keep our lawsuits down to close to zero
  • We don’t even have a product yet, and haven’t a clue what we need to secure
  • We don’t want a repeat of last week’s / month’s / year’s very public security / privacy debacle
  • We want to balance the amount we spend on security staff against the benefit it has to our bottom line – and demonstrate results

You can see from the selection of answers that not everyone has anything like the same approach, and that they don’t all line up exactly under the typical buckets of Confidentiality, Integrity and Availability.

Do you think someone can solve your security issues or set up a security team without first finding out what it is you’re protecting?

Do you think you can engage with a team on security issues without understanding what they think they’re supposed to be protecting?

Answers vary

You’ve seen from my [short] list above that there are many answers to be had between different organisations and companies.

I’d expect there to be different answers within an organisation, within a team, within a meeting room, and even depending on the time I ask the question.

“What are we protecting” on the day of the Equifax leak quickly becomes a conversation on personal data, and the damaging effect of a leak to “customers”. [I prefer to call them “data subjects”, because they aren’t always your customers.]

On the day that Yahoo gets bought by Verizon for substantially less than initially offered, the answer becomes more about company value, and even perhaps executive stability.

Next time you’re confused by a security problem, step back and ask yourself – and others – “What are we protecting?” and see how much it clarifies your understanding.

My contribution to Movember

Whether it’s “No-shave November” or “Movember”, there’s a lot of attention given this time of year to men’s health in general, and cancer in particular.

I don’t take part in either of these events, partly because I don’t like the way a beard / moustache feels, but mostly because I already spend my November extremely aware of men’s cancer issues.

So, let me tell you, on this International Men’s Day, how my right testicle tried to kill me. Warning – rude words ahead, including the dreaded “c-word”.

Here we go gathering nuts in May

A little over fifteen years ago, I was living a fantastic life.

A wife, a six-year-old son, a house in a nice suburb of Austin, working from home on my own projects, and making enough money with those projects to justify doing so.

As anyone who’s ever watched any “funny home video” shows on TV will tell you, the purpose of a six year old is to throw things at your crotch, or to swing things at your crotch, or to hit you in your crotch, or to head-butt you in your crotch.

OK, so that’s maybe not his sole purpose, but that year it seemed like this was happening more often than usual. It wasn’t, of course, but it was noticeable that I was … feeling the impact a little more keenly than usual.

It takes balls to go see a doctor

I checked, my wife checked, and we concurred – something was definitely not as it had been. I mean, everyone knows that a man’s testicles aren’t the same size and shape on each side, and I’d been blessed with a particularly disparate pair from my teenage years.

But this was something new – swelling that just increased gradually, and a firmness that was inappropriately placed.

It was time to see the doctor.

Even knowing this, and reading about how badly – and how quickly – testicular diseases can impact men, it was extraordinarily difficult to face the task of picking up the phone, calling to speak to a doctor’s [female] receptionist, and tell them exactly why I wanted to come and see the doctor. Nonetheless, I girded my loins as much as I could, swallowed hard, and made the call.

The key is to remind yourself that this is probably the fifth call that receptionist has received this week on the same topic, and that she wouldn’t be working in a doctor’s office if she weren’t ready to hear medical terms briefly describing anatomical parts. I’m surprised how quickly I came to this conclusion, given how many decades it took me to learn that when a doctor asks “so, how are you doing today?”, they actually want to hear the details, rather than “oh, fine, thanks, and you?”

“Good luck”

The doctor’s visit was quick and clinical, just what you’d hope for. A flashlight applied to the nether regions, in much the same way you might check a hen’s egg for occupants, a little uncomfortable palpation, and a quick inspection of nearby things while you have your underpants down.

“You’ve got a hydrocele,” he said, doing that thing with the rubber gloves where you snap them off, startling an already nervous patient. “A short surgery should fix that.”

Relief. Nothing quite as horrifying or scary as I had expected.

“I’ll set you up with a urologist, and we’ll get that taken care of in the next couple of weeks. Good luck.”

I’d never had a doctor wish me “good luck” before, and it quite chilled me.

You’re never ready for the C-word

I visited the urologist, got set up for surgery, and discussed plans with my wife.

It was always in the back of my head that this could be something more than merely having a little extra fluid to drain.

So we talked about the C-word. I think of it that way, because on all the forms since, this is the one word the medical establishment goes out of its way to avoid writing in full. There are long words, foreign words, culturally taboo words, and all of them are written in full on some or other medical forms. There are abbreviations, but no word more than this one results in hardened medical professionals ceding to decency and refusing to name it in full:

Cancer

You kind of guessed that was going to be the result, right?

We kind of did, too, and had discussed the idea that if there was any cancerous signs, that quite frankly I preferred being a living eunuch, if that was necessary, to being a dead, but otherwise intact, cancerous corpse. It seems such an obvious decision to make, but it’s still a very hard one to bring to bear.

And my wife did so on her own.

Because the only way to tell if the testicle looked cancerous was while I was under general anaesthetic in the operating room.

And sure enough, the doctor came out mid-surgery, while I’m away with the fairies, to talk to my wife about the situation at hand. I can only imagine how that conversation went, so I shan’t try to replay it here. I can only express how truly grateful I am that my wife gave consent to do what we had already discussed – to remove that cancerous nasty thing and send it to a lab for study.

So I woke up to a woman looking unutterably upset at the prospect that she’d had to make life-altering medical decisions, for which I have always been truly grateful. There literally isn’t a day that goes by that I wish she’d made any other choice.

And yet even to this day, it still bothers her – that’s how upsetting it is to be on the outside of this disease.

It wasn’t much fun on the inside, either, to be honest, and that’s my story which I can tell.

Time to hulk up!

This was all in the week before Thanksgiving, 2002, a year when the first movie  featuring an all-CGI Incredible Hulk was being advertised on the TV.

Poor Bruce Banner, strapped to a table, unable to move, while gamma rays coursed through his body under the control of a malfunctioning computer, turning him into the hangriest super-anti-hero ever.

After a trip to San Antonio, during which I felt every pothole on the I-35 from Austin, to have Thanksgiving dinner with my inlaws, we returned home and started observational and preventive treatment as follow up for good ole “testicular C”.

First, the tattoos. I have five tattoos now, each one a single dot, in the shape of a cross.

For targetting.

I wasn’t exactly strapped to a table, but I was unable to move, while gamma rays coursed through my body, laser cross-hairs ensuring that the focused radiation hit only the right parts of my intestines. They call it radiotherapy, and when you go to an oncologist / radiologist to get radiotherapy in Austin in 2002, you sit in a waiting room surrounded by inspirational photos of Lance Armstrong. Whatever you feel about his drug use while winning the Tour de France competing against others who almost certainly used most of the same drugs themselves, he continues to be inspirational to many cancer survivors like myself, simply for having survived enough to be able to ride a bike.

Testicular cancer doesn’t travel across, it goes up – so the process is, remove the testicle, fry the intestines lightly, and monitor the chest with ongoing X-rays just to make sure. Removing the testicle is called an “orchiectomy” – true story, the orchid plant is named after testicles, because that’s what the plant’s bulbs allegedly look like. This is why testicular cancer awareness pins are orchid-coloured.

Insurance companies make me sick

One of the side effects you think of with any cancer treatment is serious nausea, and this is definitely the case with radiotherapy. It makes you feel uncomfortably unwell. American medical care being run by insurance companies, I was given leave to have fifteen anti-nausea pills. For 25 days of treatment. During which I’d need multiple pills per day.

The only thing to do – snack on saltine crackers, and where possible actually cook some meals at least for my son. Bland food was really pretty much all I could manage. To this day, he quite rightly refuses to eat chicken and rice.

Because my wife had to return to work, and was travelling as a result, I drove myself to appointments, and that’s probably my biggest mistake in all of this – the American Cancer Society offers free rides to patients attending hospital and doctor appointments, and has many other services besides. Take advantage of them, I donate to them specifically for you to use their services.

Cat scans and pregnancy tests

After that, every six months to a year, I’d get a CT scan of my abdomen, and a blood test every month. CT scans are not the most comfortable of procedures, particularly with the iodine contrast dyes.

Once in a while, the person administering the blood test would question whether the test was really for me. On my doctor’s advice, I would ask them to re-check the form. It turns out that I was basically being given a monthly pregnancy test, to ensure the cancer wasn’t coming back.

More surgeries, and more insurance

Still more surgeries were in my future over the next year – apparently, skin likes to stick to skin in unusual situations and in uncomfortable ways.

The insurance company raised our rates – presumably in line with regular price rises, but to the point where it was difficult to afford. After all, even back before the ACA, it wasn’t right to raise insurance rates just because someone got sick. However, what WAS legal back then was the ability of other insurance providers to call the cancer a pre-existing condition, and to use that as reason to either refuse to sell me a policy, or to jack up the rates. Personal insurance policies are expensive to begin with, but when you can’t shop around (or threaten to do so), you’re really out of luck.

And that’s why I took the Microsoft job, and jacked in my personal business for the most part. Because American health insurance kills the American dream more often than it deserves to.

And now, the lesson

So, the final lesson – and there always is one – is that if you are a man, aged between twenty and thirty-five, or you know someone who fits, or will fit, that description, know that it’s important to check your health – actually touch and feel your body, particularly your ‘man parts’ – on a regular basis. When things change in a way that isn’t expected, it’s really important to give your doctor a call. That week. Perhaps even that day that you notice it. The person who takes your call has heard it all before – and if you aren’t comfortable talking to them, you can actually ask to speak to a nurse, a physician’s assistant, and even specifically to a man, if that’s what you need to feel comfortable to cover this.

Your doctor will tell you if it’s important, or something not to worry about. They’ll give you advice on what to watch for in future, and wish you good luck if you need it.

Above all, don’t literally die of embarrassment.

Lessons from my first official bounty

Sometimes, it’s just my job to find vulnerabilities, and while that’s kind of fun, it’s also a little unexciting compared to the thrill of finding bugs in other people’s software and getting an actual “thank you”, whether monetarily or just a brief word.

About a year ago, I found a minor Cross-Site Scripting (XSS) flaw in a major company’s web page, and while it wasn’t a huge issue, I decided to report it, as I had a few years back with a similar issue in the same web site. I was pleased to find that the company was offering a bounty programme, and simply emailing them would submit my issue.

Lesson 1: The WAF… it does nothing!

The first thing to notice, as with all XSS issues, is that there were protections in place that had to be got around. In this case, some special characters or sequences were being blocked. But not all. And it’s really telling that there are still many websites which have not implemented widespread input validation / output encoding as their XSS protection. So, while the WAF slowed me down even when I knew the flaw existed, it only added about 20 minutes to the exploit time. So, my example had to use “confirm()” instead of “alert()” or “prompt()”. But really, if I was an attacker, my exploit wouldn’t have any of those functions, and would probably include an encoded script that wouldn’t be detected by the WAF either. WAFs are great for preventing specific attacks, but aren’t a strong protection against an adversary with a little intelligence and understanding.

Lesson 2: A quick response is always welcome

My email resulted in an answer that same day, less than an hour after my initial report. A simple “thank you”, and “we’re forwarding this to our developers” goes a long way to keeping a security researcher from idly playing with the thought of publishing their findings and moving on to the next game.

Lesson 3: The WAF… still does nothing!

In under a week, I found that the original demo exploit was being blocked by the WAF – but if I replaced “onclick” with “oclick”, “onmouseover” with “omouseover”, and “confirm” with “cofirm”, I found the blocking didn’t get in the way. Granted, since those aren’t real event handlers or JavaScript functions, I can’t use those in a real exploit, but it does mean that once again, all the WAF does is block the original example of the attack, and it took only a few minutes again to come up with another exploit string.

Lesson 4: Keep communicating

If they’d told me “hey, we’re putting in a WAF rule while we work on fixing the actual bug”, I wouldn’t have been so eager to grump back at them and say they hadn’t fixed the issue by applying their WAF and by the way, here’s another URL to exploit it. But they did at least respond to my grump and reassure me that, yes, they were still going to fix the application.

Lesson 5: Keep communicating

I heard nothing after that, until in February of this year, over six months later, I replied to the original thread and asked if the report qualified for a bounty, since I noticed that they had actually fixed the vulnerability.

Lesson 6: Keep communicating

No response. Thinking of writing this up as an example of how security researchers still get shafted by businesses – bear in mind that my approach is not to seek out bounties for reward, but that I really think it’s common courtesy to thank researchers for reporting to you rather than pwning your website and/or your customers.

Lesson 7: Use a broker

About a month later, while looking into other things, I found that the company exists on HackerOne, where they run a bug bounty. This renewed my interest in seeing this fixed. So I reported the email exchange from earlier, noted that the bug was fixed, and asked if it constituted a rewardable finding. Again, a simple “thanks for the report, but this doesn’t really rise to the level of a bounty” is something I’ve been comfortable with from many companies (though it is nice when you do get something, even if it’s just a keychain or a t-shirt, or a bag full of stickers).

Lesson 8: Keep communicating

3/14: I got a reply the next day, indicating that “we are investigating”.

3/28: Then nothing for two weeks, so I posted another response asking where things were going.

4/3: Then a week later, a response. “We’re looking into this and will be in touch soon with an update.”

4/18: Me: Ping?

5/7: Me: Hey, how are we doing?

5/16: Anything happening?

Lesson 9: Deliver

5/18: Finally, over two months after my report to the company through HackerOne, and ten months after my original email to the first bug bounty address, it’s addressed.

5/19: The severity of the bug report is lowered (quite rightly, the questionnaire they used pushed me to a priority of “high”, which was by no means warranted). A very welcome bounty, and a bonus for my patience – unexpected but welcome, are issued.

Lesson 10: Learn

The cheapest way to learn things is from someone else’s mistakes. So I decided to share with my readers the things I picked up from this experience.

Here are a few other lessons I’ve picked up from bug bounties I’ve observed:

Lesson 11: Don’t start until you’re ready

If you start a bug bounty, consider how ready you might be. Are you already fixing all the security bugs you can find for yourself? Are you at least fixing those bugs faster than you can find more? Do your developers actually know how to fix a security bug, or how to verify a vulnerability report? Do you know how to expand on an exploit, and find occurrences of the same class of bug? [If you don’t, someone will milk your bounty programme by continually filing variations on the same basic flaw]

Lesson 12: Bug Bounty programmes may be the most expensive way to find vulnerabilities

How many security vulnerabilities do you think you have? Multiply that by an order of magnitude or two. Now multiply that by the average bounty you expect to offer. Add the cost of the personnel who are going to handle incoming bugs, and the cost of the projects they could otherwise be engaged in. Add the cost of the developers, whose work will be interrupted to fix security bugs, and add the cost of the features that didn’t get shipped on time before they were fixed. Sure, some of that is just a normal cost of doing business, when a security report could come at you out of the blue and interrupt development until it’s fixed, but starting a bug bounty paints a huge target on you.

Hiring a penetration tester, or renting a tool to scan for programming flaws, has a fixed cost – you can simply tell them how much you’re willing to pay, and they’ll work for that long. A bug bounty may result in multiple orders of magnitude more findings than you expected. Are you going to pay them all? What happens when your bounty programme runs out of money?

Finding bugs internally using bug bashes, software scanning tools or dedicated development staff, has a fixed cost, which is probably still smaller than the amount of money you’re considering on putting into that bounty programme.

12.1: They may also be the least expensive way to find vulnerabilities

That’s not to say bug bounties are always going to be uneconomical. At some point, in theory at least, your development staff will be sufficiently good at resolving and preventing security vulnerabilities that are discovered internally, that they will be running short of security bugs to fix. They still exist, of course, but they’re more complex and harder to find. This is where it becomes economical to lure a bunch of suckers – excuse me, security researchers – to pound against your brick walls until one of them, either stronger or smarter than the others, finds the open window nobody saw, and reports it to you. And you give them a few hundred bucks – or a few thousand, if it’s a really good find – for the time that they and their friends spent hammering away in futility until that one successful exploit.

At that point, your bug bounty programme is actually the least expensive tool in your arsenal.

1 2 3 50