Introduction

When you have something digital, having backups is something fundamental to keep your data safe. There are many threats over there that can destroy your data: hardware failures, viruses, natural disasters are some of the ways to make all your data vanish from one moment to the other.

I use to keep my data in several places (you can never be 100% sure Smile), making cloud and local backups, and I can say that they have saved me more than once. For cloud backups, there are several services out there and I won’t discuss them, but for local backups, my needs are very specific, and I don’t need the fancy stuff out there (disk images, copying blocked data, and so on). I just need a backup that has these features:

  • Copies data in a compressed way – it would be better that it’s a standard format, like zip files, so I can open the backups with normal tools and don’t need to use the tool to restore data.
  • Allows the copy of selected folders in different drives. I don’t want to copy my entire disk (why keep a copy of the Windows installation, or the installed programs, if I can reinstall them when I need).
  • Allows the exclusion of files and folders in the copy (I just want to copy my source code, there is no need to copy the executables and dlls).
  • Allows incremental (only the files changed) or full backup (all files in a set)
  • Can use backup configurations (I want to backup only my documents or only my source code, and sometimes both)
  • Can be scheduled and run at specified times without the need of manually starting it.

With these requirements, I started to look for backup programs out there and I have found some free ones that didn’t do everything I wanted and some paid ones that did everything, but I didn’t want to pay what they were asking for. So, being a developer, I decided to make my own backup with the free tools out there.

The first requirement is a compressed backup, with a standard format. For zip files, I need zip64, as the backup files can be very large and the normal zip files won’t handle large files. So, I decided to use the DotNetZip library (https://dotnetzip.codeplex.com/), an open source library that is very simple to use and supports Zip64 files. Now I can go to the next requirements. That can be done with a normal .NET console program.

Creating the backup program

In Visual Studio, create a new Console Program and, in the Solution Explorer, right-click the References node and select “Manage NuGet packages” and add the DotNetZip package. I don’t want to add specific code for working with the command line options, so I added a second package, CommandLineParser (https://github.com/gsscoder/commandline), that does this for me. I just have to create a new class with the options I want and it does all the parsing for me:

class Options
{
    [Option(DefaultValue = "config.xml", 
      HelpText = "Configuration file for the backup.")]
    public string ConfigFile { get; set; }

    [Option('i', "incremental", DefaultValue= false,
      HelpText = "Does an increamental backap.")]
    public bool Incremental { get; set; }

    [HelpOption]
    public string GetUsage()
    {
        return HelpText.AutoBuild(this,
          (HelpText current) => HelpText.DefaultParsingErrorsHandler(this, current));
    }
}

To use it, I just have to pass the command line arguments and have it parsed:

var options = new Options();
CommandLine.Parser.Default.ParseArguments(args, options);

It will even give me a –help command line for help:

image

The next step is to process the configuration file. Create a new class and name it Config.cs:

public class Config
{
    public Config(string fileName)
    {
        if (!File.Exists(fileName))
            return;
        var doc = XDocument.Load(fileName);
        if (doc.Root == null)
            return;
        IncludePaths = doc.Root.Element("IncludePaths")?.Value.Split(';');
        ExcludeFiles = doc.Root.Element("ExcludeFiles")?.Value.Split(';') ?? new string[0] ;
        ExcludePaths = doc.Root.Element("ExcludePaths")?.Value.Split(';') ?? new string[0];
        BackupFile = $"{doc.Root.Element("BackupFile")?.Value}{DateTime.Now:yyyyMMddhhmmss}.zip";
        ExcludeFilesRegex =
            new Regex(string.Join("|", string.Join("|", ExcludeFiles), string.Join("|", ExcludePaths)));
    }

    public Regex ExcludeFilesRegex { get; }
    public IEnumerable IncludePaths { get; }
    public IEnumerable ExcludeFiles { get; }
    public IEnumerable ExcludePaths { get; }
    public string BackupFile { get; }
}

To make it easy to select the paths and files to be excluded, I decided to give it a Regex style and create a Regex that will match all files. For example, if you want to remove all mp3 files, you would add something like “\.mp3$” (starts with a “.”, then mp3 and then the end of the string). If you want to remove mp3 and mp4 files, you can add this: “\.mp[34]$”. For the paths, you get the same thing, but they start and end with a slash (double slash, for the regex).

With this in place, we can start our backup. Create a new class and call it Backup.cs. Add this code to it:

class Backup
{
    private readonly FileFinder _fileFinder = new FileFinder();

    public async Task DoBackup(Config config, bool incremental)
    {
        var files = await _fileFinder.GetFiles(config.IncludePaths.ToArray(), 
               config.ExcludeFilesRegex, incremental);
        using (ZipFile zip = new ZipFile())
        {
            zip.UseZip64WhenSaving = Zip64Option.AsNecessary;
            foreach (var path in files)
                zip.AddFiles(path.Value, false, path.Key);
            zip.Save(config.BackupFile);
        }
        foreach (var file in files.SelectMany(f => f.Value))
            ResetArchiveAttribute(file);
        return 0;
    }

    public void ResetArchiveAttribute(string fileName)
    {
        var attributes = File.GetAttributes(fileName);
        File.SetAttributes(fileName, attributes & ~FileAttributes.Archive);
    }
}

This class uses a FileFinder class to find all files that match the pattern we want and creates a zip file. The GetFiles method from FileFinder returns a dictionary structured like this:

  • The key is a path related to the search path. As the paths can be on any disk of your system and they can have the same names (ex C:\Temp and D:\Temp), and that would not be ok in the zip file, the paths are changed to reflect the same structure, but their names are changed to allow to be added to the zip files. That way, if I am searching in C:\Temp and in D:\Temp, the keys for this dictionary would be C_Drive\Temp and D_Drive\Temp. That way, both paths will be stored in the zip and they wouldn’t clash. These keys are used to change the paths when adding the files to the zip
  • The value is a list of files found in that path

The files are added to the zip and, after that, their Archive bit is reset. This must be done, so the incremental backup can work in the next time: incremental backups are based on the Archive bit: if it’s set, the file was modified and it should be backed up. If not, the file was untouched. This is not a foolproof method, but it works fine for most cases. A more foolproof way to do this would be to keep a log file every full backup with the last modified dates of the files and compare them with the current file dates. This log should be updated every backup. For my case, I think that this is too much and the archive bit is enough.

The FileFinder class is like this one:

class FileFinder
{
    public async Task<ConcurrentDictionary<string, List>> GetFiles(string[] paths, 
        Regex regex, bool incremental)
    {
        var files = new ConcurrentDictionary<string, List>();
        var tasks = paths.Select(path =>
            Task.Factory.StartNew(() =>
            {
                var rootDir = "";
                var drive = Path.GetPathRoot(path);
                if (!string.IsNullOrWhiteSpace(drive))
                {
                    rootDir = drive[0] + "_drive";
                    rootDir = rootDir + path.Substring(2);
                }
                else
                    rootDir = path;
                var selectedFiles = Enumerable.Where(GetFilesInDirectory(path), f => 
                     !regex.IsMatch(f.ToLower()));
                if (incremental)
                    selectedFiles = selectedFiles.Where(f => (File.GetAttributes(f) & FileAttributes.Archive) != 0);
                files.AddOrUpdate(rootDir, selectedFiles.ToList(), (a, b) => b);
            }));
        await Task.WhenAll(tasks);
        return files;
    }

    private List GetFilesInDirectory(string directory)
    {
        var files = new List();
        try
        {
            var directories = Directory.GetDirectories(directory);
            try
            {
                files.AddRange(Directory.EnumerateFiles(directory));
            }
            catch
            {
            }
            foreach (var dir in directories)
            {
                files.AddRange(GetFilesInDirectory(Path.Combine(directory, dir)));
            }
        }
        catch
        {
        }

        return files;
    }
}

The main method of this class is GetFiles. It is an asynchronous method, I will create a new task for every search path. The result is a ConcurrentDictionary, and it has to be so, because there are many threads updating it at once and we could have concurrency issues. The ConcurrentDictionary handles locking when adding data from different threads.

The GetFilesInDirectory finds all files in one directory and, after all files are found, the data is filtered according to the Regex and, if the user asks for an incremental backup, the files are checked for their archive bit set. With this set of files, I can add them to the zip and have a backup file that can be read with standard programs.

Just one requirement remains: to have a scheduled backup. I could make the program stay in the system tray and fire the backup at the scheduled times, but there is an easier way to do it: use the Windows task scheduler. You just need to open a command prompt and type the command:

schtasks /create /sc daily /st "08:15" /tn "Incremental Backup" /t
r "D:\Projetos\Utils\BackupData\BackupData\bin\Debug\Backupdata.exe -i"

That will create a scheduled task that will run the incremental backup every day at 8:15. The main program for this backup is very simple:

static void Main(string[] args)
{
    var options = new Options();
    CommandLine.Parser.Default.ParseArguments(args, options);
    if (string.IsNullOrWhiteSpace(options.ConfigFile))
        return;
    if (string.IsNullOrWhiteSpace(Path.GetDirectoryName(options.ConfigFile)))
    {
        var currentDir = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
        if (!string.IsNullOrWhiteSpace(currentDir))
            options.ConfigFile = Path.Combine(currentDir, options.ConfigFile);
    }
    var config = new Config(options.ConfigFile);
    var backup = new Backup();
    var result = backup.DoBackup(config, options.Incremental).Result;

}

I will parse the arguments, read and parse the config file, create the backup and exit. As you can see, the last line calls DoBackup.Result. This is because the Main method cannot be async and, if I just run it without calling async, it would not wait and would exit without running the backup. Calling result, the program will wait for the task completion.

Just one issue, here – if you wait for the task schedule to fire, you will see that a console window appears, and we don’t want that this happens while we are doing something else. One way to hide the console window is to go to the app properties and set the output type as a Windows application. That will be enough to hide the console window:

image

Conclusions

As you can see, it’s not too difficult to make a backup program using open source tools we have available. This program is very flexible, small and not intrusive. It can run anytime you want and have different configuration files. Not bad, huh?

The full source code for the project is in  https://github.com/bsonnino/BackupData

Acabo de concluir uma série de vídeos sobre o uso de sensores em UWP, que publiquei no Channel 9. São vídeos curtos, com até 15 minutos cada. Vale a pena dar uma conferida e ver como usar os sensores disponíveis no Windows 10 (os programas funcionam tanto no desktop, em tablets, como no smartphone Windows 10):

https://channel9.msdn.com/Series/Windows-Development/Trabalhando-com-Sensores-em-UWP-Parte-1-Sensor-de-Luz – Sensor de Luz – Mostra como usar o sensor de luz para mudar a visualização conforme a luminosidade do ambiente

https://channel9.msdn.com/Series/Windows-Development/Trabalhando-com-Sensores-em-UWP-Parte-2-Bssola – Bússola – Mostra como usar a bússola para dizer ao usuário a sua orientação

https://channel9.msdn.com/Series/Windows-Development/Trabalhando-com-Sensores-em-UWP-Parte-3-Inclinmetro – Inclinômetro – Usa o inclinômetro para mover uma bola na tela conforme o usuário inclina seu dispositivo para a direita ou para a esquerda

https://channel9.msdn.com/Series/Windows-Development/Trabalhando-com-Sensores-em-UWP-Parte-4-Acelermetro – Acelerômetro – Usa o acelerômetro para fazer uma bola pular na tela quando se chacoalha o dispositivo

https://channel9.msdn.com/Series/Windows-Development/Trabalhando-com-Sensores-em-UWP-Parte-5-Geolocalizao – Geolocalização – Usa o sensor de localização para obter o local atual, consultar um serviço de meteorologia e saber se irá chover no seu local atual

After showing the desktop icons (see my last post), I right clicked the desktop and hid again the desktop icons. That way I’ve missed the context menu again (If you are running Build 10547 and hide the desktop icons, you lose the context menu).

But this time, Shift-F10 didn’t work for me. So, I decided to use another solution. I remember I’ve developed an utility for PCMag.com, Wintidy (http://www.pcmag.com/article2/0,2817,2372867,00.asp) which, among other things, shows or hides the desktop icons. I ran it and clicked on the “Show desktop icons” and voilà, the icons were back again!

Now I have a permanent solution, no more flaky hotkeys. I hope the Windows team solves this bug soon.

 

 

I am running Windows 10, build 10547, and I was having a strange problem: I had disabled my icons in the desktop and the right click wasn’t working in the desktop – it seems to be a known problem – if you have your icons hidden, right clicking on the desktop don’t show the context menu for the desktop. I didn’t know it at that time and, after doing the wrong searches and not finding anything useful, I thought it could be a Windows extension that might be causing this.

So, I decided to boot my computer in Safe Mode. I pressed Windows+R to open the Run window and typed msconfig to open the System Configuration. Then, I went to the Boot tab and selected Safe Boot with network:

MsConfig

I restarted my computer and then my saga began. The computer restarted fine, asked my password and entered the desktop, showing Safe Mode. But it was unusable – the wait cursor was spinning and I couldn’t do anything. I decided to wait. After one hour, nothing had happened, the cursor was still spinning.

I decided that the icons in my desktop didn’t worth the wait and decided to go back to normal mode. But how? With Ctrl-Alt-Del I could restart the computer and return to Safe mode again, but there was no way to return to normal mode. Pressing Shift while clicking on the “Restart” menu didn’t work – my computer was showing a message that there was no boot. I went to the boot options and selected the Uefi boot again and was back again to Safe mode (oh, no).

Then I thought – I have another computer with Windows 10, so I’ll make a recovery disk in a USB drive. It took more than one hour to make this recovery disk (I don’t know why did it take so long – Windows takes less than half hour to reinstall itself, but at that time I was becoming a little desperate). With the recovery disk, I booted my computer and it booted to recovery – good, at least I have hope. But there wasn’t any options to restore to normal mode. I tried to do a system restore, but it showed me a cryptic error message to select an operating system and nothing else.

I opened a command prompt and typed “bcdedit”, in the hope I could have some light – nothing, it just showed me an error message and no entries. I scanned all drives in the command prompt, but my C drive wasn’t there. My options were narrowing. Then, I thought I could create a bootable Windows disk in the USB drive and try to use it, if not to repair my Windows, at least to reinstall it (so you can see how desperate I was).

I downloaded the Windows ISO from the MSDN site (I have a subscription) and then downloaded the Windows USB tool from https://www.microsoft.com/en-us/download/windows-usb-dvd-download-tool and followed the instructions to create the USB. The ddownload of the ISO and creation of the bootable USB was way faster than creating the recovery drive.

I changed my computer boot to boot from the USB and there it was – the prompt to install. I went through the first screen and there was a prompt to Repair my computer:

Install

I clicked on it and now I tried a system restore again – It showed me the Windows 10 system and showed me a restore point. Now we’re running – I selected the restore point and it began to work and rebooted. When the system booted, I saw with despair that it was still on Safe Mode – it had done a system restore, but hadn’t changed the boot.

Back to the install drive again, I had now some hope – the repair in the drive was able to show me my system. So I opened the command prompt again and typed “bcdedit”. It showed me the entries, and the {default}  one was the one I was searching, it had the clause “safeboot Network” there. I had to delete it. Then, I typed:

bcdedit /deletevalue {default} safeboot

After seeing that the command was issued, I restarted the computer and there it was – normal mode again. After 4 hours of struggling with boots, recovery disks, and install usbs, I was back at where I had started – still no desktop icons, but happy again.

My conclusion? Safe mode in Windows 10 didn’t work for me. If it’s to be like that, it’s better to remove it. Recovery disks didn’t work either – slow to create and no use for it. The only thing that worked was the install disk – this one is precious – you should keep an USB drive with a copy of Windows ready to boot.

And did I solve my icons’ problem? Yes. After doing a good search (“show desktop icons windows 10 build 10547”), I saw that it was a bug in this build and saw the fix – press Shift-F10 to show the context menu. It worked!