Wednesday, January 8, 2014

Combining All Files into a Single Batch

One thing I figured out how to do was to simplify the installation of a complex set of files that are used as auxiliary files for a master batch file.  For example, for the tool that I built over the last few weeks, I ended up with gatherer.vbs, entitycleaner.pl, and tsv2csv.pl that were all called from within the master batch file.  While I was developing things, it was handy to have these as individual files because I could easily edit them individually.  However, when installing the tool on the final server where it needed to go, I had to zip up all the files (including some other files that I haven't released yet), copy them over, and unzip them (replacing the existing ones if they were already there).  This wasn't too efficient and I needed an easier way to transmit the scripts.

The solution I came up with was to embed the contents of the auxiliary files into the master batch file.  I then instructed the master batch file to echo the auxiliary file contents into the files at runtime.  This meant that I only had to transmit the single batch file to the target server and run it.  When it ran, it would create the files it needed, use them, then destroy them.  This also kept the working directory clean.

There were a few caveats, however.  The biggest hurdle is trying to echo the special characters that other programming languages use from within a batch file.  There were two main resources that I used to make sure things worked right.  The first is Rob VanDerWoude's web page on escaping special characters.  This helped me determine which characters from the source script would have to be escaped when echoing out from the batch file to the auxiliary file.  The second utility I used was http://text-compare.com/.  It allows you to compare one text file to another and tells you where the differences are.

The next thing I did was create a simple batch file to echo each line of the content out to a new file.  I then took the original content and compared it to the content created by the simple batch file.  For example, for tsv2csv.pl on the left, I created the batch file on the right:

You can see that the batch file on the right uses the carat (^) to escape several of the characters.  Since I used one redirector at the end (line 11) to output the entire thing into the test output file, lines 2-10 needed to be in parenthesis.  This means that all parentheses needed to be escaped.  Also, since > and < are redirectors in Windows Command, they also had to be escaped.  Interestingly enough, since the > on line 8 is in double quotes, it doesn't need to be escaped.  This is only possible since I needed the double quotes anyway in the auxiliary file.  If I had used single quotes on line 8, I would have had to escape the >.

In this way, I combined all the auxiliary files into one single batch file.  I added the lines from the test batch file into a function in the master and called the function from the main program section of the master batch.

Some other characters that have to be escaped are the ampersand (&), pipe (|), and percent (%).  The & and | were easily escaped.  However % proved to be more difficult.  Normally, the % sign is followed by a non-alphanumeric character (space or punctuation).  However, when trying to output a date format for SQL, the % sign is followed by characters (i.e. %Y-%m-%d).  This causes a problem because Windows command is reads each of those a variables.  You can't even escape them because in a batch file, the double percent (%%Y) reads as a variable too.  In the end, what I had to do was store 'Y' in a variable called 'FormatY' like this:

set formatY=Y
echo %%%formatY%

The first two percent signs read as an escaped percent sign and a single % is echoed.  The %formatY% resolves to Y and a single Y is output.  To get the whole string, I did this:

set formatY=Y
set formatm=m
set formatd=d
set formatH=H
set formati=i
set formats=s
echo %%%formatY%-%%%formatm%-%%%formatd% %%%formatH%:%%%formati%:%%%formats%

The output looks like this:

%Y-%m-%d %H:%i:%s

This is a messy way of doing things, but it works.