Easy path to memoization for UNIX text piping applications

We've all had the experience of slow commandline applications which produce text. For those applications with no significant side effects, below is a recipe to institute caching without invasive changes. This solution uses bash and perl, and is most conveniently applied to bash scripts by simply inserting a line at the head of the script as follows:

. cache $*

To apply this technique to text producing programs which are not themselves bash scripts, simply make a bash wrapper for the program to be cached, and apply this technique to the wrapper.

Below is the code, which simply calls a Perl script to execute the program which has been targeted for caching, guarding against an infinite loop by means of the __DISABLE_CACHE__ environment variable (which also gives cache-aware programs an interface to prevent this scheme from being in effect at all times, if there are known occasions where we want to disable caching).


# add . cache $* to scripts.
# (TOO SLOW W/ some rubys, so using perl instead
# (of ruby -w $DROP/bin/ruby/cache.rb $*))

if [ -z "$__DISABLE_CACHE__" ]; then
export __DISABLE_CACHE__=yes
perl -w $DROP/bin/perl/cache.pl $0 $*
exit
fi

The perl script which does the work is likewise very straightforward. If there is any output on stderr, the script assumes there was trouble and does not cache anything; otherwise, you should just get a single real execution per permutation of intput arguments.


use strict;

my $__trace = 0;
my $cmd = join ' ', @ARGV;

my $cached_output = $cmd;
$cached_output =~ s{[^\w]}{_}g;
$cached_output = $ENV{'TMP'} . "/cache." . $cached_output;

if (! -f $cached_output)
{
my $cmd_with_redirects = "$cmd > $cached_output 2> $cached_output.err";
`$cmd_with_redirects`;
if ($__trace)
{
print "Executed $cmd_with_redirects\n";
}
if ( `cat $cached_output.err` eq '' )
{
if ($__trace)
{
print "No error output, so deleting $cached_output.err\n";
}
unlink "$cached_output.err";
}
}
if ($__trace)
{
print `ls -l $cached_output $cached_output.err`;
}

print `cat $cached_output`;
if (-f "$cached_output.err" )
{
print STDERR `cat $cached_output.err`;
# assume trouble if there was output to stderr, and remove the cached output:
unlink "$cached_output.err";
unlink $cached_output;
}