BASH Cures Cancer

Yesterday a 20 line shell script caught a race condition in one of software I work on. Our best engineers looked at the problem, a vendor case was filed, and there was thoughts of automatically restarted the application to fix the issue at hand.

In the end, a 20 line shell script to strace for file operations continuously provided the needed visibility into the issue at hand.

Update: The shell script.

The shell script itself, though trivial, is owned by the company I work for, but the basic concept was as follows:

#!/bin/bash

# loop forever
while sleep 60
do
pid=$(ps -ef | grep application | egrep -v grep | awk ‘{print $2}’)
if [[ -n $pid ]]
then
outfile=strace-application-$(date +%Y%M%d-%H:%M:%S).out.gz
strace -tt -f -e trace=file -p $pid 2>&1 | gzip -c > $outfile
fi
done