BASH Cures Cancer

Yes, everyone should write shell scripts. No matter what you are doing on a computer, it’s often you want to get it done more than once. At that point, it should be in a shell script.

I teach Apache Hadoop and in our courses we have students complete labs. The very first lab in one course is simply pushing files to HDFS:


$ hadoop fs -put file .
$ hadoop fs -ls


Even these trivial examples should be scripted. Why? Paul MacCready. Paul is one of the two inventors of the first Human Powered Aircraft. As this excellent post argues, Paul’s solution to the problem was not that he was a better Engineer than those who had tried previously (though that could be true as well), but that Paul knew they had to iterate fast.

Paul realized that what we needed to be solved was not, in fact, human powered flight. That was a red-herring. The problem was the process itself, and along with it the blind pursuit of a goal without a deeper understanding how to tackle deeply difficult challenges. He came up with a new problem that he set out to solve: how can you build a plane that could be rebuilt in hours not months.

One again from The Wrong Problem.

Thus, the reason that you should write shell scripts is so that you can iterate faster. Simply typing `hadoop fs -put’ the first time is OK. Once you understand the command, you should be scripting it. This is how I solve over 75% of my problems. I write a script called “run.sh” which simply sets up the environment, deletes previous output files, and executes the command sequence I think will resolve my problem. If it’s a little off, I just make a small change and iterate. Instead of:


$ cmd1
$ cmd2
$ cmd3
$ ls -l
$ rm -rf ouptut
$ cmd1
$ cmd2
$ cmd2.5
$ cmd3
$ ls -l
$ rm -rf ouptut


It’s


$ ./run.sh
$ vim run.sh
$ ./run.sh


This my friend, is much, much faster. Fast iterations == success.