BASH Cures Cancer

I asked “What do you want” and you said scripting. Which is good, because I have felt like scripting lately!

I help a website hosting company, Idologic, on the weekends. (Side note: I highly recommend Idologic. I have worked with and been a customer of many other hosting companies. I really doubt you will find better customer service elsewhere.) Like many businesses these days, Idologic has quite a few Linux servers. When presented with many servers, I typically want to parallelize my work.

As such, I have written a script called dssh (previous version), which allows you to execute commands on n hosts, in parallel. This can be used to find information on the hosts, such as load average, number of processes by user, number of processes by process name, etc.

There are other options such as pssh and p-run, however I wanted to create a shell solution which could be easily and simply “installed”. Dssh reads standard input. It expects one host per line. Host specific ssh options are supported. Here is my sample hosts file:

$ cat hosts
mojito
-l noland kodiak
mojito
kodiak
-C mojito
-i /home/noland/.ssh/id_rsa kodiak

There is nothing restricting you from generating this output from some type of meta data (I.E. database). Here are some examples of output:

$ ./dssh.sh "uptime" < hosts
First time huh? Think your cmd over and then try again.
$ ./dssh.sh "uptime" < hosts
mojito:O:0:19:16:45 up 3 days, 14 min,  5 users,  load average: 0.22, 0.22, 0.20
kodiak:O:0:13:24:00 up 20:00,  1 user,  load average: 0.42, 0.16, 0.05
mojito:O:0:19:16:45 up 3 days, 14 min,  5 users,  load average: 0.22, 0.22, 0.20
kodiak:O:0:13:24:00 up 20:00,  1 user,  load average: 0.42, 0.16, 0.05
mojito:O:0:19:16:45 up 3 days, 14 min,  5 users,  load average: 0.22, 0.22, 0.20
kodiak:O:0:13:24:00 up 20:00,  1 user,  load average: 0.42, 0.16, 0.0

$ ./dssh.sh "pgrep -u noland | wc -w" < hosts
mojito:O:0:60
kodiak:O:0:5
mojito:O:0:60
kodiak:O:0:5
mojito:O:0:60
kodiak:O:0:5

$ ./dssh.sh "ls not_a_file" < hosts
mojito:E:2:ls: not_a_file: No such file or directory
kodiak:E:2:ls: not_a_file: No such file or directory
mojito:E:2:ls: not_a_file: No such file or directory
kodiak:E:2:ls: not_a_file: No such file or directory
mojito:E:2:ls: not_a_file: No such file or directory
kodiak:E:2:ls: not_a_file: No such file or directory

Notes:

1. With great power comes even greater responsibility. Running rm -rf / as root with this script would do exactly that.
2. I don’t reccomend doing anything with this script that “changes state”.
3. I make no warranties or promises.
4. You need ssh keys to use this. I recommend using ssh-agent.
5. By default dssh will execute 10 children in parallel. If you have a large host, increase this.
6. When looping through the hosts, if the maximum number of children are still processing, the script will sleep 500ms. If your version of sleep does not support fractional seconds, you will need to change this.

Here is an outline of the script:

1. Read from standard input a list of hosts
2. Configure trap to remove temporary files on exit
3. For each host
   1. Sleep while we have more children than the maximum number of children
   2. Generate three temporary files, one for each of
      1. Standard Output
      2. Standard Error
      3. Exit value
   3. Create a child process saving stdin, stderr, and the exit value in their respective files.
4. Wait for all children to exit
5. For each host
   1. If the standard output or error files are of size greater than zero, print the content, prefacing each line with the hostname, standard error/output indicator, and exit status.
   2. Else print something to indicate we executed a process and have an exit value.

Once again, here is the script I am calling dssh.