BASH Cures Cancer

The web services paradigm of development is based on the Unix philosophy of “small is good”.  Web services should do one job, and do it well, allowing users to develop complex solutions by combining small, reliable and proven services.
Why not then, expose the power of familiar Unix commands like sort, grep, gzip… to the web?

Here is a proof of concept python script (Python 2.3 version) to demonstrate.

Start services:

$ ./to_web.py -p8008 sort &
Thu Mar 27 13:45:54 2008 sort server started - 8008
$ ./to_web.py -p8009 gzip &
Thu Mar 27 13:46:29 2008 gzip server started - 8009

Use the services:

$ for i in {1..10}; do echo ${RANDOM:0:2}; done | \
> curl --data-binary @- "http://swat:8008/sort+-nr" | \
> curl --data-binary @- "http://swat:8009/gzip" | \
> gunzip
97
37
23
23
21
18
11
11
10
10

In my position, we have a database with host information – which has a command line interface. This tool has dependencies which are a painful to resolve. With to_web.py, we can turn the command line tool into a web service and access the data without having to satisfy those additional dependencies.

This is guest post by my esteemed colleague Adam Fokken. He can be reached here: Sadly, he does not have a blog.

There are situations where, if you want a Python, PERL, PHP, etc script to be portable among a few different servers, it makes sense to wrap the script in shell. A few years ago I was trying to use the Python cx_Oracle module. This module is a wrapper for the native Oracle database driver. However, it requires the driver library directory be in the LD_LIBRARY_PATH environment variable.

No problem I thought. I’ll use the os.environ dict to set the variable.  Example script:

$ cat python-only.sh
#!/usr/bin/python
import sys, os
sys.path.append("/usr/local/lib/python2.4/site-packages/")
if not os.environ.has_key('LD_LIBRARY_PATH'):
        os.environ['LD_LIBRARY_PATH'] = "/home/noland/oracle-lib"
else:
        os.environ['LD_LIBRARY_PATH'] = "/home/noland/oracle-lib:" + os.environ['LD_LIBRARY_PATH']
print "LD_LIBRARY_PATH looks OK in Python: LD_LIBRARY_PATH = ", os.environ['LD_LIBRARY_PATH']
os.system('echo LD_LIBRARY_PATH looks OK via os.system: LD_LIBRARY_PATH = $LD_LIBRARY_PATH')
try:
        import cx_Oracle
        print "Imported cx_Oracle! LD_LIBRARY_PATH was set correctly."
except ImportError, e:
        print "Woops, LD_LIBRARY_PATH was not set correctly: ", e

This method does not work:

$ ./python-only.sh
LD_LIBRARY_PATH looks OK in Python: LD_LIBRARY_PATH =  /home/noland/oracle-lib
LD_LIBRARY_PATH looks OK via os.system: LD_LIBRARY_PATH = /home/noland/oracle-lib
Woops, LD_LIBRARY_PATH was not set correctly:  libclntsh.so.10.1: cannot open shared object file: No such file or directory

This seems to be a common problem. However, when I was dealing with this a few years ago, I could not find a good resource on Google. I bite the bullet and wrote a separate shell script wrapper – hating invocation of the shell script. However, there is absolutely no reason I needed a separate shell script. I could have embedded the Python within a shell script. Example:

$ cat python-and-bash.sh
#/bin/bash
export LD_LIBRARY_PATH=/home/noland/oracle-lib:$LD_LIBRARY_PATH
/usr/bin/python<<END_OF_PYTHON
import sys
sys.path.append("/usr/local/lib/python2.4/site-packages/")
try:
        import cx_Oracle
        print "Imported cx_Oracle! LD_LIBRARY_PATH was set correctly."
except ImportError, e:
        print "Woops, LD_LIBRARY_PATH was not set correctly: ", e
END_OF_PYTHON

Ahh, much better:

$ ./python-and-bash.sh
Imported cx_Oracle! LD_LIBRARY_PATH was set correctly.

Of course I could have just set this variable in my profile. However, this creates an additional external dependency – which is what I was trying to avoid.

Quite some time ago, someone wrote me to ask about a possible article on process substitution. Sadly, I could not find the email so I cannot credit them. As you likely have guessed, I am finally writing a post on process substitution.

Many times I have used pipelines and temporary files when process substitution would be a much cleaner solution.

First, I am going to create two test files:

$ dd if=/dev/urandom of=file-small count=750001
$ dd if=/dev/urandom of=file-large count=1000000
$ ls -l file-*
-rw-r--r-- 1 noland noland 512000000 Mar 23 08:53 file-large
-rw-r--r-- 1 noland noland 384000512 Mar 23 08:49 file-small

I thought of writing this article while writing a script to test ftp servers and file locking. As such I will upload the small file to a file named append-example:

$ curl -T file-small --user noland ftp://localhost/append-example
Enter host password for user 'noland':
$ ls -l append-example
-rw-r--r-- 1 noland noland 384000512 Mar 23 11:52 append-example

Now I will append  the large file:

$ curl -s -a -T file-large --user noland ftp://localhost/append-example
Enter host password for user 'noland':
$ ls -l append-example
-rw-r--r-- 1 noland noland 896000512 Mar 23 11:54 append-example

I am going to use dd and process substituion to caculate the MD5 hash of the first upload:

$ md5sum file-small <(dd if=append-example count=750001 status=noxfer)
dfabff7441bd814145a804e03d333864  file-small
1000000+0 records in
1000000+0 records out
dfabff7441bd814145a804e03d333864  /dev/fd/63

Now the portion that was appended:

$ md5sum file-large <(dd if=append-example  skip=750001 status=noxfer)
1b8daed9e435fc90b4a49d74b55f96f4  file-large
1000000+0 records in
1000000+0 records out
1b8daed9e435fc90b4a49d74b55f96f4  /dev/fd/63

When you place a command inside <( ) the shell sets standard output of the command to pipe inside /dev/fd/ and replaces the command with that pipe. Here is the classic example:

$ echo <(echo) <(echo) <(echo) <(echo)
/dev/fd/63 /dev/fd/62 /dev/fd/61 /dev/fd/60

In my script I use process substitution as below (effectively) which feels exeedingly clean:

$ read hash name < <(md5sum <(dd if=append-example skip=750001 status=noxfer))
1000000+0 records in
1000000+0 records out
$ printf "hash=%s name=%s\n" $hash $name
hash=1b8daed9e435fc90b4a49d74b55f96f4 name=/dev/fd/63

UPDATE: Including the one I added after posting and Elias‘ quoting exampling the comments we are up to eight.

After reading Shell Scripting Recipes, I became more interested in the speed of shell operations. In his book, Chris says “Command Substitution Is Slow.” He is correct!

$ f() { echo -n }; time for i in {0..100}; do v=$( f ); done

real    0m4.189s
user    0m0.000s
sys     0m4.188s

$ f() { _F="" }; time for i in {0..100}; do f; v=$_F; done

real    0m0.006s
user    0m0.000s
sys     0m0.000s

I found a few other equivalent operations which can be used to speed up shell scripts to varying degrees (none like the above) depending on the task at hand.  As Chris says, “the extra few milliseconds … may not seem significant, but scripts often loop hundred of even thousands of times.”

${#array[@]} is faster than () when expanding an array (#7)

$ a=(); time for i in {0..1000}; do a=(${a[@]} $i);done; echo ${#a[@]}

real    0m3.545s
user    0m3.544s
sys     0m0.000s
1001

$ a=(); time for i in {0..1000}; do a[${#a[@]}]=$i;done; echo ${#a[@]}

real    0m0.043s
user    0m0.040s
sys     0m0.003s
1001

< is faster than cat

$ time for i in {0..10000}; do var=`cat out`;done

real    0m9.328s
user    0m2.892s
sys     0m6.436s

$ time for i in {0..10000}; do var=`<out`;done

real    0m5.930s
user    0m1.412s
sys     0m4.520s

echo is faster than printf (though not nearly as powerful)

$ time for i in {0..100000}; do printf "\n"; done >/dev/null

real    0m4.446s
user    0m4.076s
sys     0m0.236s

$ time for i in {0..100000}; do echo; done >/dev/null

real    0m3.291s
user    0m3.100s
sys     0m0.184s

Arithmetic Evaluation is faster than let

$ i=0; time while :; do let "i = i + 1"; [[ $i -gt 100000 ]] && break;done

real    0m8.211s
user    0m7.900s
sys     0m0.304s

$ i=0; time while :; do ((i++)); [[ $i -gt 100000 ]] && break;done

real    0m5.287s
user    0m4.980s
sys     0m0.304s

UPDATE: This appears to still be true, but by a different margin. See comments.

List expansion is faster than seq and command substitution (though not always available)

$ time for i in $(seq 0 1000000); do :; done

real    0m28.482s
user    0m28.066s
sys     0m0.412s

$ time for i in {0..1000000}; do :; done

real    0m24.563s
user    0m24.402s
sys     0m0.156s

UPDATE: On BSD systems the apparent seq equivalent (jot) is faster than list expansion. See comments.

: is faster than true

$ i=0; time while true; do ((i++)); [[ $i -gt 1000000 ]] && break;done

real    0m57.360s
user    0m53.967s
sys     0m3.392s

$ i=0; time while :; do ((i++)); [[ $i -gt 1000000 ]] && break;done

real    0m54.138s
user    0m50.571s
sys     0m3.560s

Missing space – deleting open files

I ran into this one again today. If a file is open when deleted, it will not appear in a directory listing, but will take up space.

# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      72G   58G   11G  86% /
# cat - >>large-file &
[1] 8958
# lsof large-file
COMMAND  PID USER   FD   TYPE DEVICE       SIZE    NODE NAME
cat     8958 root    1w   REG  253,0 5120000000 4300883 large-file
# rm -f large-file
# lsof | grep large-file
cat       8958      root    1w      REG      253,0 5120000000    4300883 /root/large-file (deleted)
# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      72G   58G   11G  86% /
# kill -9 8958
# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      72G   53G   15G  79% /
[1]+  Killed                  cat - >>large-file

uuencode/uudecode on RHEL (CentOS)

Earlier today I was looking to use uuencode on my RHEL host. Unfortunately, yum did not help:

# yum search uuencode
Loading "installonlyn" plugin
Setting up repositories
base                      100% |=========================| 1.1 kB    00:00
updates                   100% |=========================|  951 B    00:00
addons                    100% |=========================|  951 B    00:00
extras                    100% |=========================| 1.1 kB    00:00
Reading repository metadata in from local files
No Matches found

Furthermore, I struggled to find the correct search terms for Google to provide me with an answer. The correct package is “sharutils.” Anyways, for good measure, here is a quick demo of uuencode/uudecode:

$ echo "BASH Cures Cancer" > test.txt
$ zip test.zip test.txt
  adding: test.txt (stored 0%)
$ uuencode < test.zip -
begin 664 -
M4$L#!`H``````-%9=3@7HDD\$@```!(````(`!4`=&5S="YT>'155`D``^G>
MXT?IWN-'57@$`/0!]`%"05-(($-U<F5S($-A;F-E<@I02P$"%P,*``````#1
M674X%Z))/!(````2````"``-```````!````M($`````=&5S="YT>'155`4`
?`^G>XT=5>```4$L%!@`````!``$`0P```$T`````````
`
end
$ uuencode < test.zip - | uudecode > test2.zip
$ unzip test2.zip
Archive:  test2.zip
replace test.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 extracting: test.txt
$ cat test.txt
BASH Cures Cancer

From the manual: “Uuencode reads file (or by default the standard input) and writes an encoded version to the standard output.  The encoding uses only printing ASCII characters and includes the mode of the file and the operand name for use by uudecode. “

I realized today, that it may not be obvious that command substitution can be done in parallel with compound commands. Meaning that variable assignment and tests can be done inline. For example, the exit status of the command inside an assignment operation is the exit value of the assignment operation:

$ val=$( false ); echo $?
1
$ val=$( true ); echo $?
0

I used this today when I wrote a long running test which slept 60 seconds between loops. Midway through the test I decided I wanted it to sleep only 15 seconds. Here was my command to resolve this situation without stopping my test:

$ while :; do p=$(pgrep sleep) && sleep 15 && kill $p; sleep 1;done

If pgrep finds and any “sleep” process, the command sleeps 15 seconds and then kills the previously described process. After which it sleeps one second and looks for more “sleep” processes to kill. I am assigning the output of pgrep to the variable p and using pgrep’s exit status to decided whether any processes need to be killed.

The alternative would have been something like:

$ while :; do p=$(pgrep sleep); [[ ! -z "$p" ]] && sleep 15 && kill $p; sleep 1;done

Demonstration

Here is my base iptable INPUT chain:

# iptables -L INPUT -n
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22

As you can see, I am dropping all packets except TCP packets on port 22. I am going to open up port 4550:

# iptables -A INPUT -p tcp --dport 4550 -j ACCEPT
# iptables -L INPUT -n
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:4550

Here I am using a netcat and an infinite loop as a simple “server” to send “i = $i” when someone connects to port 4550:

# i=0;while :; do echo i = $i | nc -l 192.168.6.20 4550; ((i++)); echo $i;done
1
2
3

In another terminal I have connected to port 4550 three times:

# time nc -w 120 -v 192.168.6.20 4550
Connection to 192.168.6.20 4550 port [tcp/*] succeeded!
i = 0

real    0m0.842s
user    0m0.001s
sys     0m0.014s
# time nc -w 120 -v 192.168.6.20 4550
Connection to 192.168.6.20 4550 port [tcp/*] succeeded!
i = 1

real    0m0.822s
user    0m0.000s
sys     0m0.007s
# time nc -w 120 -v 192.168.6.20 4550
Connection to 192.168.6.20 4550 port [tcp/*] succeeded!
i = 2

real    0m0.526s
user    0m0.002s
sys     0m0.009s

Now I am going to delete the ACCEPT rule and add a REJECT rule:

# iptables -D INPUT 2
# iptables -A INPUT -p tcp --dport 4550 -j REJECT
# iptables -L INPUT -n
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:4550 reject-with icmp-port-unreachable

Here is the output of the “client” netcat command after adding the REJECT rule:

# time nc -w 120 -v 192.168.6.20 4550
nc: connect to 192.168.6.20 port 4550 (tcp) failed: Connection refused

real    0m1.113s
user    0m0.000s
sys     0m0.005s

As you can see the command returned after ~1 second with an error. Now I am going to delete the REJECT rule. The default rule, DROP, will now be in effect:

# iptables -D INPUT 2
# iptables -L INPUT -n
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22

Here is the output of the “client” program in another terminal session:

# time nc -w 120 -v 192.168.6.20 4550
nc: connect to 192.168.6.20 port 4550 (tcp) timed out: Operation now in progress

real    2m0.152s
user    0m0.000s
sys     0m0.001s

The command took two minutes to return with the error. The -w 120 option causes netcat to timeout if no reply is recieved after 120 seconds.

Explanation

iptables is often used to block a specific ip address or subnet whom are doing something maclious. A REJECT rule will cause the maclicious host to recieve an error shortly after the connection attempt. A DROP rule will act differently. If they have set a client timeout, the malicious host will wait until said timeout is satisfied. Most likely several seconds. This should significantly slow the malicious program. A program without a client timeout will sit for hours waiting for a reply.

In the past, my SSH sessions died due to inactivity. In order to solve this, I used to:

while true; do uptime; sleep 5;done

Obviously, this eventually clears your terminal history. BASH to rescue! My noop script solves this problem. (Please see comments, there maybe a better solution, thanks David!) noop, standing for no operation, is a processor instruction and is common in protocols. You may find it interesting, that exploit code is filled with NOP’s. The operation increases your chances of exploiting buffer overflows

The source:

$ cat /usr/bin/noop
#!/bin/bash
backspace() {
        echo -e "\b\c"
}
cleanup() {
        backspace
        exit
}
trap "cleanup" 2
while :
do
        num=${RANDOM:0:1}
        printf $num
        sleep ".$num"
        backspace
done

For the hell of it, I made a video of noop in action.

If your wondering how the script works, here is a quick explanation. The script defines two functions. backspace and cleanup. Backspace prints the special characters \b and \c.  Backslash b is a backspace, and backslash c, stops echo from printing a trailing newline:

backspace() {
        echo -e "\b\c"
}

The cleanup function prints a backspace and then exits.  The cleanup function is run by trap when it receives a SIGINT (2):

cleanup() {
        backspace
        exit
}
trap "cleanup" 2

The main body of the script, is an infinite loop which generates, a random number using the special variable $RANDOM. This random is assigned to the variable num, utilizing only the first digit. After printing that number, the script sleeps num tenths of seconds, and the backspace function is called:

while :
do
        num=${RANDOM:0:1}
        printf $num
        sleep ".$num"
        backspace
done

When organizations need to create an application (most likely doing CRUD), they create both the application logic and user interface. Typically, this is done via a web application whose user interface is HTML. This essentially decides how the user can best utilize application logic.

CRUD applications can be used seamlessly in a GUI, via the command line, or inside other applications by following these three principles:

1. Decouple the user interface from the application
2. Use a standard and stateless authentication mechanism
3. REST

Decouple the user interface from the application

Do not send HTML to the browser, send XML and an associated style sheet. The browser will then render the document. My sitemap is an example. This makes the page both readable in a browser and machine processable. (Note, this is very basic style sheet.)

This way, anyone can create a client side user interface to your “application.” Your user interface, simply becomes the default user interface. Anyone can create their own. Bonus points if you provide an easy method of sharing these alternative user interfaces.

Use a standard and stateless authentication mechanism

Use only HTTP Basic Authentication over SSL. Being stateless and standard, this protocol is simple and leverages a ton pre-built tools. While Apache/IIS implement Basic Authentication, it is important to understand that Basic Authentication is simply a protocol for communicating credentials. You can use any authentication store. PHP.net has a good overview of HTTP Authentication.

REST

I had never heard of REST until last year. While speaking with an exceedingly intelligent colleague of mine – I explained how if I had designed this particular GUI I would have let users query data by simply modifying the URL. Example:

http://gui/servers?platform=linux&active=true

He said, “REST!”

This is SO simple, just use GET, be stateless, use logical names, and allow selection via all characteristics. UPDATE: This is not REST, but will get the job done. I’d prefer if you implemented REST. (See comments.)

Final Thoughts

Not all data nicely fits on a single line or few lines. However, in the vast majority of cases, records can be displayed in a grep’able format. As such, its trivial to create a parameter, say f=pt, which will output the data in some line based format. At the very least, xml can be displayed in a format with is grep’able. Instead of:

<records>
<record id="1">
<key name="abc" val="123" />
</record>
<record id="2">
<key name="def" val="456" />
</record>
</records>

Do this:

<records>
<record id="1"><key name="abc" val="123" /></record>
<record id="2"><key name="def" val="456" /></record>
</records>

Many times, a separate “Web Services API” is created to allow people to extract data in a machine processable format. However, if you follow the these three principles, your GUI and API are one in the same. There is no need to create a separate non-human API. Furthermore, in my experience, there is rarely a need for reference documentation. The API is self explanatory.

The other day I came across Solaris’s implementation of /bin/true. I use “true” for infinite while loops quite often. However, I began to wonder what legitimate uses existed for “false”. Example: disabling shell access for user accounts. Do you know of another legitimate use of false? Submit an example by commenting on this post and win Chris F.A. Johnson’s Shell Scripting Recipes.

Rules:

1. “Disabling shell access for user accounts” is not a valid submission.
2. You cannot submit someone else’s submission.
3. One submission per person.
4. I will take submissions until March 9, 2008 10PM CST.
5. One winner will be randomly chosen from the valid submissions.