BASH Cures Cancer

Quite some time ago, someone wrote me to ask about a possible article on process substitution. Sadly, I could not find the email so I cannot credit them. As you likely have guessed, I am finally writing a post on process substitution.

Many times I have used pipelines and temporary files when process substitution would be a much cleaner solution.

First, I am going to create two test files:

$ dd if=/dev/urandom of=file-small count=750001
$ dd if=/dev/urandom of=file-large count=1000000
$ ls -l file-*
-rw-r--r-- 1 noland noland 512000000 Mar 23 08:53 file-large
-rw-r--r-- 1 noland noland 384000512 Mar 23 08:49 file-small

I thought of writing this article while writing a script to test ftp servers and file locking. As such I will upload the small file to a file named append-example:

$ curl -T file-small --user noland ftp://localhost/append-example
Enter host password for user 'noland':
$ ls -l append-example
-rw-r--r-- 1 noland noland 384000512 Mar 23 11:52 append-example

Now I will append  the large file:

$ curl -s -a -T file-large --user noland ftp://localhost/append-example
Enter host password for user 'noland':
$ ls -l append-example
-rw-r--r-- 1 noland noland 896000512 Mar 23 11:54 append-example

I am going to use dd and process substituion to caculate the MD5 hash of the first upload:

$ md5sum file-small <(dd if=append-example count=750001 status=noxfer)
dfabff7441bd814145a804e03d333864  file-small
1000000+0 records in
1000000+0 records out
dfabff7441bd814145a804e03d333864  /dev/fd/63

Now the portion that was appended:

$ md5sum file-large <(dd if=append-example  skip=750001 status=noxfer)
1b8daed9e435fc90b4a49d74b55f96f4  file-large
1000000+0 records in
1000000+0 records out
1b8daed9e435fc90b4a49d74b55f96f4  /dev/fd/63

When you place a command inside <( ) the shell sets standard output of the command to pipe inside /dev/fd/ and replaces the command with that pipe. Here is the classic example:

$ echo <(echo) <(echo) <(echo) <(echo)
/dev/fd/63 /dev/fd/62 /dev/fd/61 /dev/fd/60

In my script I use process substitution as below (effectively) which feels exeedingly clean:

$ read hash name < <(md5sum <(dd if=append-example skip=750001 status=noxfer))
1000000+0 records in
1000000+0 records out
$ printf "hash=%s name=%s\n" $hash $name
hash=1b8daed9e435fc90b4a49d74b55f96f4 name=/dev/fd/63