BASH Cures Cancer » Links

The best in command line xml: XMLStarlet

Brock Noland — Tue, 24 Jun 2008 04:32:48 +0000

Quite some time ago I wrote about using xsltproc to process xml on the command line. Thank fully someone pointed out XMLStarlet. I now use XMLStarlet almost every day. I work with a variety of REST based API’s gather information. XMLStartlet along with a simple for loop or xargs gives you an exceedingly powerful set of tools.

Here is a quick introduction into the power of XMLStarlet. This is just a teaser as I cannot share the data I work with. However, you should be able to see the power of this tool.

All the links from my RSS feed:

$ curl -s 'http://bashcurescancer.com/rss/' | xml sel -t -m '//link' -v '.' -n

http://bashcurescancer.com

http://bashcurescancer.com/processing-xml-on-the-command-line.html

http://bashcurescancer.com/do-not-close-stderr.html

http://bashcurescancer.com/prepend-to-a-file-with-sponge-from-moreutils.html

http://bashcurescancer.com/bug-in-curl-is-fixed.html

http://bashcurescancer.com/using-kill-to-see-if-a-process-is-alive.html

http://bashcurescancer.com/performance-testing-with-curl.html

http://bashcurescancer.com/new-command-prepend.html

http://bashcurescancer.com/shell-function-which-webserver-does-that-site-run.html

http://bashcurescancer.com/exposing-command-line-programs-as-web-services.html

http://bashcurescancer.com/wrapping-dynamic-languages-in-shell-without-an-extra-script.html

Or how about “Title: link”

$ curl -s 'http://bashcurescancer.com/rss/' | xml sel -t -m '//item' -v 'title' -o ': ' -v 'link' -n

Processing XML on the Command Line: http://bashcurescancer.com/processing-xml-on-the-command-line.html

Do not close stderr: http://bashcurescancer.com/do-not-close-stderr.html

prepend to a file with sponge from moreutils: http://bashcurescancer.com/prepend-to-a-file-with-sponge-from-moreutils.html

Bug in Curl is fixed: http://bashcurescancer.com/bug-in-curl-is-fixed.html

using kill to see if a process is alive: http://bashcurescancer.com/using-kill-to-see-if-a-process-is-alive.html

Performance testing - with curl: http://bashcurescancer.com/performance-testing-with-curl.html

New command: prepend: http://bashcurescancer.com/new-command-prepend.html

Shell Function - Which Webserver Does That Site Run?: http://bashcurescancer.com/shell-function-which-webserver-does-that-site-run.html

Exposing command line programs as web services: http://bashcurescancer.com/exposing-command-line-programs-as-web-services.html

Wrapping dynamic languages in shell without an extra script: http://bashcurescancer.com/wrapping-dynamic-languages-in-shell-without-an-extra-script.html

You may need to do some reading on xpaths and xsl stylesheets to use the full power of the tool.

using kill to see if a process is alive

Brock Noland — Thu, 10 Apr 2008 04:43:55 +0000

I am making some changes to the moreutils sponge command. Sponge provides a method of prepending which is less specialized than my prepend util. However, it has trouble with large amounts of input.

Regardless, while testing my changes, I want to watch it operate. Normally, you would just do so from a second terminal. That is a pain. kill -0 can be very useful for this. After backgrounding the command, I assign the pid (via the variable $!) to $pid using eval. eval is needed to stop BASH from expanding $! until after the background operation.

After that, I enter a while loop on kill -0 $pid, which will not kill $pid, but will return successfully until $pid has died:

# cat large-file-GB | ./sponge large-file-GB-copy & eval 'pid=$!'; while kill -0 $pid; do sleep 10; ls -lh large-file* /tmp/sponge.*; echo;done
[1] 7937
-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw------- 1 root root 128M 2008-04-09 17:23 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw------- 1 root root 384M 2008-04-09 17:23 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw------- 1 root root 877M 2008-04-09 17:24 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw-r--r-- 1 root root  20M 2008-04-09 17:24 large-file-GB-copy
-rw------- 1 root root 896M 2008-04-09 17:24 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw-r--r-- 1 root root 413M 2008-04-09 17:25 large-file-GB-copy
-rw------- 1 root root 896M 2008-04-09 17:24 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw-r--r-- 1 root root 836M 2008-04-09 17:25 large-file-GB-copy
-rw------- 1 root root 896M 2008-04-09 17:24 /tmp/sponge.JMsBWG

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw-r--r-- 1 root root 920M 2008-04-09 17:25 large-file-GB-copy
[1]+  Done                    cat large-file-GB | ./sponge large-file-GB-copy
ls: cannot access /tmp/sponge.*: No such file or directory

-rw-r--r-- 1 root root 977M 2008-04-09 16:18 large-file-GB
-rw-r--r-- 1 root root 977M 2008-04-09 17:25 large-file-GB-copy
-bash: kill: (7937) - No such process
# md5sum large-file-GB*
b5c667a723a10a3485a33263c4c2b978  large-file-GB
b5c667a723a10a3485a33263c4c2b978  large-file-GB-copy

Linux Command Line Tips & Reference Sheet

Brock Noland — Mon, 22 Oct 2007 05:41:59 +0000

This “Linux Command Line Tips” is a supberb listing of options, commands, and pipe lines. I’d suggest bookmarking.

Note: I want BASH Cures Cancer to be original content. However, this list was too good, so I had to share.