BASH Cures Cancer

Often I need or want to do some type of performance testing. Given my ideas on software development, I can usually do this by making simple HTTP requests. I use curl for this. While you may be tempted to do this in a for loop (or worse, actually write something!):

$ time for i in {1..1000}; do curl -s "http://bashcurescancer.com/blank.html";done

real    0m23.436s
user    0m6.416s
sys     0m7.351s

Curl provides the same functionality:

$ time curl -s "http://bashcurescancer.com/blank.html?[1-1000]"

real    0m6.561s
user    0m0.294s
sys     0m0.494s

Here are the details from the curl manual:

The URL syntax is protocol dependent. You’ll find a detailed description in RFC 3986.

You can specify multiple URLs or parts of URLs by writing part sets within braces as in:

http://site.{one,two,three}.com

or you can get sequences of alphanumeric series by using [ ] as in:

ftp://ftp.numericals.com/file[1-100].txt
ftp://ftp.numericals.com/file[001-100].txt    (with leading zeros)
ftp://ftp.letters.com/file[a-z].txt

No nesting of the sequences is supported at the moment, but you can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order.

Since curl 7.15.1 you can also specify step counter for the ranges, so that you can get every Nth number or letter:

http://www.numericals.com/file[1-100:10].txt
http://www.letters.com/file[a-z:2].txt

If you specify URL without protocol:// prefix, curl will attempt to guess what protocol you might want. It will then default to HTTP but try other protocols based on often-used host name prefixes. For example, for host names starting with “ftp.” curl will assume you want to  speak FTP.

Curl  will  attempt  to re-use connections for multiple file transfers, so that getting many files from the same server will not do multiple connects / handshakes. This improves speed. Of course this is only done on files specified on a single  command  line  and  cannot  be  used
between separate curl invokes.

This is important as it helps measure the actual change being tested. A for loop, by creating a new process every loop, will fill up your test with “local” time. Using a single curl process eliminates this - which should allow you to see the results of your test in a more transparent manner.

For example, lets say you have a change that reduces page production time. Your not sure how long, so you decide to run 1000 tests. Eliminating a second from a 23 second tests is not 5 percent. While removing a second from a 6 second test, is almost 20%.