10 Steps to Beautiful Shell Scripts
October 16th, 2007
Linux gurus don’t use cut, awk, and sed when they want to replace or strip out a portion if a variable. They use parameter substitution. You can learn parameter substitution in less than 2 minutes. Here is the ten things you need to know:
- Shell variables can be specified like $var or this ${var}.
$ var='a.ads,fssd2342%asd234#@.,&%,sdfgsdfgas4352'
echo ${var}
a.ads,fssd2342%asd234#@.,&%,sdfgsdfgas4352
- ${#var} is the length of the variable.
$ echo ${#var}
42 - ${var:pos} substrings the variable starting at pos.
$ echo ${var:10}
2342%asd234#@.,&%,sdfgsdfgas4352 - ${var:pos:len} substrings the variable starting at pos with a max length of len.
$ echo ${var:10:5}
2342% - ${var#pattern} strips pattern from the front or left hand side of the variable. This form is not greedy meaning it stops as soon as the pattern is matched. ${var##pattern} is the greedy form.
$ echo ${var#*,}
fssd2342%asd234#@.,&%,sdfgsdfgas4352
$ echo ${var##*,}
sdfgsdfgas4352 - ${var%pattern} strips pattern from the back or right hand side of the variable. This form is not greedy meaning it stops as soon as the pattern is matched. ${var%%pattern} is the greedy form.
$ echo ${var%,*}
a.ads,fssd2342%asd234#@.,&%
$ echo ${var%%,*}
a.ads - ${var/pattern/replacement} replaces pattern with replacement once.
$ echo ${var/a/A}
A.ads,fssd2342%asd234#@.,&%,sdfgsdfgas4352 - ${var//pattern/replacement} replaces pattern with replacement globally.
$ echo ${var//a/A}
A.Ads,fssd2342%Asd234#@.,&%,sdfgsdfgAs4352 - ${var/#pattern/replacement} if variable beginning matches the pattern it is replaced with replacement.
$ echo ${var/#a./llll}
llllads,fssd2342%asd234#@.,&%,sdfgsdfgas4352 - ${var/%pattern/replacement} if variable end matches the pattern it is replaced with replacement.
$ echo ${var/%352/llll}
a.ads,fssd2342%asd234#@.,&%,sdfgsdfgas4llll
October 16th, 2007 at 1:19 pm
Excellent ! thanks a lot !
October 16th, 2007 at 10:26 pm
I’ll make good use of some of these…
There’s a typo at No.5 though, it should be a curly brace not a parenthesis, I only noticed this because I’ve printed this out to look at again later.
October 16th, 2007 at 11:15 pm
Woops, your right. I’ll fix that. Thanks!
October 18th, 2007 at 8:45 am
Hmmm, seems cool.
Joe joestain13@yahoo.com
October 18th, 2007 at 5:37 pm
Cool
but I think at least for now, I’ll stick with cut and awk because I like to read what I wrote last week
October 18th, 2007 at 6:24 pm
It definitely takes some getting used to.
October 18th, 2007 at 9:55 pm
Very nice – I love Bash and have written many smaller shell scripts. This is a most useful reference.
October 18th, 2007 at 10:36 pm
Sweet, but I like to keep it more compatible with “true /bin/sh” which means that all this fancy stuff does belong to sed, awk, tr…
October 19th, 2007 at 2:16 am
Wow very cool, didn’t know about this, I’m always piping through sed…
October 19th, 2007 at 12:49 pm
Wow – I did not know bash was this powerful. Thanks.
October 19th, 2007 at 2:25 pm
Getting into practice with these might reduce the number of times I launch perl for some trivial text manipulations.
October 19th, 2007 at 11:30 pm
[...] 10 Steps to Beautiful Shell Scripts Linux gurus don’t use cut, awk, and sed when they want to replace or strip out a portion if a variable. They use parameter substitution. You can learn parameter substitution in less than 2 minutes. (tags: regex) [...]
October 22nd, 2007 at 2:45 pm
Interesting article.
You’ve inverted the info for greedy and non-greedy though.
You say, for example, that ${var##pattern} is the greedy form, but then, in the example, it’s the version that returns the smallest string :
$ echo ${var#*,}
fssd2342%asd234#@.,&%,sdfgsdfgas4352
$ echo ${var##*,}
sdfgsdfgas4352
October 22nd, 2007 at 11:53 pm
Thierry,
The single # stops when it finds the shortest match. The double ## stops when it finds the longest match. Thus the double ## is described as “greedy”. Here is another example:
$ v='1,2,3,4'
$ echo ${v#*,}
2,3,4
$ echo ${v##*,}
4
October 26th, 2007 at 4:53 pm
“Sweet, but I like to keep it more compatible with “true /bin/sh” which means that all this fancy stuff does belong to sed, awk, tr…”
Which “true /bin/sh” are you referring to? A true Bourne shell? Or a standard UNIX/POSIX shell?
It has been many years since a *nix system came without a POSIX shell, which includes some of the expansions in the article. On many, if not most *nix systems, /bin/sh is a POSIX shell; the others have a POSIX shell tucked away somewhere.
These three articles describe the Bourne expansions, POSIX expansions, and the extended expansions in ksh93 and bash:
http://wiki.ittoolbox.com/index.php/Parameter_expansions
http://wiki.ittoolbox.com/index.php/POSIX_shell_expansions
http://wiki.ittoolbox.com/index.php/Extended_parameter_expansions
October 30th, 2007 at 10:54 pm
Parameter substitution is truly a remarkable thing. The other night, I was trying to get the ‘rename’ function to alter the *start* of a group of filenames, to no avail. I’m renaming files, so I should use rename, right?
Then, I thought, “why am I doing this?”
Solved with,
for file in *
do
mv $file ${file/#somestring/someotherstring}
done
All too easy…
November 1st, 2007 at 2:20 pm
Thanks for this tip. I have been looking for something like this for years and never knew it existed. This is so much cleaner than sed for so many of my daily tasks.
November 8th, 2007 at 3:17 am
Please also include variable substitutions like
${var:?”message”}-That prints message if varible does not exist,${var:-word},…
February 12th, 2008 at 1:37 am
[...] If your interested on why I had to create a patch, read on. Otherwise you may be interested in an older post 10 Steps to Beautiful Shell Scripts. [...]
March 13th, 2008 at 6:26 pm
[...] a random number using the special variable $RANDOM. This random is assigned to the variable num, utilizing only the first digit. After printing that number, the script sleeps num tenths of seconds, and the backspace function is [...]
April 2nd, 2008 at 4:16 am
Very informative.
I have been working as a Linux Admin for almost a year now. Found this info very interesting.
Some more pages like this and we will have superb BASH scripts coming from everyone.
Great post !
Thanks
July 4th, 2008 at 7:46 am
superb
July 5th, 2008 at 2:10 am
I guess one of the mistakes we all make at the beginning is using the wrong tool for the job because we do not yet know what else is available. I have been abusing sed for a while now and getting frustrated with it. Once I found out about parameter substitution I was much happier.
July 22nd, 2008 at 2:25 am
Thanks! I was trying to understand a shell script which was written by an expert and he had extensively used these things. Searched a lot and finally got ur article. I can understand the script easily now.
Thanks once again
Karthik
July 22nd, 2008 at 4:34 am
What if we have a long record with all fiels as pipe seperated?
eg:
abc|def1|ghi2|jkl3|mno4|pqr5|stu6|vwx7
Here if i need to extract the 5th field, is there a way to do this using parameter substitution in a variable (supposing I initialize the variable with the content of the record mentioned above?
Any way this was a nice thing to learn and it will take some time to master it.
regards
August 19th, 2008 at 10:56 am
[...] http://bashcurescancer.com/10-steps-to-beautiful-shell-scripts.html [...]
November 3rd, 2008 at 12:51 pm
> “true /bin/sh”
In BusyBox v1.1.3 examples 1,2,5 are working.
January 9th, 2009 at 4:56 pm
[...] cannot search Google for special characters, I Googled “Bash greedy” and luckily found 10 Steps to Beautiful Shell Scripts, which just so happened to contain the technique I was looking for [...]
April 29th, 2009 at 11:44 pm
It is really helpful, thanks a lot