A very common cause is where the file was created on a Windows host and then uploaded to a Unix host for execution. (Think of the millions of websites using shared hosting.) The problem here, as no doubt you maybe aware, is that the file contains the dreaded “carriage return” before every newline. This is so common, there is even a standard command dos2unix for converting these files.

However, I had yet to see a reason as to WHY that carriage return after the hash bang causes a problem. I just accepted it as fact. Today, I decided to figure out why and where it failed.

As it turns out, it fails in the kernel. To figure this out, I downloaded the latest kernel (2.6.23.12) to my RHEL 5 host and compiled it with the standard config file that comes with RHEL 5. The configuration file was for a much older kernel, but worked for my purposes. I booted it to make sure I had a working copy of 2.6.23.12. Then I went into the kernel source tree and started modifying stuff!

To be sure, I am no kernel hacker, nor even a C hacker. I had never successfully modified the kernel source, though I haven’t tried in many years. But after three compilations, I was able to figure out where in (2.6.23.12) exec system call was taking place and was able to insert some code to print a custom message. The file I modifed was “fs/exec.c”:

[root@test1 linux-2.6.23.12]# ls -l fs/exec.c
-rw-rw-r– 1 root root 42231 Jan 7 17:44 fs/exec.c

In that file, inside the function search_binary_handler() I inserted the following code:

1139 printk(“{“);
1140 int z = 0;
1141 for(; z < strlen(bprm->buf); z++) {
1142 if(bprm->buf[z] == ‘\r’) {
1143 printk(“(carriage return)”);
1144 } else {
1145 printk(“%c”, bprm->buf[z]);
1146 }
1147 }
1148 printk(“}\n”);

The line numbers may be slightly off as I had inserted and removed code above. After compliation and booting, I was able to give it a shot. I created a sample script with a carriage return after the hash bang interpreter portion.

[root@test1 ~]# echo -e ‘#!/bin/bash\r\n/usr/bin/id’ >id.sh
[root@test1 ~]# chmod +x id.sh

And BAM, it worked!

[root@test1 ~]# ./id.sh
{#!/bin/bash(carriage return)}
-bash: ./id.sh: /bin/bash^M: bad interpreter: No such file or directory

As you can see, the kernel was actually trying to execute “/bin/bash\r” which of course, not a valid file. Here’s a screen shot:

Update: In response to reader requests, I wrote an explanation of my investigation: On the case of carriage returns and kernel exec system calls.