Bash Scripts — Part 4 — Input and Output
Last time, in Part 3 of this bash scripting series, we talked about command line parameters and switches. Our topic today is input, output, and everything related to this.
You are already familiar with two methods of working with what command-line scripts output:
- Display the output data on the screen.
- Redirecting output to a file.
Sometimes something needs to be shown on the screen, and something needs to be written to a file, so you need to understand how input and output are handled in Linux, which means that you need to learn how to send the results of the scripts where you need to. Let’s start by talking about standard file descriptors.
Standard file descriptors
Everything in Linux is files, including input and output. The operating system identifies files using descriptors.
Each process is allowed to have up to nine open file descriptors. The bash shell reserves the first three descriptors with IDs 0, 1, and 2. Here’s what they mean.
0,STDIN- standard input stream.1,STDOUT- standard output stream.2,STDERR- standard error stream.
These three special descriptors handle script input and output. You need to understand the standard streams well. They can be compared to the foundation on which the interaction of scripts with the outside world is built. Let’s consider the details about them.
STDIN
STDIN- this is the standard input of the shell. For a terminal, standard input is the keyboard. When scripts use the input redirection character - <, Linux replaces the standard input file descriptor with the one specified in the command. The system reads the file and processes the data as if it were entered from the keyboard.
Many bash commands take input from STDINunless a file is specified on the command line to retrieve data from. For example, this is true for a team cat.
When you enter a command caton the command line without specifying parameters, it accepts input from STDIN. After you enter the next line, it catjust prints it to the screen.
STDOUT
STDOUT- the standard output of the shell. By default, this is the screen. Most bash commands output data STDOUTto the console, which causes it to appear in the console. The data can be redirected to a file by attaching it to its contents using the command >>.
So, we have a certain data file, to which we can add other data using this command:
pwd >> myfileWhat it outputs pwdwill be added to the file myfile, while the data already in it will not go anywhere.
raevskym@DESKTOP-JNF3L6H:~/bash_course$ pwd >> myfile
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./myfile
this is old file content
/home/raevskym/DesktopSo far so good, but what if you try to do something like the one shown below, accessing a non-existent filexfile, thinking all this so thatmyfilethe error message gets into the file.
ls –l xfile > myfileAfter executing this command, we will see error messages on the screen.
raevskym@DESKTOP-JNF3L6H:~/bash_course$ ls -l xfile > myfile
ls: cannot access '-l': No sudh file or directory
ls: cannot access 'xfile': No such file or directoryAn attempt to access a nonexistent file generates an error, but the shell did not redirect the error messages to the file, displaying them. But we wanted the error messages to go to the file. What to do? The answer is simple — use the third standard descriptor.
STDERR
STDERR is the standard error stream of the shell. By default, this handle points to the same thing it points to STDOUT, which is why when an error occurs, we see a message on the screen.
So, suppose you want to redirect error messages to, say, a log file, or somewhere else, instead of printing them to the screen.
Redirect error stream
As you already know, a file descriptor forSTDERR is 2. We can redirect errors by placing this descriptor before the redirection command:
ls -l xfile 2>myfile
cat ./myfileThe error message will now go to the file myfile.
raevskym@DESKTOP-JNF3L6H:~/bash_course$ ls -l xfile 2> myfile
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./myfile
ls: cannot access 'xfile': No such flie or directoryRedirecting error and output streams
When writing command-line scripts, a situation may arise where you need to organize both error message redirection and standard output redirection. In order to achieve this, you need to use the redirection commands for the corresponding descriptors, indicating the files where errors and standard output should go:
ls –l myfile xfile anotherfile 2> errorcontent 1> correctcontentLet’s see closer:
raevskym@DESKTOP-JNF3L6H:~/bash_course$ ls -l myfile xfile anotherfile 2> errorcontent 1> correctcontent
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./correctcontent
myfile
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./errorcontent
ls: cannot access '-l': No sudh file or directory
ls: cannot access 'xfile': No such file or directory
ls: cannot access 'anotherfile': No such file or directoryThe shell will redirect what the commandlsnormally sends toSTDOUTa filecorrectcontentdue to its construction1>. The error messages that would have endedSTDERRup in the file areerrorcontentdue to the redirection command2>.
If necessary, andSTDERR, andSTDOUTcan be redirected to the same file using the command&>:
raevskym@DESKTOP-JNF3L6H:~/bash_course$ ls -l myfile xfile anotherfile &> content
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./content
ls: cannot access '-l': No sudh file or directory
ls: cannot access 'xfile': No such file or directory
ls: cannot access 'anotherfile': No such file or directory
myfileAfter executing the command, what is intended forSTDERRandSTDOUTis in the filecontent.
Redirecting output in scripts
There are two methods for redirecting output in command-line scripts:
- Temporary redirection, or redirection of the output of one line.
- Permanently redirecting, or redirecting all or part of the output in the script.
Temporarily redirecting output
In a script, you can redirect the output of a single line to STDERR. In order to do this, it is enough to use the redirection command, specifying a descriptor STDERR, and in front of the descriptor number, you need to put an ampersand ( &) symbol :
#!/bin/bash
echo "This is an error" >&2
echo "This is normal output"If you run the script, both lines will go to the screen, since, as you already know, by default, errors are displayed in the same place as normal data.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
This is an error
This is normal outputLet’s run the script so that the outputSTDERRgoes to a file.
./myscript 2> myfile
As you can see, now the usual output is done to the console, and error messages are sent to the file.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript 2> content
This is normal output
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./content
This is an errorPermanent output redirection
If a script needs to redirect a lot of displayed data, it is echoinconvenient to add the appropriate command to each call. Instead, you can set the output to be redirected to a specific descriptor for the duration of the script using the command exec:
#!/bin/bash
exec 1>outfile
echo "This is a test of redirecting all output"
echo "from a shell script to another file."
echo "without having to redirect every line"Let’s run the script:
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat outfile
This is a test of redirecting all output
from a shell script to another file.
without having to redirect every lineIf you look at the file specified in the output redirection command, it turns out that everything that the output of the commandechowent to that file.
The commandexeccan be used not only at the beginning of the script but also in other places:
#!/bin/bash
exec 2>myerror
echo "This is the start of the script"
echo "now redirecting all output to another location"
exec 1>myfile
echo "This should go to the myfile file"
echo "and this should go to the myerror file" >&2This is what happens after running the script and viewing the files to which we redirected the output.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
This is the start of the script
now redirecting all output to another location
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./myerror
and this should go to the myerror file
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./myfile
This should go to the myfile fileFirst, the commandexecsets the redirection of output fromSTDERRto a filemyerror. Then the output of several commands isechosent toSTDOUTand displayed on the screen. After that, the commandexecsets up sending whatever goes toSTDOUTa filemyfile, and finally, we use the redirection commandSTDERRin the commandecho, which leads to writing the corresponding line to the file.myerror.
Having mastered this, you can redirect the output to where you want. Now let's talk about input redirection.
Redirecting input in scripts
To redirect input, you can use the same technique that we used to redirect output. For example, the command execallows you to make a data source for STDINa file:
exec 0< myfileThis command tells the shell to source its input from a file myfile, not a regular one STDIN. Let's see input redirection in action:
#!/bin/bash
exec 0< testfile
count=1
while read line
do
echo "Line #$count: $line"
count=$(( $count + 1 ))
doneThis is what will appear on the screen after running the script.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
Line #1: this is the first line
Line #2: this is the second line
Line #3: this is the third lineIn one of the previous articles, you learned how to use a commandreadto read user input from the keyboard. If you redirect input by making the data source a file, then the commandread, when trying to read data fromSTDIN, will read it from the file, and not from the keyboard.
Some Linux administrators use this approach to read and then process log files.
Creating your own output redirection
When redirecting input and output in scripts, you are not limited to the three standard file descriptors. As mentioned, you can have up to nine open descriptors. The other six numbered 3 through 8, can be used to redirect input or output. Any of them can be assigned to a file and used in the script code.
You can assign a descriptor for data output using the command exec:
#!/bin/bash
exec 3>myfile
echo "This should display on the screen"
echo "and this should be stored in the file" >&3
echo "And this should be back on the screen"After running the script, some of the output will go to the screen, and some will go to the file with the descriptor 3.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
This should display on the screen
And this should be back on the screen
raevskym@DESKTOP-JNF3L6H:~/bash_course$ cat ./myfile
and this should be stored in the fileCreating file descriptors for data input
You can redirect input in a script in the same way as output. Store STDINin another descriptor before redirecting input.
After finishing reading the file, you can restore STDINand use it as usual:
#!/bin/bash
exec 6<&0
exec 0< myfile
count=1
while read line
do
echo "Line #$count: $line"
count=$(( $count + 1 ))
done
exec 0<&6
read -p "Are you done now? " answer
case $answer in
y) echo "Goodbye";;
n) echo "Sorry, this is the end.";;
esacLet’s test the script:
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
Line #1: First line
Line #2: Second line
Line #3: Third line
Are you done now? y
Goodbye
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
Line #1: First line
Line #2: Second line
Line #3: Third line
Are you done now? n
Sorry, this is the end.In this example, file descriptor 6 was used to store a reference toSTDIN. Then input redirection was done, the data source for theSTDINfile became. After that, the input for the commandreadcame from the redirectedSTDIN, that is, from the file.
After reading the file, we revertSTDINto its original state by redirecting it to a descriptor6. Now, in order to check that everything is working correctly, the script asks the user a question, waits for input from the keyboard, and processes what is entered.
Closing file descriptors
The shell automatically closes file descriptors after the script finishes. However, in some cases, it is necessary to close the descriptors manually before the script finishes working. In order to close the handle, it needs to be redirected to &-. It looks like this:
#!/bin/bash
exec 3> myfile
echo "This is a test line of data" >&3
exec 3>&-
echo "This won't work" >&3After executing the script, we will receive an error message.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
./myscript: line 5: 3: Bad file descriptorThe point is that we tried to access a non-existent file descriptor.
Be careful when closing file descriptors in scripts. If you sent data to a file, then closed the handle, then opened it again, the shell will replace the existing file with the new one. That is, everything that was written to this file earlier will be lost.
Retrieving information about open handles
In order to get a list of all open descriptors in Linux, you can use the command lsof. Many distributions, like Fedora, have the utility lsofin /usr/sbin. This command is very useful as it displays information about each handle open on the system. This includes what is opened by processes running in the background and what is opened by users who are logged in.
This command has many keys, let's look at the most important ones.
-pAllows you to specify aIDprocess.-dAllows you to specify the number of a descriptor about which you want to get information.
In order to find out the PID of the current process, you can use a special environment variable $$to which the shell writes the current one PID.
The key is -aused to perform a boolean operation ANDon the results returned by using the other two keys:
raevskym@DESKTOP-JNF3L6H:~/bash_course$ lsof -a -p $$ -d 0,1,2
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
myscript 24 raevskym 0u CHR 4,1 25332 /dev/tty1
myscript 24 raevskym 1u CHR 4,1 25332 /dev/tty1
myscript 24 raevskym 2u CHR 4,1 25332 /dev/tty1The type of files associated withSTDIN,STDOUTandSTDERR — CHR (character mode). Since they all point to a terminal, the filename matches the device name assigned to the terminal. All three standard files are available for reading and writing.
Let's look at calling a commandlsoffrom a script in which, in addition to the standard, other descriptors are open:
#!/bin/bash
exec 3> myfile1
exec 6> myfile2
exec 7< myfile3
lsof -a -p $$ -d 0,1,2,3,6,7This is what happens if you run this script.
raevskym@DESKTOP-JNF3L6H:~/bash_course$./myscript
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
myscript 24 raevskym 0u CHR 4,1 25332 /dev/tty1
myscript 24 raevskym 1u CHR 4,1 25332 /dev/tty1
myscript 24 raevskym 2u CHR 4,1 25332 /dev/tty1
myscript 24 raevskym 3w REG 0,14 0 13510 ~/bash_course/myfile1
myscript 24 raevskym 6w REG 0,14 0 13229 ~/bash_course/myfile2The script opened two descriptors for output (3and6) and one for input (7). The paths to the files used to configure the descriptors are also shown here.
Output suppression
Sometimes you need to make sure that the commands in the script, which, for example, can be executed as a background process, do not display anything on the screen. To do this, you can redirect the output to /dev/null. This is something like a "black hole".
For example, here's how to suppress error messages:
ls -al badfile anotherfile 2> /dev/nullThe same approach is used if, for example, you need to clean up a file without deleting it:
cat /dev/null > myfileOutcome
Today you learned about how input and output work in command-line scripting. Now you know how to handle file descriptors, create, view and close them, you know about redirecting input, output and error streams. These are all very important in the business of developing bash scripts.
Next time, let’s talk about Linux signals, how to handle them in scripts, how to run scheduled jobs, and background tasks.
