Bash Scripts — Part 7 — Word Processing With Sed

Last time we talked about functions in bash scripts, in particular, how to call them from the command line. Our topic today is a very useful tool for processing string data — a Linux utility called sed. It is often used to work with texts in the form of log files, configuration files, and other files.

If you manipulate data in some way in bash scripts, you should be familiar with the sed and gawk tools. Here we will focus on sed and text manipulation, as this is a very important step in our journey through the vast expanses of bash script development. Now we will cover the basics of working with sed, as well as look at more than three dozen examples of using this tool.

Sed basics

The sed utility is called a stream text editor. In interactive text editors like nano, texts are manipulated using the keyboard, editing files, adding, deleting, or changing texts. Sed allows you to edit data streams based on developer-defined sets of rules. This is how this command is called:

$ sed options file

By default, sed applies the specified command set rules to STDIN. This allows data to be passed directly to sed.

For example, like this:

~$ echo "This is a test" | sed 's/test/another test/'
This is a another test

In this case, sed replaces the word “test” in the line passed for processing with the words “another test”. For the formatting of the rule for processing text enclosed in quotation marks, forward slashes are used. In our case, the command of the form is applieds/pattern1/pattern2/. The letter "s" is an abbreviation of the word "substitute", that is, we have a replacement command. Sed, executing this command will scan the transmitted text and replace it found in fragments (about - exactly what, discussed below), the correspondingpattern1onpattern2.

Above is a primitive sed example to get you up to speed. In fact, sed can be used in much more complex word processing scenarios, such as working with files.

Below is a file that contains a piece of text, and the results of processing it with this command:

~$ cat myfile
This is a test

~$ sed 's/test/another test' ./myfile
This is a another test

This is the same approach that we used above, but sed now processes the text stored in the file. However, if the file is large enough, you may notice that sed processes the data in chunks and prints what has been processed to the screen, without waiting for the entire file to be processed.

Sed does not change data in the processed file. The editor reads the file, processes what it read, and sends the output toSTDOUT. To make sure that the original file has not changed, it is enough to open it after it has been passed to sed. If necessary, the output of sed can be redirected to a file, possibly overwriting the old file. If you are familiar with one of the previous materials In this series on redirecting input and output streams, you should be able to do just that.

Executing command sets when invoking sed

To perform multiple actions on the data, use a key -ewhen invoking sed. For example, here's how to arrange the replacement of two pieces of text:

~$ sed -e 's/This/That/; s/test/another test/' ./myfile
That is a another test

Both commands are applied to each line of text in the file. Separate them with a semicolon, with no space between the end of the command and the semicolon. To enter several patterns of text processing when invoking sed, you can, after entering the first single quote, press Enter, and then enter each rule on a new line, not forgetting the closing quote:

~$ sed -e '
> s/This/That/
> s/test/another test/' ./myfile
That is a another test

Reading commands from a file

If you have many sed commands to process text with, it is usually best to write them to a file first. To specify a file containing commands to sed, use the switch -f:

Here are the contents of the file mycommands:

s/This/That/
s/test/another test/

Let’s call sed, passing the command file and the file to be processed to the editor. The result when calling such a command is similar to that obtained in the previous examples:

~$ sed -f mycommands myfile
That is a another test

Replacement command flags

Take a close look at the following example. This is what is in the file and what sed will get after processing it:

~$ cat myfile
This is a test and I like this test.
This is the next test of the test scripts.

~$ sed 's/test/another test/' myfile
This is a another test and I like this test.
This is the next another test of the test scripts.

The replace command normally processes a file consisting of several lines, but only the first occurrences of the desired text fragment in each line are replaced. In order to replace all occurrences of a pattern, use the corresponding flag.

The scheme for writing the replacement command when using flags looks like this:

s/pattern/replacement/flags

The execution of this command can be modified in several ways.

• When transmitting the number, the sequence number of the occurrence of the pattern in the string is taken into account; this particular occurrence will be replaced.
• The flag gindicates to process all occurrences of the pattern in the string.
• The flag pindicates to display the contents of the original string.
• The view flag w filetells the command to write the text processing results to a file.

Consider the use of the first version of the replace command, indicating the position of the replaced occurrence of the desired fragment:

~$ sed 's/test/another test/2' myfile
This is a test and I like this another test.
This is the next test of the another test scripts.

Here we have indicated, as the replacement flag, the number 2. This led to the fact that only the second occurrence of the desired pattern in each line was replaced. Now let’s try the global replacement flag -g:

~$ sed 's/test/another test/g' myfile
This is a another test and I like this another test.
This is the next another test of the another test scripts.

As you can see from the output, this command replaced all occurrences of the pattern in the text.

The replace command flagpallows you to print lines that match, while the option-nspecified when invoking sed suppresses normal output:

~$ cat myfile
This is a test.
This is a different one.

~$ sed -n 's/test/another test/p' myfile
This is a another test.

As a result, when sed is run in this configuration, only the lines (in our case, one line) are displayed in which the specified text fragment was found.

Let’s use the flagwthat allows saving the text processing results to a file:

~$ sed 's/test/another test/w output' myfile
This is a another test.
This is a different one.

~$ cat output
This is a another test.

It is clearly seen that during the operation of the command, the data is output to STDOUT, while the processed lines are written to the file whose name is specified afterw.

Separators

Imagine what needs to be replaced /bin/bashwith /bin/cshin the file /etc/passwd. The task is not so difficult:

~$ sed 's/\/bin\/bash/\/bin\/csh/' /etc/passwd

However, all this does not look very good. The fact is that since forward slashes are used as separator characters, the same characters in the lines passed to sed must be escaped. As a result, the readability of the command suffers.

Fortunately, sed allows us to define delimiter characters ourselves for use in the replacement command. The delimiter is the first character that is encountered after s:

~$ sed 's!/bin/bash!/bin/csh!' /etc/passwd

In this case, an exclamation mark is used as a separator, as a result, the code is easier to read and looks much cleaner than before.

Selecting text fragments for processing

So far, we’ve called sed to process the entire data stream passed to the editor. In some cases, sed only needs to process a portion of the text — a particular line or group of lines. To achieve this goal, you can take two approaches:

• Set a limit on the numbers of processed lines.
• Specify the filter for which rows are to be processed.

Let’s consider the first approach. There are two possible options. The first, discussed below, provides for specifying the number of one line to be processed:

~$ cat myfile
This is a test.
This is the second test.
This is the

~$ sed '2s/test/another test/' myfile
This is a test.
This is the second another test.
This is the third test.
This is the fourth test.

The second option is a range of lines:

~$ sed '2,3s/test/another test/' myfile
This is a test.
This is the second another test.
This is the third another test.
This is the fourth test.

In addition, you can call the replace command so that the file is processed from a certain line to the end:

~$ sed '2,$s/test/another test/' myfile
This is a test.
This is the second another test.
This is the third another test.
This is the fourth another test.

In order to process only the lines matching the specified filter using the replace command, the command must be called as follows:

~$ sed '/raevskym/s/bash/csh/' /etc/passwd
raevskym:x:1000:1000:raevskym,,,:/home/raevskym:/bin/csh

Similar to what was discussed above, the template is passed before the command name s.

Here we have used a very simple filter. In order to fully reveal the capabilities of this approach, you can use regular expressions. We will talk about them in one of the next materials in this series.

Deleting lines

Sed is good for more than just replacing one sequence of characters in strings with another. With its help, namely, using the command d, you can remove lines from the text stream.

The command call looks like this:

~$ sed '3d' myfile

We want the third line to be removed from the text. Please note that this is not a file. The file will remain unchanged, the deletion will only affect the output that sed will generate.

~$ cat myfile
This is a test.
This is the second test.
This is the third test.
This is the fourth test.

~$ sed '3d' myfile
This is a test.
This is the second test.
This is the fourth test.

If youddo not specify the line number to be deleted when calling the command, all lines in the stream will be deleted.

Here's how to apply the commanddto a range of strings:

~$ sed '2,3d' myfile
This is a test
This is the fourth test.

Here’s how to delete lines starting with a given one and ending at the end of the file:

~$ sed '3,$d' myfile
This is a test.
This is the second test.

Lines can also be deleted by a pattern:

~$ cat myfile
This is a test.
This is the second test.
This is the third test.
This is the fourth test.
This is another line.

~$ sed '/test/d' myfile
This is another line.

When calling,dyou can specify a pair of patterns - the lines in which the pattern is found, and those lines that are between them will be deleted:

~$ cat myfile
This is a test.
This is the second test.
This is the third test.
This is the fourth test.
This is another line.

~$ sed '/second/,/fourth/d' myfile
This is a test.
This is another line.

Inserting text into a stream

With sed, you can insert data into a text stream using the commands iand a:

• The command iadds a new line before the given one.
• The command aadds a new line after the given one.

Let’s consider an example of using the command i:

~$ echo "Another test" | sed 'i\First test '
First test
Another test

Now let’s take a look at the commanda:

~$ echo "Another test" | sed 'a\First test '
Another test
First test

As you can see, these commands add text before or after data from the stream. What if you need to add a line somewhere in the middle?

Here we will be helped by specifying the number of the reference line in the stream, or template. Note that addressing strings as a range will not work here. Let’s call the commandispecifying the line number before which we need to insert a new line:

~$ sed '2i\This is the inserted line.' myfile
This is a test.
This is the inserted line.
This is the second test.
This is the third test.
This is the fourth test.
This is another line.

Let’s do the same with the commanda:

~$ sed '2a\This is the appended line.' myfile
This is a test.
This is the second test.
This is the appended line.
This is the third test.
This is the fourth test.
This is another line.

Note the difference in the behavior of theiand commandsa. The first one inserts a new line before the specified one, the second one after.

Replacing strings

The command callows you to change the content of an entire line of text in a data stream. When calling it, you need to specify the line number, instead of which you need to add new data to the stream:

~$ sed '3c\This is a modified line.' myfile
This is a test.
This is the second test.
This is a modified line.
This a the  fourth test.
This another line.

If you use a plain text or regular expression pattern when invoking a command, all strings matching the pattern will be replaced:

~$ cat myfile
This is a test.
This is the second test.
This is the third test.
This is the fourth test.
This is another line.

~$ sed '/This is/c This is a changed line of text.' myfile
This is a changed line of test.
This is a changed line of test.
This is a changed line of test.
This is a changed line of test.
This is a changed line of test.

Replacing characters

The command yworks with individual characters, replacing them according to the data passed to it when called:

~$ cat myfile
This is a test.1
This is the second test.2
This is the third test.3
This is the fourth test.4
This is another line.5

~$ sed 'y/123/567/' myfile
This is a test.5
This is the second test.6
This is the third test.7
This is the fourth test.4
This is another line.5

When using this command, you must take into account that it applies to the entire text stream, you cannot restrict it to specific occurrences of characters.

Displaying line numbers

If you invoke sed with a command =, it will print the line numbers in the data stream:

~$ sed '=' myfile
1
This is a test.
2
This is the second test.
3
This is the third test.
4
This is the fourth test.
5
This is another line

The Streaming Editor printed line numbers before their contents.

If you pass a pattern to this command and use the sed option-n, only the line numbers that match the pattern are printed:

~$ sed -n '/test/=' myfile
1
2
3
4

Reading data to insert from a file

Above, we looked at how to insert data into a stream, specifying what to insert, right when you call sed. You can also use a file as a data source. To do this, use a command rthat allows you to insert data from the specified file into the stream. When calling it, you can specify the line number, after which you need to insert the file content or a template.

Let's consider an example:

~$ cat newfile
First line in newfile
Second line in newfile

~$ sed '3r newfile' myfile
This is a test.
This is the second test.
This is the third test.
First line in newfile
Second line in newfile
This is the fourth test.
This is another line.

Here the file content hasnewfilebeen inserted after the third line of the filemyfile.

This is what happens when you apply arpattern when invoking a command:

~$ sed '/test/r newfile' myfile
This is a test.
First line in newfile
Second line in newfile
This is the second test.
First line in newfile
Second line in newfile
This is the third test.
First line in newfile
Second line in newfile
This is the fourth test.
First line in newfile
Second line in newfile
This is another line

The contents of the file will be inserted after each line matching the template.

Example

Let’s imagine the following task. There is a file in which there is a certain sequence of characters, in itself meaningless, which must be replaced with data taken from another file. Namely, let it be a file newfilein which a sequence of characters plays the role of a placeholder DATA. The data to be substituted DATAis stored in a file data.

You can solve this problem using the commands rand the dstream editor sed:

~$ cat newfile
First part of the data
Second part of the data

~$ cat myfile
This is a test.
This is the second test.
This is the third test.
DATA
This is the fourth test.
This is another line.

~$ sed '/DATA>/ {
r newfile
d}' myfile

This is a test.
This is the second test.
This is the third test.
First part of the data
Second part of the data
This is the fourth test.
This is another line.

As you can see, instead of a placeholderDATAsed has added two lines from a file to the output streamdata.

Outcome

Today we covered the basics of working with the sed stream editor. In fact, sed is a huge topic. Learning it can be compared to learning a new programming language, however, once you understand the basics, you can master sed at any level you need. As a result, your ability to process texts with it will be limited only by your imagination.

That’s all for today. Next time, let’s talk about awk data processing language. Dear Readers! Do you use sed in your day to day work? If so, please share your experience.

Source: https://habr.com/ru/company/ruvds/