Useful sed commands

Because I always for get these things and get error messages

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note: I was using these commands on Ubuntu and find that they are not working on Amazon Linux.

Some of these may have shortcuts but I like to keep things readable (as possible with sed).

I’m using the file temp.txt below and keeping my operations separate for readability versus some tricky twisted short command. Sed is confusing enough.

I’m not a sed master but have used it successfully to manipulate output into formats I desire repeatedly. It’s a very handy tool. I’m a hacker who hasn’t memorized everything sed can do and what all the commands mean yet. I just find what works quickly as I’m usually in a hurry! So I’ll probably add commands to this over time as I use them.

I like sed better than awk. I almost never use awk — only as a last resort when I happen to stumble across an example and cannot figure out how to to it with sed — because I find it even more cumbersome than sed.

Sed is also faster than Python or just about any other option for quick text transformations.

-e

You can use -e to use multiple commands on a file or block of text.

pipe |

You can use a pipe | to run other sed commands on your initial output — so you can join different functionality together.

Replace text in output

This is the command I use most.

Replace text in some output, such as replacing a double quote with nothing to remove it.

echo '"something"'

output: 
"something"

echo '"something"' | sed 's/"//g'

output:
something

You can also use it with files with the -i switch — it will replace the characters in the content. But you’re better off testing it first and when you get the output you want, then run it on the file.

sed -i [some sed command] your-file.txt

Find and replace a value in all the files in a directory:

find . -type f -exec sed -i 's/mysecret/xxxxxxx/g' {} +

Replaces paths in a file that have a / character in them. Change the sed separator to anything else and it will work. In this case I changed the sed separator to | instead of / and then I can put the file paths in my two values to find and replace.

 find . -type f -exec sed -i 's|container/shared|job/shared|g' {} +

If you want to test first use cat to output the file, then run the command against it as shown in the next example.

Remove everything from the beginning up to a a character — an open curly brace in this example {:

sed 's/^.*{//g'

Remove everything from a character to the end — a close curly brace in this example:

sed 's/}.*$//g'

Delete all characters except alphanumeric:

sed 's|[^[:alnum:]]||g'

Delete all characters except alphanumeric and /

sed's|[^[:alnum:]/]||g'

Delete all characters except alphanumeric and / and dash -.

To treat a dash as a normal (not special) character add -r and make sure the deash is the last character in the list.

sed -r 's|[^[:alnum:]/-]||g'

Delete everything after a phrase in a file:

cat temp.txt | sed '/^###END###$/q'

or

sed -e '/^###END###$/q' temp.txt

Delete everything before a phrase in a file:

sed -e '1,/###START###/d' temp.txt

Those commands can be used to dump the results to a new file:

sed [some command and file] > some-new-file.txt

Or you can edit the file directly once you get the command working by changing -e to -i and it will update the file contents.

To remove everything up to the last occurrence of a phrase in a string

$ echo "abc123" | sed 's/^.*abc/abc/'

To remove everything up to the first occurrence of a phrase in a string:

sed ‘s/[^{{]*{{//’

To remove everything after a phrase in a string

sed 's|test.*||'

^ to insert at the beginning

sed 's/^/something/'

$ to insert something at the end

sed 's/$/something/'

Delete everything before ###START### and everything after ###END###

sed -e '/^###END###$/q' -e '1,/###START###/d' temp.txt

Delete everything before ###START### and everything after ###END### PLUS the start and end phrases

sed -e '/^###END###$/q' -e '1,/###START###/d' temp.txt | sed 's/###START###//' | sed 's/###END###//'

Replace a value in a file (use -i)

sed -i 's/somevalue/someothervalue/g' [filename]

Escape a special character like \ or ‘ with another \

sed -i 's\some\\va\'lue|someothervalue|g' [filename]

Use a different delimiter (you can use anything so in the example below I change / to | and you don’t have to escape the \.

sed -i 's|some\va\'lue|someothervalue|g' [filename]

Other resources:

You also might to check out the cut command. I often use these two commands together.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab