Advanced Linux Commands for DevOps Engineers: Part-3

grep like a pro — Basics to Advanced Usages

The grep, egrep, and fgrep commands are part of the powerful text search utilities in Linux. They allow users to search through files or output for patterns of text using regular expressions (regex). Let's dive deep into these commands, their advanced usage, and how they differ.

grep stands for Global Regular Expression Print. It searches for patterns in files or input and prints lines matching the pattern.

Basic Syntax:

grep [OPTIONS] PATTERN [FILE...]

Common Options:

• -i: Case-insensitive search.
• -v: Invert match (show lines that do not match the pattern).
• -r: Recursively search files in directories.
• -l: Show only filenames that contain matches.
• -c: Show the count of matching lines.
• -n: Show line numbers with matching lines.
• --color: Highlight matching patterns in the output.
• -f option reads patterns from the given file
• -F skips the regex engine, it's faster than using normal grep

Common Examples:

# Case-Insensitive Search: Search for "error" in a file, ignoring case
grep -i "error" /var/log/syslog 

# Recursive Search in Directory: Search for the pattern "disk" in all files under /etc/, including subdirectories:
grep -r "disk" /etc/

# Display Line Numbers: Display lines containing "bash" in ~/.bashrc, including line numbers
grep -n "bash" ~/.bashrc

# Invert Match: Display all lines not containing "root" in /etc/passwd
grep -v "root" /etc/passwd

# Search Multiple Patterns with -E: Search for lines containing either "warn" or "error"
grep -E "warn|error" /var/log/syslog

egrep : Extended grep with More Power

egrep is short for Extended GREP. It is essentially the same as grep -E, meaning it uses extended regular expressions, which support more advanced regex features like +, ?, |, and parentheses without needing to escape them.

Note: In modern versions of grep, using grep -E is recommended as egrep is deprecated, but it still works.

#  Find lines containing "error" or "fail" without escaping the |
egrep "error|fail" /var/log/syslog 

# Equivalent to: 
grep -E "error|fail" /var/log/syslog

# Find lines with "color" or "colour"
egrep "colou?r" file.txt

# Advanced Regex – Match Patterns: Find lines where "log" appears, followed by 1 or more digits
egrep "log[0-9]+" logfiles.txt

Advanced Usages for grep Command:

1. Monitor System Logs in Real Time

You can use grep with the tail command to monitor logs dynamically and filter specific patterns:

tail -f /var/log/syslog | grep --color=auto -E "error|warn|critical"

• Explanation: This command monitors /var/log/syslog in real-time and filters for lines containing "error", "warn", or "critical". The --color option highlights the matches.

2. Find and Replace Patterns in Multiple Files

Use grep with sed to replace patterns in files. This is useful for bulk updates across multiple configuration files.

grep -rl "old-string" /path/to/directory | xargs sed -i 's/old-string/new-string/g'

• Explanation: This command searches recursively (-r) for all files containing "old-string" and passes them to sed to replace "old-string" with "new-string" (-i to edit the files in place).

3. Extract Specific Data from Large Logs

To extract specific types of information, like IP addresses or timestamps from a log file, advanced regular expressions can be used.

Example: Extract all IP addresses from a log file:

grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/apache2/access.log

• Explanation: This uses -oE to match only the IP addresses (a regex-matching IPv4 address format) and print them.

Example: Extract all timestamps from log files:

grep -oE "\[([0-9]{2}/[A-Za-z]+/[0-9]{4}):([0-9]{2}:[0-9]{2}:[0-9]{2})" /var/log/apache2/access.log

• Explanation: This extracts Apache log timestamps of the format [dd/Mon/yyyy:hh:mm:ss].

4. Audit File Permissions

If you’re performing security audits, you can use grep to find files with specific permissions.

Example: Find all files with 777 permissions in a directory and its subdirectories:

find / -type f -perm 777 2>/dev/null | grep --color=auto "^"

• Explanation: find searches for files with 777 permissions and pipes the output to grep to optionally highlight them.

5. Search Files Based on Metadata (Size, Time, Ownership)

grep can be used in combination with find to search files based on more specific metadata, like modification times, sizes, and owners.

Example: Find and display files larger than 100MB with the word “ERROR”:

find /var/log/ -type f -size +100M -exec grep -H "ERROR" {} \;

• Explanation: This uses find to locate files larger than 100MB in /var/log/, and -exec runs grep on those files to find lines containing "ERROR".

Example: Find files modified within the last 7 days and search for “failures”:

find /etc/ -type f -mtime -7 -exec grep -H "failure" {} \;

• Explanation: find looks for files modified within the last 7 days and passes them to grep to search for "failure".

6. Parallel Grep for Performance

For searching large directories with many files, running grep in parallel can significantly improve performance.

Example: Parallel search for the string “timeout” across a large directory:

find /path/to/dir -type f | xargs -P 4 grep "timeout"

• Explanation: xargs -P 4 runs grep in parallel with 4 processes, speeding up the search.

7. Searching for Multiple Complex Patterns

You can use grep to search for multiple complex patterns by using extended regular expressions.

Example: Search for lines containing “error” followed by a number, or any line with “timeout”:

grep -E "error[[:digit:]]+|timeout" /var/log/syslog

• Explanation: This finds lines where “error” is followed by one or more digits, or lines containing “timeout”.

Example: Search for lines containing either “failed login” or “successful login” in logs:

grep -E "(failed|successful) login" /var/log/auth.log

• Explanation: This uses the extended regular expression to match either “failed login” or “successful login”.

8. Search Binary Files and Ignore Non-Printable Characters

grep can be used to search for strings within binary files while ignoring non-printable characters.

Example: Search for the string “password” in all binary files:

grep -a "password" /usr/bin/*

• Explanation: The -a option treats binary files as text files, allowing you to search for human-readable strings inside binary files.

9. Filter and Count Unique Occurrences

You can use grep with sort and uniq to filter and count unique patterns, such as IP addresses, usernames, or errors.

Example: Count unique IP addresses accessing a web server:

grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/apache2/access.log | sort | uniq -c | sort -nr

• Explanation: This extracts IP addresses from the access log, sorts them, counts unique occurrences, and then sorts them by frequency.

Example: Find the most frequent error types in a log file:

grep "ERROR" /var/log/syslog | awk '{print $5}' | sort | uniq -c | sort -nr

• Explanation: This extracts error types (assuming the 5th field is the error type), counts unique errors, and sorts them by frequency.

10. Combine Grep with Other Tools for Automation

Example: Send an Alert if a Specific Error Appears in the Logs

You can create an automated alerting system based on specific patterns in your logs using grep.

grep -i "critical error" /var/log/syslog | mail -s "Critical Error Found" [email protected]

• Explanation: This searches for “critical error” in the syslog and, if found, sends an email with the matching lines to an administrator.

11. Extract Specific Data from a Range of Lines

Using grep in combination with sed or awk, you can extract specific data from a range of lines.

Example: Extract lines between two patterns:

sed -n '/START_PATTERN/,/END_PATTERN/p' file.txt | grep "important_string"

• Explanation: This extracts the lines between START_PATTERN and END_PATTERN and passes them to grep to search for important_string.

12. Grep from Remote Servers using SSH

You can remotely search logs or files on a different server using SSH and grep.

Example: Search for “error” in remote logs:

ssh user@remote_server 'grep "error" /var/log/syslog'

• Explanation: This executes the grep command on a remote server over SSH.

13. Find Executed Commands in Bash History

To track down a specific command used in the past by users, you can search through the bash history files.

Example: Search for all mount commands executed in the history of all users:

grep "mount" /home/*/.bash_history

• Explanation: This checks all users’ bash history files for any mount commands they executed.

14. Advanced Pattern Matching with Negative Lookahead

grep does not directly support lookahead, but you can achieve similar behavior with creative regex.

Example: Match lines that contain “error” but not “network error”:

grep -E "error" file.txt | grep -v "network error"

• Explanation: First, this finds all lines containing “error” and then excludes those with “network error”.

These advanced use cases demonstrate the versatility of grep, egrep, and fgrep when applied to practical and complex scenarios. Whether you're parsing logs, searching through massive datasets, or automating administrative tasks, these powerful tools help you efficiently find exactly what you need.

✅✅feel free to connect with us.

LinkedIn: https://www.linkedin.com/in/karthick-dkk/

Follow my Medium Account (To get valuable information)

For more updates: subscribe to this medium account.

Follow for more: ✌️

LinkedIn: https://www.linkedin.com/in/karthick-dkk/