Summary

The web content provides an in-depth guide to advanced Linux commands essential for DevOps engineers, focusing on rsync for file synchronization and tar for archiving files, with practical examples and use cases.

Abstract

The article "Advanced Linux Commands for DevOps Engineers: Part-1" is a comprehensive resource aimed at enhancing the command-line skills of DevOps professionals. It delves into the functionalities of the rsync command, illustrating its utility in efficiently transferring and synchronizing files across servers, with features such as bandwidth limitation, file exclusion, and preservation of file attributes. The article also explores various applications of the tar command, demonstrating how it can be used to create, extract, and manage compressed archives, including handling large files, incremental backups, and preserving file metadata. The guide emphasizes the importance of these commands in streamlining tasks related to system administration, backups, and deployments.

Opinions

The author emphasizes the importance of understanding advanced Linux commands for DevOps engineers to effectively manage complex tasks.
rsync is presented as a versatile tool for file synchronization, particularly suitable for backup and deployment operations.
The use of tar for archiving is highly recommended for efficient handling of project files and backups, with the command's ability to preserve file permissions and ownership being a key feature.
The article suggests that using the --delete option with rsync is beneficial for maintaining consistency between source and destination directories.
The author provides examples of how to limit bandwidth with rsync and split large archives with tar, showcasing the commands' flexibility in different scenarios.
The guide advocates for the use of tar in conjunction with split for managing large files, which can be a common challenge in DevOps environments.
Incremental backups using tar's --listed-incremental option are highlighted as a method to save time and storage space by only backing up changed files since the last full backup.
The article encourages readers to follow the author on LinkedIn and subscribe to the Medium account for more valuable information and updates in the field.

Advanced Linux Commands for DevOps Engineers: Part-1

For a DevOps engineer, understanding advanced Linux commands is crucial for handling complex system administration, automation, and troubleshooting tasks. Below are some advanced Linux commands along with practical use cases:

Let’s dive into it without wasting any seconds.!

1. rsync (Remote Sync)

Use case: Efficiently synchronize files between two servers, ideal for backups and deployments.

rsync [options] source destination

# synchronize a local directory to a remote server
rsync -avz /local/dir/ user@remote:/path/to/dir

# synchronize and delete files from the destination that are not present in the source:
rsync -avz --delete /local/directory/ user@remote_host:/remote/directory/

# preserve all file attributes, including extended attributes and ACLs
rsync -aAX /source/directory/ /destination/directory/

# SSH server runs on a non-standard port
rsync -avz -e "ssh -p 2222" /local/directory/ user@remote_host:/remote/directory/

# exclude/include files matching a specific pattern
rsync -avz --exclude '*.tmp' /local/directory/ user@remote_host:/remote/directory/

rsync -avz --include '*.jpg' --exclude '*' /local/directory/ user@remote_host:/remote/directory/

# To limit bandwidth during transfer (e.g., to 1000 KB/s
rsync -avz --bwlimit=1000 /local/directory/ user@remote_host:/remote/directory/

# create backups before overwriting existing files
rsync -avz --backup --backup-dir=/backup/directory /local/directory/ user@remote_host:/remote/directory/

-a: Archive mode, preserves file permissions, ownership, etc.
-v: Verbose, shows details of the operation.
-z: Compresses files during transfer to save bandwidth.
-h: Human-readable output; makes file sizes easier to read.
--include: Include files matching a pattern, useful with --exclude
--delete: Delete extraneous files from the destination that are not present in the source
--dry-run: Perform a trial run without making any changes.

2. tar (Archive Files)

Use case: Compress or extract large project files for backups or deployments.

tar [options] [archive-file] [file or directory to be archived]

# Create a tar.gz file
tar -czvf backup.tar.gz /path/to/files/

# Extract a tar.gz file
tar -xzvf backup.tar.gz

# Create a gzip compressed archive
tar -czvf archive.tar.gz /path/to/file1  /path/to/file2 /path/to/file3

# Create a bzip2 compressed archive
tar -cjvf archive.tar.bz2 /path/to/files

# Create an xz compressed archive
tar -cJvf archive.tar.xz /path/to/files

# Extract an archive to a specified directory instead of the current one
tar -xvf archive.tar.gz -C /path/to/destination

# View the list of files inside the archive.
tar -tvf archive.tar.gz

-c: Create an archive.
-C: archive destination location.
-z: Compress with gzip.
-v: Verbose, shows the progress.
-f: Output to the archive file archive.tar.gz
-j: Compress with bzip2.
-J: Compress with xz.
-t: List the contents of the archive.
-k: Do not overwrite existing files.
-d: Compares files in the archive with the filesystem.

Extracting an Archive

# Extracting a Specific File from an Archive
tar -xvf archive.tar.gz path/to/extract/file.txt

# Extract files but avoid overwriting existing files with the same name.
tar -xkzvf archive.tar.gz

# Compare the files in an archive with the current state of the files on disk.
tar -df archive.tar.gz

Appending to an Existing Archive & Splitting Large Archives

Add the files to an existing archive without recreating it again.

-r: Append files to the existing archive.

tar -rvf archive.tar /path/to/newfile

# Split a large archive into multiple smaller files.
# split -b 500M: Splits the archive into 500MB chunks.
tar -czvf - /path/to/largefiles | split -b 500M - largearchive.tar.gz.part

# To Multiple archive files to single archive
cat largearchive.tar.gz.part* | tar -xzvf -

Excluding Files or Directories

Exclude specific files or directories when creating an archive.

--exclude: Excludes the specified file or directory.

tar --exclude='/path/to/exclude' -czvf archive.tar.gz /path/to/files

# exclude by file type (multiple patterns)
tar --exclude='*.log' --exclude='*.tmp' -czvf archive.tar.gz /path/to/files

Archiving Files Based on a File List

Create an archive from a list of files specified in a file. Save the location on files like below.

cat filelist.txt /etc/nginx/conf.d/

/var/log/nginx/

/opt/config

tar -czvf archive.tar.gz -T filelist.txt

Backups Solution

Perform incremental backups using the --listed-incremental option.

This allows you to store information about files and their status, so future backups only include changes since the last archive.

Full Backup: Take original copy of all the files.

Incremental Backup: Take a copy of only modified files or changed files that are already taken by a full backup.

# Full backup
tar --listed-incremental=/path/to/snapshot.file -czvf full-backup.tar.gz /path/to/files

# Incremental backup
tar --listed-incremental=/path/to/snapshot.file -czvf incremental-backup.tar.gz /path/to/files

Preserving Permissions, Ownership, and Timestamps

When creating and extracting an archive, use options to preserve file permissions, ownership, and timestamps.

Well suitable for system configurations, and user file backup.

--preserve-permissions: Keep file permissions.
--preserve-order: Keep the order of files and directories.

tar --preserve-permissions --preserve-order -czvf archive.tar.gz /path/to/files

Advanced Linux Commands for DevOps Engineers: Part-1

1. rsync (Remote Sync)

2. tar (Archive Files)

Advanced Linux Commands for DevOps Engineers: Part-2

Welcome back, fellow terminal warriors! 🗡️. Advanced Linux Commands for DevOps Engineers: Part-2