Shell Scripting: Automating System Administration

Shell scripting is the system administrator’s most essential tool. A well-written bash script can automate repetitive tasks, enforce consistency, and save hours of manual work. This article covers the fundamentals of robust shell scripting — error handling, file operations, scheduling, and logging — with practical examples you can adapt immediately.

Writing Robust Scripts

Every production script should start with set -euo pipefail. This combination of options makes bash behave more predictably: -e exits immediately if any command fails (instead of continuing with errors), -u treats unset variables as errors (preventing typos from silently expanding to empty strings), o pipefail makes a pipeline fail if any command in it fails (not just the last one). Without these options, a script might silently continue after a critical failure, leading to corrupted data or inconsistent state.

#!/bin/bash
set -euo pipefail

# Configuration
BACKUP_DIR="/var/backups/$(date +%Y%m%d)"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=30

# Logging function
log() {
    local level="$1"
    local message="$2"
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $message" | tee -a "$LOG_FILE"
}

log "INFO" "Starting backup to $BACKUP_DIR"
mkdir -p "$BACKUP_DIR"

# Compress large log files
for file in /var/log/*.log; do
    if [[ -f "$file" && $(stat -c%s "$file") -gt 1048576 ]]; then
        log "INFO" "Compressing $file ($(stat -c%s "$file") bytes)"
        gzip "$file"
    fi
done

log "INFO" "Backup complete"

The log function demonstrates a common pattern: a centralized logging function that adds timestamps, severity levels, and writes to both stdout and a log file using tee -a. This gives you both real-time visibility during manual execution and a persistent log for later review or automated monitoring.

File and Directory Operations

Bash provides powerful file-test operators for checking file properties before operating on them. Always test conditions explicitly with [[ ]] rather than assuming a file exists or a command succeeded. The most useful tests are -f (regular file exists), -d (directory exists), -s (file exists and is non-empty), and -x (file is executable). Use stat for detailed metadata like file size, modification time, and permissions.

# File existence checks
if [[ ! -d "$BACKUP_DIR" ]]; then
    log "ERROR" "Backup directory does not exist"
    exit 1
fi

# Check if file is older than N days
find /tmp -name "*.tmp" -mtime +7 -delete

# Size-based filtering
for f in /data/*.csv; do
    size=$(stat -c%s "$f")
    if [[ $size -gt 100000000 ]]; then  # > 100 MB
        log "WARN" "$f is $size bytes — splitting recommended"
    fi
done

Scheduling with Cron

Cron is the standard job scheduler on Linux. A crontab entry has five time fields (minute, hour, day of month, month, day of week) followed by the command to execute. Always use absolute paths in cron jobs because cron runs with a minimal environment — PATH is often just /usr/bin:/bin. Redirect both stdout and stderr to a log file so that errors are captured. To avoid overlapping executions (if a job takes longer than its interval), use a lock file with flock.

# Crontab format: minute hour day month weekday command

# Run backup daily at 2 AM
0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

# Rotate logs every Sunday at midnight
0 0 * * 0 /usr/local/bin/rotate-logs.sh

# Health check every 5 minutes
*/5 * * * * /usr/local/bin/health-check.sh

# Avoid overlapping runs with flock
0 * * * * /usr/bin/flock -n /tmp/deploy.lock /usr/local/bin/deploy.sh

Error Handling and Notifications

When a scheduled task fails, someone needs to know. Use a notification function that sends alerts via email, Slack webhook, or a monitoring API. Trap the EXIT signal to run cleanup code regardless of how the script exits — whether successfully, by error, or by being killed.

# Trap for cleanup
cleanup() {
    local exit_code=$?
    if [[ $exit_code -ne 0 ]]; then
        log "ERROR" "Script failed with exit code $exit_code"
        curl -X POST -H 'Content-Type: application/json'             -d '{"text": "Backup script failed!"}'             https://hooks.slack.com/services/TOKEN
    fi
    rm -f /tmp/backup.lock
}
trap cleanup EXIT

# Acquire exclusive lock
exec 200>/tmp/backup.lock
flock -n 200 || { log "ERROR" "Another instance is running"; exit 1; }

Well-written shell scripts are the foundation of reliable system administration. By using strict error handling, structured logging, cron scheduling with locks, and notification on failure, you can trust your automation to run correctly — and to alert you immediately when it does not.

Idempotent Scripts and Error Handling

Production shell scripts should be idempotent—running them multiple times produces the same result as once. Use conditional checks before creating files, idempotent commands (mkdir -p, rm -f, cp -n), and cleanup traps that run even on failure. The set -euo pipefail strict mode catches errors. Logging with timestamps creates an audit trail. Provide a –yes or –force flag for CI/CD automation. Validate inputs and check that required tools are installed. These practices make scripts maintainable and safe for automated deployments.