When starting a background service using `nohup command &`, all output is redirected to the `nohup.out` file in the current directory by default. If the service runs for weeks or months, this log file can grow to several gigabytes or even larger, not only consuming valuable disk space but also making it difficult to find specific logs. Worse still, if the disk becomes full, it can cause the service to crash. Manually handling these logs is inefficient and unreliable, so we need an automated solution to periodically split and clean up the logs.

First, it's important to understand that directly deleting or moving a log file that a process is writing to will cause problems. If you simply execute `rm nohup.out` or `mv nohup.out nohup.out.old`, the process still holds the file descriptor of the original file and will continue writing to the moved file (`nohup.out.old`), while a new `nohup.out` file will be created but the process will not write to it. The correct approach is to ensure that log writing seamlessly switches to the new file.

One method is to use a shell script to implement splitting and cleanup. We can write a shell script that periodically performs the following operations: copies the current log content to a timestamped backup file, and then clears the original log file. Here is a robust implementation:

#!/bin/bash

# Filename: rotate_nohup_log.sh

# Function: Split the nohup.out log and clean up old files

# Set the log directory and filename (modify according to your actual situation)

LOG_DIR="/home/your_app"

LOG_FILE="$LOG_DIR/nohup.out"

BACKUP_DIR="$LOG_DIR/log_backups"

# Exit if the log file does not exist or is empty

if [ ! -f "$LOG_FILE" ] || [ ! -s "$LOG_FILE" ]; then

exit 0

fi

# Create the backup directory

mkdir -p "$BACKUP_DIR"

# Generate backup filenames with date and timestamps

BACKUP_NAME="nohup_$(date +%Y%m%d_%H%M%S).log"

BACKUP_PATH="$BACKUP_DIR/$BACKUP_NAME"

# Core step: Copy the current log content to the backup file

cp "$LOG_FILE" "$BACKUP_PATH"

# Clear the original log file (Note: Use truncate instead of delete)

truncate -s 0 "$LOG_FILE"

# Optional: Compress the backup file to save space

gzip -f "$BACKUP_PATH"

echo "$(date '+%Y-%m-%d %H:%M:%S') - split log to ${BACKUP_PATH}.gz" >> "$LOG_DIR/rotate.log"

# Clean up old log backups older than 30 days

find "$BACKUP_DIR" -name "nohup_*.log.gz" -mtime +30 -delete

The key point of this script is that it uses the `cp` command to copy log content instead of moving the file; it uses the `truncate -s 0` command to truncate the original file size to 0 bytes instead of deleting the file; the process will continue to write to the same file descriptor, but the file is now empty; it automatically compresses backup files and cleans up backups older than 30 days.

Method two is to use the built-in Linux logrotate tool. For more professional and standardized management, the built-in Linux `logrotate` tool is a better choice. It can handle complex needs such as log rotation, compression, and email notifications.

First, create a `logrotate` configuration file for the nohup logs, for example, `/etc/logrotate.d/myapp-nohup`:

# /etc/logrotate.d/myapp-nohup

/home/your_app/nohup.out {

daily # Rotate once a day

missingok # No error if the file does not exist

notifempty # Do not rotate if the file is empty

compress # Compress old logs after rotation

delaycompress # Delay compression once for easier viewing of the latest logs

copytruncate # Key parameter: copy first, then truncate, no need to restart the process

create 0640 user group # Permissions and owner of new files created after rotation

dateext # Use date as the suffix for rotated files

dateformat -%Y%m%d # Date format

maxage 30 # Delete backups older than 30 days

rotate 30 # Keep 30 backup files

# If you need to execute specific commands (such as restarting the service) after the rotation, you can add the following here:

# postrotate

# /bin/kill -HUP $(cat /var/run/your_app.pid 2>/dev/null) 2>/dev/null || true

# endscript

}

The `copytruncate` parameter is key here; it implements the same logic as our manual script: first copy the file content, then clear the original file. This method does not require the process to support reopening the log file.

Test if the configuration is correct:

# Run in debug mode to see the effect but not actually execute

sudo logrotate -d /etc/logrotate.d/myapp-nohup

# Force a rotation

sudo logrotate -vf /etc/logrotate.d/myapp-nohup

Regardless of the method used, a scheduled task needs to be set up for automatic execution.

For custom script methods, use crontab to schedule execution every day at 2 AM:

# Edit the current user's crontab

crontab -e

# Add the following line to execute the log splitting script at 2 AM daily

0 2 * * * /bin/bash /path/to/rotate_nohup_log.sh

# After saving and exiting, cron will automatically load the new configuration.

For the logrotate method, it is usually already configured to run automatically daily via `/etc/cron.daily/logrotate`. You can check this configuration and adjust the cron settings if you need a different execution frequency.

If your server is running multiple applications using nohup, you can configure it separately for each application:

# Process multiple log files in a script

declare -a LOG_FILES=("/app1/nohup.out" "/app2/nohup.out")

for LOG_FILE in "${LOG_FILES[@]}"; do

if [ -f "$LOG_FILE" ] && [ -s "$LOG_FILE" ]; then

# Process each log file...

fi

done

Add a disk space check to the script to clean up in advance when space is insufficient:

# Check disk usage and delete older backups when it exceeds 85%

DISK_USAGE=$(df -h "$BACKUP_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$DISK_USAGE" -gt 85 ]; then

# Delete backups older than 15 days

find "$BACKUP_DIR" -name "nohup_*.log.gz" -mtime +15 -delete

echo "$(date) - disk usage ${DISK_USAGE}%, backups older than 15 days have been cleaned" >> /var/log/cleanup.log

fi

If your service restarts periodically, creating a new nohup.out file each time, you might need to differentiate logs by service instance or startup time:

# Use timestamped log filenames when starting the service

START_TIME=$(date +%Y%m%d_%H%M%S)

nohup your_command > nohup_${START_TIME}.out 2>&1 &

# Then adjust the script to process all nohup_*.out files

Ensure the script and log directory have correct permissions. If the service is running as a specific user, the ownership and permissions of the log files need to be set accordingly:

# Set the correct directory permissions

chmod 755 /home/your_app

chmod 644 /home/your_app/nohup.out

# If the service is running as a specific user

chown app_user:app_group /home/your_app/nohup.out

By implementing these automated policies, you can ensure that the nohup logs are effectively managed, freeing up disk space while retaining necessary troubleshooting information, and ensuring the continuous and stable operation of the service.