20 Linux Log Files that are Located under /var/log Directory

If you spend lot of time in Linux environment, it is essential that you know where the log files are located, and what is contained in each and every log file.

When your systems are running smoothly, take some time to learn and understand the content of various log files, which will help you when there is a crisis and you have to look though the log files to identify the issue.

/etc/rsyslog.conf controls what goes inside some of the log files. For example, following is the entry in rsyslog.conf for /var/log/messages.

$ grep "/var/log/messages" /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

In the above output,

• *.info indicates that all logs with type INFO will be logged.
• mail.none,authpriv.none,cron.none indicates that those error messages should not be logged into the /var/log/messages file.
• You can also specify *.none, which indicates that none of the log messages will be logged.

The following are the 20 different log files that are located under /var/log/ directory. Some of these log files are distribution specific. For example, you’ll see dpkg.log on Debian based systems (for example, on Ubuntu).

1. /var/log/messages – Contains global system messages, including the messages that are logged during system startup. There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
2. /var/log/dmesg – Contains kernel ring buffer information. When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process. These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten. You can also view the content of this file using the dmesg command.
3. /var/log/auth.log – Contains system authorization information, including user logins and authentication machinsm that were used.
4. /var/log/boot.log – Contains information that are logged when the system boots
5. /var/log/daemon.log – Contains information logged by the various background daemons that runs on the system
6. /var/log/dpkg.log – Contains information that are logged when a package is installed or removed using dpkg command
7. /var/log/kern.log – Contains information logged by the kernel. Helpful for you to troubleshoot a custom-built kernel.
8. /var/log/lastlog – Displays the recent login information for all the users. This is not an ascii file. You should use lastlog command to view the content of this file.
9. /var/log/maillog /var/log/mail.log – Contains the log information from the mail server that is running on the system. For example, sendmail logs information about all the sent items to this file
10. /var/log/user.log – Contains information about all user level logs
11. /var/log/Xorg.x.log – Log messages from the X
12. /var/log/alternatives.log – Information by the update-alternatives are logged into this log file. On Ubuntu, update-alternatives maintains symbolic links determining default commands.
13. /var/log/btmp – This file contains information about failed login attemps. Use the last command to view the btmp file. For example, “last -f /var/log/btmp | more”
14. /var/log/cups – All printer and printing related log messages
15. /var/log/anaconda.log – When you install Linux, all installation related messages are stored in this log file
16. /var/log/yum.log – Contains information that are logged when a package is installed using yum
17. /var/log/cron – Whenever cron daemon (or anacron) starts a cron job, it logs the information about the cron job in this file
18. /var/log/secure – Contains information related to authentication and authorization privileges. For example, sshd logs all the messages here, including unsuccessful login.
19. /var/log/wtmp or /var/log/utmp – Contains login records. Using wtmp you can find out who is logged into the system. who command uses this file to display the information.
20. /var/log/faillog – Contains user failed login attemps. Use faillog command to display the content of this file.

Apart from the above log files, /var/log directory may also contain the following sub-directories depending on the application that is running on your system.

• /var/log/httpd/ (or) /var/log/apache2 – Contains the apache web server access_log and error_log
• /var/log/lighttpd/ – Contains light HTTPD access_log and error_log
• /var/log/conman/ – Log files for ConMan client. conman connects remote consoles that are managed by conmand daemon.
• /var/log/mail/ – This subdirectory contains additional logs from your mail server. For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file
• /var/log/prelink/ – prelink program modifies shared libraries and linked binaries to speed up the startup process. /var/log/prelink/prelink.log contains the information about the .so file that was modified by the prelink.
• /var/log/audit/ – Contains logs information stored by the Linux audit daemon (auditd).
• /var/log/setroubleshoot/ – SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files, and logs those information in this log file.
• /var/log/samba/ – Contains log information stored by samba, which is used to connect Windows to Linux.
• /var/log/sa/ – Contains the daily sar files that are collected by the sysstat package.
• /var/log/sssd/ – Use by system security services daemon that manage access to remote directories and authentication mechanisms.

Instead of manually trying to archive the log files, by cleaning it up after x number of days, or by deleting the logs after it reaches certain size, you can do this automatically using logrotateas we discussed earlier.

To view the log files use any one of the following methods. But, please don’t do ‘cat | more’.

• vi – If you are comfortable with the vi commands, use vi editor for quick log file browsing.
• tail – If you want to view the content of the log files real time, as the application is writting to it, use “tail -f”. You can also view multiple log files at the same time (using “tail -f”).
• grep – If you know exactly what you are looking for in a log file, you can quickly use grep command to grep a pattern. The 15 practical grep examples will take out all your excuses of not using grep.
• less – Less command is very powerful to browse log files. Use these 10 less command tipsto master the less command.

How to Run Cron Every 5 Minutes, Seconds, Hours, Days, Months

Crontab can be used to schedule a job that runs on certain internal. The example here show how to execute a backup.sh shell script using different intervals.

Also, don’t forget to read our previous crontab article that contains 15 practical examples, and also explains about @monthly, @daily, .. tags that you can use in your crontab.

1. Execute a cron job every 5 Minutes

The first field is for Minutes. If you specify * in this field, it runs every minutes. If you specify */5 in the 1st field, it runs every 5 minutes as shown below.

*/5 * * * * /home/suresh/backup.sh

Note: In the same way, use */10 for every 10 minutes, */15 for every 15 minutes, */30 for every 30 minutes, etc.

2. Execute a cron job every 5 Hours

The second field is for hours. If you specify * in this field, it runs every hour. If you specify */5 in the 2nd field, it runs every 5 hours as shown below.

0 */5 * * * /home/suresh/backup.sh

Note: In the same way, use */2 for every 2 hours, */3 for every 3 hours, */4 for every 4 hours, etc.

3. Execute a job every 5 Seconds

Cron job cannot be used to schedule a job in seconds interval. i.e You cannot schedule a cron job to run every 5 seconds. The alternative is to write a shell script that uses ‘sleep 5′ command in it.

Create a shell script every-5-seconds.sh using bash while loop as shown below.

$ cat every-5-seconds.sh
#!/bin/bash
while true
do
 /home/suresh/backup.sh
 sleep 5
done

Now, execute this shell script in the background using nohup as shown below. This will keep executing the script even after you logout from your session. This will execute your backup.sh shell script every 5 seconds.

$ nohup ./every-5-seconds.sh &

4. Execute a job every 5th weekday

This example is not about scheduling “every 5 days”. But this is for scheduling “every 5th weekday”.

The 5th field is DOW (day of the week). If you specify * in this field, it runs every day. To run every Friday, specify either 5 of Fri in this field.

The following example runs the backup.sh every Friday at midnight.

0 0 * * 5 /home/suresh/backup.sh
(or)
0 0 * * Fri /home/suresh/backup.sh

You can either user number or the corresponding three letter acronym for the weekday as shown below.

• 0=Sun
• 1=Mon
• 2=Tue
• 3=Wed
• 4=Thu
• 5=Fri
• 6=Sat

Note: Get into the habit of using Fri instead of 5. Please note that the number starts with 0 (not with 1), and 0 is for Sun (not Mon).

5. Execute a job every 5 months

There is no direct way of saying ‘every 5 months’, instead you have to specify what specific months you want to run the job. Probably you may want to run the job on 5th month (May), and 10th month (Oct).

The fourth field is for Months. If you specify * in this field, it runs every month. To run for the specific month, you have to specify the number that corresponds to the month. For example, to run the job on May and Oct, you should specify 5,10 (or) you can simply use the 3 letter acronym of the month and specify May,Oct.

The third field is for DOM (Day of the Month). If you specify * in this field, it runs every day of the month. If you specify 1 in this month, it runs 1st of the month.

The following example runs the backup.sh twice a year. i.e 1st May at midnight, and 1st Oct at midnight.

0 0 1 5,10 * /home/suresh/backup.sh
(or)
0 0 1 May,Oct * /home/suresh/backup.sh

Note: Don’t make the mistake of specifying 5-10 in the 4th field, which means from 5th month until 10th month. If you want only 5th and 10th month, you should use comma.

HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples

Managing log files effectively is an essential task for Linux sysadmin.

In this article, let us discuss how to perform following log file operations using UNIXlogrotate utility.

• Rotate the log file when file size reaches a specific size
• Continue to write the log information to the newly created file after rotating the old log file
• Compress the rotated log files
• Specify compression option for the rotated log files
• Rotate the old log files with the date in the filename
• Execute custom shell scripts immediately after log rotation
• Remove older rotated log files

1. Logrotate Configuration files

Following are the key files that you should be aware of for logrotate to work properly.

/usr/sbin/logrotate – The logrotate command itself.

/etc/cron.daily/logrotate – This shell script executes the logrotate command everyday.

$ cat /etc/cron.daily/logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

/etc/logrotate.conf – Log rotation configuration for all the log files are specified in this file.

$ cat /etc/logrotate.conf
weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}

/etc/logrotate.d – When individual packages are installed on the system, they drop the log rotation configuration information in this directory. For example, yum log rotate configuration information is shown below.

$ cat /etc/logrotate.d/yum
/var/log/yum.log {
    missingok
    notifempty
    size 30k
    yearly
    create 0600 root root
}

2. Logrotate size option: Rotate the log file when file size reaches a specific limit

If you want to rotate a log file (for example, /tmp/output.log) for every 1KB, create the logrotate.conf as shown below.

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        create 700 bala bala
        rotate 4
}

This logrotate configuration has following three options:

• size 1k – logrotate runs only if the filesize is equal to (or greater than) this size.
• create – rotate the original file and create the new file with specified permission, user and group.
• rotate – limits the number of log file rotation. So, this would keep only the recent 4 rotated log files.

Before the logrotation, following is the size of the output.log:

$ ls -l /tmp/output.log
-rw-r--r-- 1 bala bala 25868 2010-06-09 21:19 /tmp/output.log

Now, run the logrotate command as shown below. Option -s specifies the filename to write the logrotate status.

$ logrotate -s /var/log/logstatus logrotate.conf

Note : whenever you need of log rotation for some files, prepare the logrotate configuration and run the logroate command manually.
After the logrotation, following is the size of the output.log:

$ ls -l /tmp/output*
-rw-r--r--  1 bala bala 25868 2010-06-09 21:20 output.log.1
-rwx------ 1 bala bala        0 2010-06-09 21:20 output.log

Eventually this will keep following setup of rotated log files.

• output.log.4.
• output.log.3
• output.log.2
• output.log.1
• output.log

Please remember that after the log rotation, the log file corresponds to the service would still point to rotated file (output.log.1) and keeps on writing in it. You can use the above method, if you want to rotate the apache access_log or error_log every 5 MB.

Ideally, you should modify the /etc/logrotate.conf to specify the logrotate information for a specific log file.

3. Logrotate copytruncate option: Continue to write the log information in the newly created file after rotating the old log file.

$ cat logrotate.conf
/tmp/output.log {
         size 1k
         copytruncate
         rotate 4
}

copytruncate instruct logrotate to creates the copy of the original file (i.e rotate the original log file) and truncates the original file to zero byte size. This helps the respective service that belongs to that log file can write to the proper file.

4. Logrotate compress option: Compress the rotated log files

If you use the compress option as shown below, the rotated files will be compressed with gzip utility.

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create 700 bala bala
        rotate 4
        compress
}

Output of compressed log file:

$ ls /tmp/output*
output.log.1.gz output.log

5. Logrotate dateext option: Rotate the old log file with date in the log filename

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create 700 bala bala
        dateext
        rotate 4
        compress
}

After the above configuration, you’ll notice the date in the rotated log file as shown below.

$ ls -lrt /tmp/output*
-rw-r--r--  1 bala bala 8980 2010-06-09 22:10 output.log-20100609.gz
-rwxrwxrwx 1 bala bala     0 2010-06-09 22:11 output.log

This would work only once in a day. Because when it tries to rotate next time on the same day, earlier rotated file will be having the same filename. So, the logrotate wont be successful after the first run on the same day.

6. Logrotate monthly, daily, weekly option: Rotate the log file weekly/daily/monthly

For doing the rotation monthly once,

$ cat logrotate.conf
/tmp/output.log {
        monthly
        copytruncate
        rotate 4
        compress
}

Add the weekly keyword as shown below for weekly log rotation.

$ cat logrotate.conf
/tmp/output.log {
        weekly
        copytruncate
        rotate 4
        compress
}

Add the daily keyword as shown below for every day log rotation. You can also rotate logs hourly.

$ cat logrotate.conf
/tmp/output.log {
        daily
        copytruncate
        rotate 4
        compress
}

7. Logrotate postrotate endscript option: Run custom shell scripts immediately after log rotation

Logrotate allows you to run your own custom shell scripts after it completes the log file rotation. The following configuration indicates that it will execute myscript.sh after the logrotation.

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        postrotate
               /home/bala/myscript.sh
        endscript
}

8. Logrotate maxage option: Remove older rotated log files

Logrotate automatically removes the rotated files after a specific number of days.  The following example indicates that the rotated log files would be removed after 100 days.

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        maxage 100
}

9. Logrotate missingok option: Dont return error if the log file is missing

You can ignore the error message when the actual file is not available by using this option as shown below.

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        missingok
}

10. Logrotate compresscmd and compressext option: Sspecify compression command for the log file rotation

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create
        compress
        compresscmd /bin/bzip2
        compressext .bz2
        rotate 4
}

Following compression options are specified above:

• compress – Indicates that compression should be done.
• compresscmd – Specify what type of compression command should be used. For example: /bin/bzip2
• compressext – Specify the extension on the rotated log file. Without this option, the rotated file would have the default extension as .gz. So, if you use bzip2 compressioncmd, specify the extension as .bz2 as shown in the above example.

How to Rotate Apache Log Files in Linux

Add the following file to /etc/logrotate.d directory.

# vi /etc/logrotate.d/apache
/usr/local/apache2/logs/access_log /usr/local/apache2/logs/error_log {
    size 100M
    compress
    dateext
    maxage 30
    postrotate
      /usr/bin/killall -HUP httpd
      ls -ltr /usr/local/apache2/logs | mail -s "$HOSTNAME: Apache restarted and log files rotated" suresh@sureshkumarpakalapati.in
    endscript
}

Note: Refer to our logrotate tutorial (with 15 examples) that explains more details about how to use logrotate options.

In the above /etc/logrotate.d/apache example:

• size 100M – Once the access_log, and error_log reaches 100M, it will be rotated. You can also use 100k (for Kb), 100G (for GB). Instead of size, you can also rotate apache logs using frequency (daily, weekly, monthly).
• compress – Indicates that the rotated log file will be compressed. By default this uses gzip. So, the rotated file will have .gz extension.
• dateext - Appends the date in YYYYMMDD format to the rotated log files. i.e Instead of access_log.1.gz, it creates access_log-20110616.gz
• maxage - Indicates how long the rotated log files should be kept. In this example, it will be kept for 30 days.
• postrotate and endscript – Any commands enclosed between these two parameter will be executed after the log is rotated.

Important: Once you rotate the log files, you want apache to write the new log messages to the newly created access_log and error_log. So, you need to send the HUP signal to the apache as shown here. Make sure to do /usr/bin/killall -HUP httpd, which will restart the apache after rotating the log files (Read more about kill).

Also, you might want to send an email to yourself indicating that the log file is rotated, along with the output of ls -ltr command as the body of the email. i.e Add the following between “postrotate” and “endscript” option (after the killall command).

ls -ltr /usr/local/apache2/logs | mail -s "$HOSTNAME: Apache restarted and log files rotated" suresh@sureshkumarpakalapati.in

The /etc/cron.daily/logrotate script runs everyday that will perform log rotate of all the files as specified in the /etc/logrotate.conf and all the file under /etc/logrotate.d directory.

After adding the above /etc/logrotate.d/apache file, for testing purpose, you can manually call the logrotate script as shown below.

# /etc/cron.daily/logrotate

Once the log files are rotated, do a ls to verify them. As we explained above, the rotated log files will be kept for 30 days.

# ls /usr/local/apache2/logs
access_log
error_log
access_log-20110716.gz
error_log-20110716.gz

7 Linux chkconfig Command Examples – Add, Remove, View, Change Services

Chkconfig command is used to setup, view, or change services that are configured to start automatically during the system startup.

This article contains 7 practical examples that explains how to use the chkconfig command.

1. Check Service Startup status from Shell Script

When you execute chkconfig command only with the service name, it returns true if the service is configured for startup. The following code snippet shows how to check whether a service is configured for startup or not from a shell script.

# vi check.sh
chkconfig network && echo "Network service is configured"
chkconfig junk && echo "Junk service is configured"

# ./check.sh
Network service is configured

You can also specifically check whether it is configured for a particular run level or not.

# vi check1.sh
chkconfig network --level 3 && echo "Network service is configured for level 3"
chkconfig network --level 1 && echo "Network service is configured for level 1"

# ./check1.sh
Network service is configured for level 3

2. View Current Status of Startup Services

The –list option displays all the services with the current startup configuration status.

# chkconfig --list
abrtd   0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid   0:off   1:off   2:off   3:off   4:off   5:off   6:off
atd     0:off   1:off   2:off   3:on    4:on    5:on    6:off
...

To view only the services that are configured to be started during system startup, do the following. Please note that this assumes that your system startup level is 3.

chkconfig --list | grep 3:on

To view the startup configuration of a particular service, grep the output of ‘chkconfig –list’ for that service.

chkconfig --list | grep network

3. Add a new Service to the Startup

Use –add option to add a specific service to the list of services that will be started during system reboot.

The following example shows how to add a new service (for example, iptables) to the list of services that needs to be started. The ‘chkconfig –add’ command will also turn on level 2, 3, 4 and 5 automatically as shown below.

# chkconfig --list | grep iptables

# chkconfig --add iptables

# chkconfig --list | grep iptables
iptables       0:off   1:off   2:on    3:on    4:on    5:on    6:off

Note: “chkconfig –add” only adds an existing service to the list of startup. If the service doesn’t exist, you should first install it before adding it to the system startup list. While this is pretty obvious, it is worth to mention it, as a newbie might make this mistake.

4. Remove a Service From Startup List

The following example shows that ip6tables services is configured for startup.

# chkconfig --list | grep ip6tables
ip6tables       0:off   1:off   2:off   3:on   4:off   5:off   6:off

To remove it from the startup list, use the –del option as shown below.

# chkconfig --del ip6tables

# chkconfig --list | grep ip6tables

5. Turn-on or Turn-off a Service for a Selected Run Level

Sometimes you might not want to delete the whole service. Instead, you might just want to turn the flag on or off for a particular run level (for a particular service).

The following example will turn off nfserver service for level 5

# chkconfig --level 5 nfsserver off

You can also combine multiple levels. The following example will turn off nfsserver for both level 3 and 5.

# chkconfig --level 35 nfsserver off

6. Script Files under rc.d Subdirectories

Whenever you add or remove a service from chkconfig control, it does the following under the /etc/rc.d sub-directories.

• When chkconfig –add command is executed, it creates a symbolic link file to start and stop the service under corresponding rc directory.
• When chkconfig –del command is executed, it removes the symbolic link file from the corresponding rc directory.

The following example shows that xinetd is enabled for both run level 3 and 5.

So, xinetd will have two files under rc3.d directory, and two files under rc5.d directory. The file that starts with K is used during shutdown (K stands for kill). The file that starts with S is used during startup (S stands for start).

# chkconfig --list | grep xinetd
xinetd                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
xinetd based services:

# cd /etc/rc.d/rc3.d
# ls | grep xinetd
K08xinetd
S14xinetd

# cd /etc/rc.d/rc5.d

# ls | grep xinetd
K08xinetd
S14xinetd

7. rcx.d Directory Changes for Add Operation

When you add a new service to chkconfig control, the default run levels for that service will be turned on automatically, and files will be created under the corresponding rcx directories.

For example, if the nfsserver service doesn’t exist in the chkconfig control, no nfsserver service startup files would be present under /etc/rc.d/rc*.d directories as shown below.

# chkconfig  --list | grep nfsserver
nfsserver                 0:off  1:off  2:off  3:off  4:off  5:off  6:off

# ls /etc/rc.d/rc3.d | grep nfsserver

# ls /etc/rc.d/rc5.d | grep nfsserver

After you add the nfsserver service, you’ll see the symbolic links under these directories.

# chkconfig --add nfsserver
nfsserver                 0:off  1:off  2:off  3:on   4:off  5:on   6:off

# cd  /etc/rc.d/rc3.d
# ls -l | grep nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 K08nfsserver -> ../nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 S14nfsserver -> ../nfsserver

# cd /etc/rc.d/rc5.d
# ls -l | grep nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 K08nfsserver -> ../nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 S14nfsserver -> ../nfsserver

When you turn off the service either using –del option or –level option, the corresponding symbolic link file under rcx.d directory will be deleted as shown below.

# chkconfig --level 5 nfsserver off

# ls /etc/rc.d/rc5.d  | grep nfsserver