AWK practical tips for parsing access logs

#1 Find top 10 IPs from an access log

Now you can pipe the output to a pipe and can find top 10 IPs using sort and head -10

#2 Suppose you have an access log which have timestamp in epoch seconds like following and you want to print hourly QPS count

Now you know how to get an hourly data and you know how to pass bash variable inside awk. So using a for loop, right you can get data for all hours separately

Notes:

  • I tested this in awk version 3.1.7. You can find your awk version using awk –version
  • Assuming log is getting rotated every day

Click me for the official page of AWK

System commands inside awk

In one of my past article I explained how to pass a shell variable to an awk program. In this article I am explaining how to execute system commands inside awk.
I am explaining the same via an example. Suppose we have a cPanel server in which you want to list all email accounts mentioned in .contactemail file in each user’s home directory.

Passing shell variables to awk program

For an administrator awk is very important and it will help him in many ways. One day, I get into trouble for passing shell variables inside awk. After some GOOGLING I got the solutions.

You can follow any of the following 2 methods:

Method : 1

Limitations of this method:

a. Shell variables assigned using this method are not available in the BEGIN section

b. If variables are assigned after a filename, they will not be available when processing that filename

Method : 2