AWK practical tips for parsing access logs

#1 Find top 10 IPs from an access log

Now you can pipe the output to a pipe and can find top 10 IPs using sort and head -10

#2 Suppose you have an access log which have timestamp in epoch seconds like following and you want to print hourly QPS count

Now you know how to get an hourly data and you know how to pass bash variable inside awk. So using a for loop, right you can get data for all hours separately

Notes:

  • I tested this in awk version 3.1.7. You can find your awk version using awk –version
  • Assuming log is getting rotated every day

Click me for the official page of AWK

Difference between $* and $@ in bash?

$* and $@, both these bash special variables expands to the positional parameters, starting from the first one.

These variables are same (expand positional parameters in same way) when using without double quotes. If these variables are using inside double quotes, it will expand positional parameters differently.

$* within double quotes ("$*") is equivalent to the list of positional parameters, separated by IFS variable.

Suppose IFS is ":" and hence expansion of "$*" will be like "$1:$2:$3:…"

And $@ within a pair of double quotes ("$@") is equivalent to the list of positional parameters separated by unquoted spaces, i.e., "$1" "$2".."$N". Or in other words, it is equivalent to the list of positional parameters where each parameters are double quoted.

For sake better understanding I wrote a script named star_and_at.sh and pushed to my public github repo

You can clone my bash github public repository directly using following command

“test” operators in Bash

Bash Conditional Expressions

Here I am trying to list almost all available bash “tests” (file tests, string tests, arithmetic tests)

file operators description
-e <FILENAME> True, if exists
-f <FILENAME> True, if exists and is a regular file
-d <FILENAME> True, if exists and is a directory
-c <FILENAME> True, if exists and is a character special file
-b <FILENAME> True, if <FILENAME> exists and is a block special file
-p <FILENAME> True, if <FILENAME> exists and is a named pipe (FIFO)
-S <FILENAME> True, if <FILENAME> exists and is a socket file
-L <FILENAME> True, if <FILENAME> exists and is a symbolic link
-h <FILENAME> True, if <FILENAME> exists and is a symbolic link
-g <FILENAME> True, if <FILENAME> exists and has sgid bit set
-u <FILENAME> True, if <FILENAME> exists and has suid bit set
-r <FILENAME> True, if <FILENAME> exists and is readable
-w <FILENAME> True, if <FILENAME> exists and is writable
-x <FILENAME> True, if <FILENAME> exists and is executable
-s <FILENAME> True, if <FILENAME> exists and has size bigger than 0
-t <fd> True, if file descriptor <fd> is open and refers to a terminal
<FILENAME1> -nt <FILENAME2> True, if <FILENAME1> is newer than <FILENAME2> (mtime)
<FILENAME1> -ot <FILENAME2> True, if <FILENAME1> is older than <FILENAME2> (mtime)
<FILENAME1> -ef <FILENAME2> True, if <FILENAME1> is a hardlink to <FILENAME2>

 

string operators description
-z <STRING> True, if <STRING> is empty
-n <STRING> True, if <STRING> is not empty (this is the default operation)
<STRING1> = <STRING2> True, if the strings are equal
<STRING1> != <STRING2> True, if the strings are not equal
<STRING1> < <STRING2> True, if <STRING1> sorts before <STRING2>
<STRING1> > <STRING2> True, if <STRING1> sorts after <STRING2>

 

arithmetic operators description
<INTEGER1> -eq <INTEGER2> True, if the integers are equal
<INTEGER1> -ne <INTEGER2> True, if the integers are NOT equal
<INTEGER1> -le <INTEGER2> True, if the first integer is less than or equal second one
<INTEGER1> -ge <INTEGER2> True, if the first integer is greater than or equal second one
<INTEGER1> -lt <INTEGER2> True, if the first integer is less than second one
<INTEGER1> -gt <INTEGER2> True, if the first integer is greater than second one

Match a pattern and delete n lines after it

Suppose you have a file named file.txt with following content. Delete all 2 lines [including pattern itself]after the pattern match “Class”.

I got this trick from SERVERFAULT.COM 🙂

Situation:

Suppose you have a file named file.txt with following content
Continue reading “Match a pattern and delete n lines after it”

bash: /bin/rm: Argument list too long

bash: /bin/rm: Argument list too long. If you’re trying to delete files inside a directory and the following command is not working:

If you’re trying to delete files inside a directory and the following command is not working:

In this case you can delete all files using find with appropriate switches:

If you want to delete files in verbose mode:

mysql queries on command line

In this article I like to share some tips which I am using while dealing with mysql databases.

‘-e’ is the switch which enables command line mode for ‘mysql’ client which is installed default on every Linux distros.

Some examples:

1. For listing all mysql users on the server.

2. Show values assisned to each mysql variable

Continue reading “mysql queries on command line”

System commands inside awk

In one of my past article I explained how to pass a shell variable to an awk program. In this article I am explaining how to execute system commands inside awk.
I am explaining the same via an example. Suppose we have a cPanel server in which you want to list all email accounts mentioned in .contactemail file in each user’s home directory.

Deleting folders with false ownership.

One day I have faced a situation regarding space in backup drive. On verification I found that there are many backups with false ownership and those backups with false ownerships are of users who are deleted from the system in past. So I decided to write a one liner by which can delete/remove all those backups.

One liner:

1. The following one liner will list all those backups in /backup/serverbackup/home/ with false ownership.

Continue reading “Deleting folders with false ownership.”

Use “find” to find files which are modified within specific time interval

“find” is a very useful command and with it, through various switches and combinations we can make many very powerful commands.

In this post I like to share you a one liner with find command. This will help you to find all files which are modified before a particular day. Like wise with slight modification in the same code we can find all files which are modified with in the current day and n’th day before. Also we can find files which are modified on a particular day.
Continue reading “Use “find” to find files which are modified within specific time interval”

Execute commands on a remote machine via SSH.

In this article I like to share a tip to execute commands on a remote machine via SSH. I used this tip in my many scripts mainly related to backups. Its really very helpful and simple to use. Thanks to SSH 🙂

In this article I like to share a tip to execute commands on a remote machine via SSH. I used this tip in my many scripts mainly related to backups. Its really very helpful and simple to use. Thanks to SSH 🙂

Syntax:

The above liner opens /etc/hosts file on the remote machine from your console.
Continue reading “Execute commands on a remote machine via SSH.”