Convert UTC datetime object to epoch seconds

A timedelta object represents a duration, the difference between two dates or times.

Convert UTC time to PST time in python

I wrote a small function which converts a datetime object from a timezone to a new timezone.
file name : tz2ntz.py

In action:

You can see all available time zones using pytz.all_timezones

How to convert an epoch second to a python datetime object?

The Epoch second (POSIX time/Unix time) is the number of seconds that have elapsed since January 1, 1970. Epoch seconds are not counting leap seconds.

So how we can convert an epoch second to human readable time?

For that, first we need to create a datetime object for the day 1970/Jan/01

then put the epoch second which you would like to convert to a human readble format in a varaible

then create a timedelta object for the epoch_second

then add dobj1 to dobj2

or in short

AWK practical tips for parsing access logs

#1 Find top 10 IPs from an access log

Now you can pipe the output to a pipe and can find top 10 IPs using sort and head -10

#2 Suppose you have an access log which have timestamp in epoch seconds like following and you want to print hourly QPS count

Now you know how to get an hourly data and you know how to pass bash variable inside awk. So using a for loop, right you can get data for all hours separately

Notes:

  • I tested this in awk version 3.1.7. You can find your awk version using awk –version
  • Assuming log is getting rotated every day

Click me for the official page of AWK

How to make google drive as your automated backup location using python

Before you going to coming paragraphs of this article, I recommend you to watch following youtube video so you will get what is going to describe here:

[http://www.youtube.com/watch?v=zJVCKvXtHtE]
Read following links to learn more about the authentication mechanism google API is using :
https://developers.google.com/accounts/docs/OAuth2
https://developers.google.com/api-client-library/python/guide/aaa_oauth

As Claudio (GOOGLE) mentioned in the above video, (http://youtu.be/zJVCKvXtHtE?t=12m18s), we need to use some logic (client library) to sort your crendentials for reusing. Other wise on every execution of ‘quickstart.py’ script, you need human interaction for getting tokens. I added 6 lines to google’s ‘quickstart.py’ to keep authorization code for reusing. So as long as the user has not revoked the access granted initially to the application, you don’t need a human interaction.

For more details : https://developers.google.com/accounts/docs/OAuth2InstalledApp#refresh

You can get modified google’s quickstart.py using following

or

Goto : https://github.com/sukujgrg/google_drive

So, you got the idea how to deal with Google API and authentication to Google drive without human interaction. Now, you just need to apply your modifications to the script, suitable to your environment and then just put a cron job.

Difference between $* and $@ in bash?

$* and $@, both these bash special variables expands to the positional parameters, starting from the first one.

These variables are same (expand positional parameters in same way) when using without double quotes. If these variables are using inside double quotes, it will expand positional parameters differently.

$* within double quotes ("$*") is equivalent to the list of positional parameters, separated by IFS variable.

Suppose IFS is ":" and hence expansion of "$*" will be like "$1:$2:$3:…"

And $@ within a pair of double quotes ("$@") is equivalent to the list of positional parameters separated by unquoted spaces, i.e., "$1" "$2".."$N". Or in other words, it is equivalent to the list of positional parameters where each parameters are double quoted.

For sake better understanding I wrote a script named star_and_at.sh and pushed to my public github repo

You can clone my bash github public repository directly using following command

“test” operators in Bash

Bash Conditional Expressions

Here I am trying to list almost all available bash “tests” (file tests, string tests, arithmetic tests)

file operators description
-e <FILENAME> True, if exists
-f <FILENAME> True, if exists and is a regular file
-d <FILENAME> True, if exists and is a directory
-c <FILENAME> True, if exists and is a character special file
-b <FILENAME> True, if <FILENAME> exists and is a block special file
-p <FILENAME> True, if <FILENAME> exists and is a named pipe (FIFO)
-S <FILENAME> True, if <FILENAME> exists and is a socket file
-L <FILENAME> True, if <FILENAME> exists and is a symbolic link
-h <FILENAME> True, if <FILENAME> exists and is a symbolic link
-g <FILENAME> True, if <FILENAME> exists and has sgid bit set
-u <FILENAME> True, if <FILENAME> exists and has suid bit set
-r <FILENAME> True, if <FILENAME> exists and is readable
-w <FILENAME> True, if <FILENAME> exists and is writable
-x <FILENAME> True, if <FILENAME> exists and is executable
-s <FILENAME> True, if <FILENAME> exists and has size bigger than 0
-t <fd> True, if file descriptor <fd> is open and refers to a terminal
<FILENAME1> -nt <FILENAME2> True, if <FILENAME1> is newer than <FILENAME2> (mtime)
<FILENAME1> -ot <FILENAME2> True, if <FILENAME1> is older than <FILENAME2> (mtime)
<FILENAME1> -ef <FILENAME2> True, if <FILENAME1> is a hardlink to <FILENAME2>

 

string operators description
-z <STRING> True, if <STRING> is empty
-n <STRING> True, if <STRING> is not empty (this is the default operation)
<STRING1> = <STRING2> True, if the strings are equal
<STRING1> != <STRING2> True, if the strings are not equal
<STRING1> < <STRING2> True, if <STRING1> sorts before <STRING2>
<STRING1> > <STRING2> True, if <STRING1> sorts after <STRING2>

 

arithmetic operators description
<INTEGER1> -eq <INTEGER2> True, if the integers are equal
<INTEGER1> -ne <INTEGER2> True, if the integers are NOT equal
<INTEGER1> -le <INTEGER2> True, if the first integer is less than or equal second one
<INTEGER1> -ge <INTEGER2> True, if the first integer is greater than or equal second one
<INTEGER1> -lt <INTEGER2> True, if the first integer is less than second one
<INTEGER1> -gt <INTEGER2> True, if the first integer is greater than second one

Match a pattern and delete n lines after it

Suppose you have a file named file.txt with following content. Delete all 2 lines [including pattern itself]after the pattern match “Class”.

I got this trick from SERVERFAULT.COM 🙂

Situation:

Suppose you have a file named file.txt with following content
Continue reading “Match a pattern and delete n lines after it”

bash: /bin/rm: Argument list too long

bash: /bin/rm: Argument list too long. If you’re trying to delete files inside a directory and the following command is not working:

If you’re trying to delete files inside a directory and the following command is not working:

In this case you can delete all files using find with appropriate switches:

If you want to delete files in verbose mode:

mysql queries on command line

In this article I like to share some tips which I am using while dealing with mysql databases.

‘-e’ is the switch which enables command line mode for ‘mysql’ client which is installed default on every Linux distros.

Some examples:

1. For listing all mysql users on the server.

2. Show values assisned to each mysql variable

Continue reading “mysql queries on command line”