dandavis.com

A friend (I have lots of friends [no, really, I do]) recently asked how to use AWK to parse a really long file looking for specific strings and combinations of strings.

Unlike some of my friends, this friend knew that AWK was the right tool for job, so that was good. He just doesn't know it quite well enough to build a script to do what he wanted.

Almost everyone knows how to use AWK to extract a specific field from a line (or lines) of text… you do know how to do that, right?

awk -F: '{print $6}' /etc/passwd

This will print out the home directories of all the users in /etc/passwd:

$ awk -F: '{print $6}' /etc/passwd
/root
/bin
/sbin
/var/adm
/var/spool/lpd

And you can print multiple fields at the same time:

awk -F: '{print "User: "$1", Home Directory: "$6} /etc/passwd'

Will give a potentially more useful output showing the username and the home directory:

$ awk -F: '{print "User: "$1", Home Directory: "$6}' /etc/passwd
User: root, Home Directory: /root
User: bin, Home Directory: /bin
User: daemon, Home Directory: /sbin
User: adm, Home Directory: /var/adm
User: lp, Home Directory: /var/spool/lpd

OK, so now to the more interesting stuff…

My friend has a big ole CSV formatted log file exported from a Check Point firewall. It's got a whole bunch of useful information interspersed with some not so useful information… and it's frackin' huge. What he wants to do is search for accepted packets that are sourced from a number of subnets, destined to a single subnet, and are DNS requests. Actually a pretty straightforward parse.

Requirements:

• Field 6 is the action: accept
• Field 12 is the source: 192.168.*.*, 172.16.8.*
• Field 13 is the destination: 10.11.12.*
• Field 16 is the service: domain-udp

So, our AWK would look like:

gzcat exportedlog.csv.gz | awk -F, '\
      $6~/^accept$/ && \
      ($12~/^192\.168\./ || $12~/^172\.16\.8\./) && \
      $13~/^10\.11\.12\./ && \
      $16~/^domain/\
      {print}'

(Sorry, I don't have any output to put here, but if there were any hits from either of the two source subnets to the destination subnet, you'd see the lines printed out. Trust me… It works.)

You don't need to use the '\' and put each piece onto its own line… I just did that for readability.

I'm no AWK expert, nor do I claim to be, but hopefully you found this useful. There is a load of useful information at the GNU Awk User's Guide.