dandavis.com

A year ago at the How-To Geek: Set Gmail as Default Mail Client in Ubuntu.

I just started using Ubuntu in the past few days and found this post useful. But, because I'm me, I had to make a minor change.

My version is based upon a version in the comments of the original post with additions to support Google Apps for Your Domain.

#!/bin/bash

[[ $2 == "" ]] && domain="mail" || domain="a/$2/"
uri=`echo "$1" | sed -e 's/subject=/su=/' -e 's/^mailto:\([^&?]\+\)[?&]\?\(.*\)$/\1\&\2/'`
firefox "https://mail.google.com/${domain}?view=cm&tf=0&to=$uri"Download open_mailto.sh.

To use it, reference the original post with one small change. If you need to specify a Google Apps domain, add it after the '%s':

/home/username/open_mailto.sh %s dandavis.com

Hope it helps.

• Leave a comment... (0)

A friend (I have lots of friends [no, really, I do]) recently asked how to use AWK to parse a really long file looking for specific strings and combinations of strings.

Unlike some of my friends, this friend knew that AWK was the right tool for job, so that was good. He just doesn't know it quite well enough to build a script to do what he wanted.

Almost everyone knows how to use AWK to extract a specific field from a line (or lines) of text… you do know how to do that, right?

awk -F: '{print $6}' /etc/passwd

This will print out the home directories of all the users in /etc/passwd:

$ awk -F: '{print $6}' /etc/passwd
/root
/bin
/sbin
/var/adm
/var/spool/lpd

And you can print multiple fields at the same time:

awk -F: '{print "User: "$1", Home Directory: "$6} /etc/passwd'

Will give a potentially more useful output showing the username and the home directory:

$ awk -F: '{print "User: "$1", Home Directory: "$6}' /etc/passwd
User: root, Home Directory: /root
User: bin, Home Directory: /bin
User: daemon, Home Directory: /sbin
User: adm, Home Directory: /var/adm
User: lp, Home Directory: /var/spool/lpd

OK, so now to the more interesting stuff…

My friend has a big ole CSV formatted log file exported from a Check Point firewall. It's got a whole bunch of useful information interspersed with some not so useful information… and it's frackin' huge. What he wants to do is search for accepted packets that are sourced from a number of subnets, destined to a single subnet, and are DNS requests. Actually a pretty straightforward parse.

Requirements:

• Field 6 is the action: accept
• Field 12 is the source: 192.168.*.*, 172.16.8.*
• Field 13 is the destination: 10.11.12.*
• Field 16 is the service: domain-udp

So, our AWK would look like:

gzcat exportedlog.csv.gz | awk -F, '\
      $6~/^accept$/ && \
      ($12~/^192\.168\./ || $12~/^172\.16\.8\./) && \
      $13~/^10\.11\.12\./ && \
      $16~/^domain/\
      {print}'

(Sorry, I don't have any output to put here, but if there were any hits from either of the two source subnets to the destination subnet, you'd see the lines printed out. Trust me… It works.)

You don't need to use the '\' and put each piece onto its own line… I just did that for readability.

I'm no AWK expert, nor do I claim to be, but hopefully you found this useful. There is a load of useful information at the GNU Awk User's Guide.

• Leave a comment... (0)

This might not have much of an audience, but I needed to write it so I decided to share it…

I had a need to download log files from a number of security appliances out on a network. The problem was that I needed to access these appliances from a specific server (let's call it the Management Server [192.168.1.1]). Well, I guess that wasn't the real problem… the real problem was that the management server did not have enough hard drive space and the server that did have the space did not have direct access to the appliances.

What's a lazy scripter to do? I mean, sure, I could've put in requests to the network guys to get my storage machine routed to the appliances… and then there are the firewall requests to go with that… oh, screw it.

My management server and security appliances all accept require SSH with public key authentication. How can I leverage that into something that means less work for me?

Well, first I make sure that my public key is on the server and appliances so my scripts can login.

Then I setup my ~/.ssh/config file so that my local machine can access each appliance via a LocalForward:

Host 192.168.1.1
   LocalForward 2220 APPLIANCE_01:22
   LocalForward 2221 APPLIANCE_02:22
   LocalForward 2222 APPLIANCE_03:22
   LocalForward 2223 APPLIANCE_04:22
   LocalForward 2224 APPLIANCE_05:22
   LocalForward 2225 APPLIANCE_06:22
   LocalForward 2226 APPLIANCE_07:22
   LocalForward 2227 APPLIANCE_08:22
   LocalForward 2228 APPLIANCE_09:22
   LocalForward 2229 APPLIANCE_10:22
   LocalForward 2230 APPLIANCE_11:22
   LocalForward 2231 APPLIANCE_12:22
   LocalForward 2232 APPLIANCE_13:22

Here is a link to a file containing the same information: SSH .config file.

OK, so now I have my public key out and about, I have the SSH config file setup to tunnel some ports through my management server out to my appliances. Now it's time to do the work:

#!/bin/bash
 
function GetLogs {
   ApplianceName="$1"
   AppliancePort="$2"
   echo "["`date +'%Y-%m-%d %H:%M:%S'`"] ${ApplianceName}: RSYNC Initiating over port ${AppliancePort}."
   /usr/bin/rsync -aqz \
      -e "ssh -p ${AppliancePort}" \
      support@localhost:/logs/ /home/dandavis/appliancelogs/${ApplianceName}/ 2> /dev/null
   echo "["`date +'%Y-%m-%d %H:%M:%S'`"] ${ApplianceName}: RSYNC Completed."
}
 
# Create tunnel connection
echo -n "["`date +'%Y-%m-%d %H:%M:%S'`"] Building SSH Tunnel to Management Server... "
ssh -S /home/dandavis/.tunnel.socket -M -N -f 192.168.1.1
echo "Done."
 
# kick off the rsyncs into background
echo "["`date +'%Y-%m-%d %H:%M:%S'`"] Initiating RSYNCs."
GetLogs APPLIANCE_01 2220 &
GetLogs APPLIANCE_02 2221 &
GetLogs APPLIANCE_03 2222 &
GetLogs APPLIANCE_04 2223 &
GetLogs APPLIANCE_05 2224 &
GetLogs APPLIANCE_06 2225 &
GetLogs APPLIANCE_07 2226 &
GetLogs APPLIANCE_08 2227 &
GetLogs APPLIANCE_09 2228 &
GetLogs APPLIANCE_10 2229 &
GetLogs APPLIANCE_11 2230 &
GetLogs APPLIANCE_12 2231 &
GetLogs APPLIANCE_13 2232 &
 
# wait for rsyncs to finish
while [[ `ps -deaf | grep -v grep | grep rsync | grep APPLIANCE` ]]; do sleep 5; done
 
# Close tunnel connection
echo -n "["`date +'%Y-%m-%d %H:%M:%S'`"] Closing SSH Tunnel to Management Server... "
ssh -q -S /home/dandavis/.tunnel.socket -O exit 192.168.1.1 2> /dev/null
echo "Done."

And, just in case Wordpress… um… improved upon my formatting, here's a direct link to the script: mainsshtunnel.sh.

WTf?!?

Starting at the top and working down…

The function GetLogs has the steps we follow for each appliance. It prints a message to the console indicating that the rsync is about to begin. Then we do the actual rsync utilizing ssh over a specific port (${AppliancePort}). When it's done, it prints another message to the console.

The first real commands in the script print out a message to the console (I love messages to the console… they're so… informative). Then it creates the SSH tunnel to the management server. The options are:

• -M Puts the client into "master" mode… makes it easier to kill off later on.
• -S Related to the -M option, this sets the control path that we'll use to communicate with the master client later on.
• -N Tells the client not to execute a command on the remote host. Essentially to just connect and then chill out.
• -f Throws the client into the background so we can get on with our business.

Now that the tunnel is up, we can start our rsyncs running. With my network setup, it's no problem running a dozen or two simultaneous rsyncs. In a future post, I'll rewrite this script a bit to allow the user to control the number of concurrent rsyncs running. To start an rsync, we just call the GetLogs function with the appliance name (for directory structure only cause the rsync-ssh actually connects to localhost), the local port on which the ssh client is listening for that particular appliance, and & to throw the whole thing into the background.

After all the rsyncs are started up, we get into a while loop waiting for them to finish up (or die off, whatev). You might need to tweak the ps and grep commands a bit to get them just right for your setup.

Once all the rsyncs are done and the while loop exits, we send a control command to the master ssh client telling it to exit.

That's it. We've got our logs downloaded.

• Leave a comment... (1)

OK, first off, I'll admit that there might be a better way to do this automatically. I don't use Wordpress.com, so I don't know. I know with my own copy of Wordpress, I have plugin options for backing up my database.

A friend of mine asked if I knew a way to backup his Wordpress.com blog. He knew how to download the XML dump, but he wanted an automated way to do it.

I hacked this out in a few minutes… and it seems to work. I haven't tested it extensively, but my friend seems happy with it.

#!/bin/bash

# Edit these to match your blog.
USERNAME="<WORDPRESS.COM USERNAME>"
PASSWORD="<WORDPRESS.COM PASSWORD"
URL="http://<BLOG URL>.wordpress.com"
   
DATE=`date +'%Y%m%d'`
   
if [[ $1 == "" ]]; then
   # If not specified on the command line, the default directory is...
   BACKUP="."
else
   # Otherwise we'll use the one that the user wants.
   BACKUP=$1
   # if BACKUP doesn't exist, make it.
   [[ ! -d ${BACKUP} ]] && mkdir -p ${BACKUP}
fi
BACKUP="${BACKUP}/backup.${DATE}.xml"
   
# Set some default file names...
COOKIES="/tmp/cookies.txt.${DATE}.$$"
OUTPUT="/tmp/output.html.${DATE}.$$"
   
# This is the command we'll use to connect to wordpress.com...
WGET="wget --load-cookies=${COOKIES} --save-cookies=${COOKIES} --keep-session-cookies -q -O"
   
# First, we'll login and get the cookies...
POSTDATA="log=${USERNAME}&pwd=${PASSWORD}"
POSTDATA="${POSTDATA}&rememberme=forever&wp-submit=Log%20In"
POSTDATA="${POSTDATA}&redirect_to=wp-admin/&testcookie=1"
${WGET} ${OUTPUT} --post-data="${POSTDATA}" ${URL}/wp-login.php
   
# Then, using those ever so tasty cookies, we'll download an XML back from wordpress.com...
${WGET} ${BACKUP} "${URL}/wp-admin/export.php?author=all&download=true"
   
# Remove our temporary files...
rm -f ${OUTPUT} ${COOKIES}

I called it wpbackup.sh.

To use it, you can just run it at the command-line with no options and it'll pop out an XML backup of your blog. Or, you can set it up in cron to save a backup periodically depending on how much your content changes. Run from cron, I would suggest specifying a command-line option for the directory into which to save the backups:

wpbackup.sh /home/me/wpbackups

Enjoy. And if you find any bugs, post a comment. If you improve on it in other ways, post it somewhere and take credit… just post a comment here letting me know… maybe you'll have an idea I can steal and use in my own scripts.

• Leave a comment... (0)

In this post, I gave a command to serve up a Mozilla search plugin and icon file so you could use a JavaScript call to add the search plugin without restarting Firefox… that makes more sense if you read the other article…

Well, I decided to put that code into a bash script in case I needed it again… enjoy. Or, not.

#!/bin/bash

FILE=$1
PORT=$2

if [[ "x" == "x${FILE}" || "x" == "x${PORT}" ]]; then
        echo "error: server <filename> <port>"
        exit
fi

if [[ "x" == "x${UID}" ]]; then
        UID=`id -u`
fi

if [[ ${PORT} < 1024 && ${UID} > 0 ]]; then
        echo "error: unable to use privileged port"
        exit
fi

(echo "HTTP/1.1 200 OK" ; \\
 echo "Content-Length: "`wc -c ${FILE} | \\
      awk '{print $1}'` ; \\
 echo "Connection: close" ; \\
 echo ; cat ${FILE} ) | nc -l -p ${PORT}

• Leave a comment... (0)