Monday, July 26, 2010

Some Web Server Management and a Plan for Backups

It has been quite a while since I spent some time administering my personal websites. My sites are hosted using GoDaddy's shared host, which isn't as bad as some of the reviews make it out to be. The big thing that I have been putting off is implementing a reliable and automated backup system. My previous strategy for backups was to simply dump the databases and copy down all of the files once a month, if I remembered. It would not be easy to replace the content of my websites if it were to be lost.


The first step on developing my backup strategy was to clean up my content and current installs. I deleted some web applications and code that I was playing with, but no longer used. These applications were some things that I was playing around with, but never did anything with. Once I did that I made a backup of everything by hand and then upgraded all of my web apps to the latest version. I was now ready to develop my automated system.

The first step was to get a local backup on the web server itself. This was done through shell access to the server using SSH and developing a shell script that performed all of the necessary steps. The two things that need to be backed up were the databases and the actual files. The MySQL databases are simple to backup using the mysqldump command and compressing the output and storing it to a file. The files can be backed up using a simple tar command which can also compress the files down to a reasonable size.

Once all of my database and files were compressed and organized, I took them all and packaged them up in another tar which was my final backup, a single file. This script was set to run as a cron job and the automated backup process was half way complete. The only thing left to do was to find a way to transfer the backup off-site.

My first though was to copy the backup to my personal Linux server. I eventually found a way to automate this process using scp and was happy with the results. It would have been fairly simple to automate this process, but it just didn't seem to be the solution I was looking for. The solution I went with was to store the backups on Amazon S3 using s3-bash. S3 provided very cheap storage and was easily accessed using open source tools that made the process of transferring files very painless. My estimates place the total cost of backups that will be stored on S3 at less than $0.40 a month!

Deciding to use a paid service meant that it would not be logical to store all of my backups indefinitely, and I needed to come up with a plan on how long to keep each backup. I also needed some way to delete backups after they were no longer needed. The solution I came up with was extremely simple. The backup script would run every night and generate and transfer the complete backup, about 45 MB, to the S3 servers. My plan was to keep the backup created on the first of each month for a year, this way I'll avoid data loss if something went wrong that was a long term problem. Additionally, I would keep a backup for each day of the week helping me to avoid loss of data in the short term. After 12 months of operation I would have a total of 19 backup files that would continue to be replaced as time went on. The old backups would not need to be deleted, because by uploading a file with the same key (or file name) it over writes the older version, thereby deleting the old backup.

My backup script has only been running for a few days, but I am very pleased with the results. I still want to do some testing to insure that by backups are comprehensive, but initial inspection shows reveals no problems. This set it and forget it approach is exactly what I was hoping to implement.

Friday, July 9, 2010

The Finish Line is in Sight - Summer 2010

This semester is unique in that I have a large number of projects & papers that I will have completed. Two of the projects are for the classes I am taking this semester and the other two are papers that I hope to have published. I've already heard the good news about one of the papers and will learn about the fate of my other paper in August. First my class projects...

OpenNoteSecure is an Android application that I created for my CECS 564 Cryptology Term Paper. The goal of the application was to store information securely on an Android phone using encryption. The paper I have written about the project, Storing Encrypted Plain Text Files Using Google Android, is almost finished and I will post it to my website after the semester is over. This is my first Android application that I have created and it is available on the Android Market. I have learned a lot about using cryptology libraries and developing Android applications through this project.

This semester I am also taking IE 563 Experimental Design, a class that has proved to be very useful. My project for that class was testing the accuracy of Window's handwriting recognition software. The source code I used (other than the database schema) was included as part of DPX and is available in the subversion repository. The paper about the project, tentatively titled An Analysis of Type II Errors Using Windows Handwriting Recognition on Individual Words and Numbers, covers the statistical analysis based on the data I collected from handwriting samples. After this semester is over I also plan on posting this paper along with the source code, executable, and some instructions to my website.

The paper that I coauthored with three of my classmates comparing my groups capstone project to another similar project was accepted by CGames 2010. Our paper, Comparing Multiple Game Engine Designs To Develop A Unified, Abstract Layer For Supporting Multiple Game Play Scenarios, discusses a method for abstracting the common elements of our game engines and proposes a new design pattern. Both projects were unique in that they implemented game engines and the similarities and differences between the two projects are very interesting. We will be attending the conference here in Louisville where we will present our paper. Luckily, the conference falls after the semester is over so there will be time to prepare our presentation and attend the conference.

Lastly, and probably what I am most proud of, was the paper that I submitted to WIPTE 2010. Titled, A Method For Automating The Analysis Of Tablet PC Ink Based Student Work Collected Using Dyknow Vision, my paper discusses a tool I developed, DPX Grader, for automatically extracting handwritten scores form panels. My goal is to use what I learned from my analysis of handwriting recognition in the above mentioned class project to develop a tool I am calling DPX Answers that extends what I have done with DPX Grader. My goal is to semi-automate the grading process of student work submitted using a Tablet PC. This is rather ambitious, but I am excited to start major development during the break between semesters. I hope to present this application along side DPX Grader at WIPTE 2010 should my paper be accepted.

There is just over a full week of school left in the semester and I am not finished yet. My class projects are not finished, I have a few other smaller projects and homework assignments standing between me and the end of the semester. Back to work!

Thursday, July 1, 2010

My First Android Application: OpenNoteSecure

Yesterday I published my first Android application, OpenNoteSecure (which is open source), to the Android market. It is a simple application that stores encrypted text files on your phone using AES or DES. It was built for my Cryptology project this semester as a demonstration for securely storing information on your phone.


I will eventually have a paper that will analyze the security of storing information on an Android phone using the encryption that is implemented in the application. Once I finish the paper at the end of the semester I will post it online. It is a fairly simple project, but learning about how to develop Android applications has been worth the time.

You can download the application using the QR code. To my surprise it has already been downloaded a few times. If someone finds this application useful it will have been worth the effort.

Powered By Blogger