Wednesday, August 09, 2006

Little League Dilemma

I've coached little league ( a long time ago), it was fun, competitive, and more than anything my kids all had fun win or lose (but we never lost). Which brings us to a story from Salt Lake that just blew my mind. Read the full details here, but here's a quick synopsis:

Salt Lake, little league championship. Last inning. The best hitter on the opposing team is up with two outs. The kid has already hit a homer and a triple. Any rational coach would think about intentionally walking the best hitter so that he doesn't hit another homer, makes perfect sense to me. One problem though, the kid that hits after him is a cancer survivor. So they walk the best hitter, and the cancer kids starts crying after he gets two strikes. Oh boy. What were they thinking? The kid strikes out, crying his eyes out. The opposing coach is furious.

So, if you were the coach would you have walked the best hitter to get to the kid w/ cancer? I'm not sure I'd be able to do much celebrating if I did that. Maybe the bigger question is what in the world is the cancer kid doing batting behind the best hitter. You don't put Mario Mendoza behind Babe Ruth...Idiot.

How many of you would play for the win in that situation?

Tuesday, August 08, 2006

Amazon S3 Part 2

OK, so I got a few complaints (you know who you are) about part 1 of my Amazon S3 post being too long and boring. So....I'll keep part 2 short and sweet. Basically, I wrote a backup / restore script for my linux files using PHP and REST (instead of SOAP). The bottom line is that I love the Amazon REST interface. It's simple and easy to use, forget SOAP, use REST.

The only other issue w/ S3 that I've noticed is that it will fail 1 out of every 2000 or so requests. Based on the forums, you simply have to code around this issue. It's easy to do, but I'm not sure why it fails so frequently. But I've never seen it fail twice in a row, so simply re-trying the request after a failure seems to always work.

Blacksox Turning It Around

My baseball team won this past Sunday 20-2 over a pretty good team. I was amazed. Overall we are only 4-8, in 5th place (8 teams total). But, we have improved dramatically over the last 3 weeks. I'm really looking forward to the remaining games. I'm hitting .310, which isn't bad, but it only is 6th best on the team. My brother is hitting .462 with 17 RBI.

I haven't picked up a golf club in a couple of months, I'm just too busy, so I'll probably wait until the fall to pick it up again.

Our company softball team has formed, I've had fun at the two practices I've been able to attend. It should be an interesting first year, I believe our games should be starting sometime soon.

Friday, July 21, 2006

Amazon S3 Part 1

Two days ago I began looking into Amazon's S3 web service. For those of you who don't know, S3 is described as follows by Amazon:

"Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers."

Hmm...web-scale computing made easy...as a geek there's no way I'm not checking this out, so I went ahead and signed up for an account. The signup process was really easy, I just had to break out my Discover card and they promised to bill me monthly, excellent deal. Once I was signed in I had access to all of the documentation and example code. The cost by the way are as follows:

Pricing
* Pay only for what you use. There is no minimum fee, and no start-up cost.
* $0.15 per GB-Month of storage used.
* $0.20 per GB of data transferred.

I figure I'll owe them a few pennies each month, no big deal. I really just want to play around and figure out what kind of cool stuff I could build if I had the time.

I downloaded the .NET C# SOAP example. Installed it, put in my super secret key, recompiled and kaboom, I was up and running. The example showed me how to program the core concepts with S3 via a command prompt. Here are the technical concepts a developer would need to understand:

1. Buckets

A bucket is simply a container for objects. Each user can have up to 100 buckets. This may sound low, but in reality you only need 1 bucket.

2. Objects

Objects are like files, but they have meta-data around them. Meta-data is data about the objects, key/value pairs. You also have to setup ACL's (Access Control Lists) for each object. You can have an unlimited number of objects. At first glance you may be wondering how to organize all of the objects in a bucket. What is really cool is that you can can use any type of delimeter you want to group objects. UNIX people are use to the "/" seperator, Windows uses the "\" seperator, but you can use whatever fits your application.

3. Keys

Every object has a unique key.

4. Operations

Example operations:

a. Create a bucket
b. Write an object
c. Read an object
d. Delete an object
e. List Keys

That's it. Pretty easy concepts to understand, but it's pretty powerful. So the example project showed me the basic concepts but I wanted to build something useful. So I decided to improve "My Internet Based File System" by creating a program that will allow me to

1. Upload a folder to S3
2. View a list of objects in my S3 account.
3. Download the object to my local hard disk.
4. Delete objects from S3.

After about an hour I had a working version. The hardest part was fixing their example code to handle binary files as well as text files. Once I got that it was just a matter of hammering out the code.

Here is a screenshot of the working version (and yes I do design work on the side):


The main problem with this code is that it reads the entire file into memory before sending the object to S3. Not a problem with small files, but if you ever wanted to upload a big mp3 or something it wouldn't work. In order to get this to work, you really need to "stream" the object to S3. But, doing this via SOAP is rather hard. The basic problem is that SOAP is primarily XML going back and forth. You would need to either dig deep and format the objects yourself (not recommended) or use a concept called "DIME Attachments". Good luck finding example code. And a bigger problem for me was that Microsoft switch their "Web Services Enhancements (WSE)" to use MTOM instead of DIME between versions 2.0 and 3.0. I don't really have the time to try to get this mess working. But I have some other ideas I'm going to play with first to see if I can get this working in a simpler manner, stay tuned.

If you are wondering how S3 works on the backend, the API docs give some clues, here is one:

"If the object already exists in the bucket, the new object overwrites the existing object. S3 orders all of the requests that it receives. It is possible that if you send two requests nearly simultaneously, we will receive them in a different order than they were sent. The last request received is the one which is stored in S3. Note that this means if multiple parties are simultaneously writing to the same object, they may all get a successful response even though only one of them wins in the end. This is because S3 is a distributed system and it may take a few seconds for one part of the system to realize that another part has received an object update. In this release of Amazon S3, there is no ability to lock an object for writing -- such functionality, if required, should be provided at the application layer."

Thursday, July 20, 2006

Some helpful hints when using mysqldump

I always have to go to the mysql reference manual for this, so I'm going to blog about it so that I'm fewer clicks away and I don't have to re-read the mysql manual every 3 days.

First, mysqldump is useful for a variety of reasons, primarily backup purposes, but it can also be very useful for dumping a very small subset of data. We do this commonly to dump live data to test databases.

Here is an example:

Say you have a database called 'webmail' with a table called 'address' and you want to dump all of the address data for a particular user ('owner') from the live system to the test system to debug a problem. You would run mysqldump in the following manner:

mysqldump --skip-opt --quick --extended-insert --no-create-info --user=testing -p --where="owner = 'kevin@domain.com'" webmail address > tmp.txt

mysqldump by default uses the parameter "--opt" which means:
--add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset

Most likely you don't want most of those in this scenario, so that's what all of the options are about.

And to import the data run:

mysql --user=testuser -p webmail < tmp.txt

The most important tip is to make sure mysqldump doesn't add the "DROP TABLE" commands!

Perl

Sometimes perl is a little too forgiving, this little bug has existed for over two years in one of my programs and I just now spotted it...

if(length($hostname < 5)) {
#precaution
return 1;
}

Wednesday, July 19, 2006

You throw like a girl

Yesterday I tried to give Jil and Velvet some pointers on how to throw a softball. Both of them had issues turning their shoulders properly, which isn't a big deal if throwing small distances, but if you really want to throw like the boys you have to turn those shoulders. My other big tip is to get your elbow up. By the end of the practice they were both throwing much better.

I think girls can throw just as well as boys, but I think they don't get enough practice. You won't throw like a girl for very long if you keep practicing because it's just not going to go the distance you want nor have the accuracy you want until your mechanics are good. Eventually you body will train itself to put the arm, legs, and shoulders in the proper positions. So my biggest piece of advice is to keep practicing. Be sure to start slowly though, throwing a ball overhand is not a natural motion for the human arm. It is very easy to get a sore elbow or shoulder so be sure to stretch and warm up properly.

I'm looking forward to our softball season, but I hope it cools down soon, yesterday my Jeep told me the temperature was 105....

Photos From My Phone

It's been over a year since I cleaned up the photos on my phone, here are some good ones...





Tuesday, July 18, 2006

Colin Cowherd UVA Rant

If you haven't heard this clip and are a VT Hokie fan, check it out... UVA fans probably shouldn't listen.

Local News

Webmail.us announced today that we are now staffed in our customer care division 24 x 7 x 365. And yes, all of our customer care staff works from our Blacksburg, VA office. I'm very excited about this announcement because it is an important milestone for our company.

In somewhat related local news, it seems a lot of local communities are really pushing forward with providing wireless Internet access for all of their citizens. Check out this Roanoke Times article. I'm amazed that the smaller communities such as Bland, Hillsville, Radford, and Pulaski are actually leading the way in this area. At the same time I can't help but wonder how they plan on supporting their users.

I have a good friend who actually started a wireless ISP two years ago that primarily services the rural parts of Montgomery county plus some of Christiansburg. He's been very successful and I'm sure he's earned a lot of loyalty due to the excellent customer service he provides. I'm sure his business will continue to grow as long as he is able to continue providing great customer service.

Wednesday, July 05, 2006

More Debugging Tools

Here are a couple of cool debugging tools that will help you accomplish more in less time:

strace
------

strace allows you to attach to a process and watch every system call that the process executes. It is very helpful for tracking down exactly what a program is doing without having to use gdb. It is run like this (use the -f option to strace forks):

strace -tt -o log.txt -p

For an example run this:

strace -tt "uptime"

lsof
---
Another cool command is the "lsof" command. It will give you a list of open files (including sockets) for a process.

lsof -p 3387

Wednesday, June 14, 2006

Using GDB to Debug Linux Applications

I had to refresh my memory over the last two week on how to use the debugging application called 'gdb'. I have needed to step through both Pound and Dovecot's delivery applications to help me nail down a couple of tough problems. Debugging Dovecot's delivery application is very simple, so let's walk through it really quickly.

First, understand how the application is normally executed; normally the MTA will pipe in the email content to the deliver process, but we can fake it by using simple re-direction.

>deliver -d kevin@boebel.us < email.txt

But we want to step through the code, so we need to run it through gdb.

>gdb deliver

Before we issue the 'run' command, we need to set a breakpoint, so let's say we are trying to debug this portion of code in deliver.c (lines 540-546)

env_tz = getenv("TZ");
home = getenv("HOME");
env_clean();
if (env_tz != NULL)
env_put(t_strconcat("TZ=", env_tz, NULL));
if (home != NULL)
env_put(t_strconcat("HOME=", home, NULL));


(gdb) break deliver.c:540
Breakpoint 1 at 0x8056e72: file deliver.c, line 540.

Ok, the breakpoint is set, let's run the program, giving it the correct command line arguments:

(gdb) run -d kevin@boebel.us < email.txt
Starting program: /var/home/kminnick/dovecot/dovecot-lda-0.1/src/deliver -d kevin@boebel.us < email.txt

Breakpoint 1, main (argc=3, argv=0xbfffb3b4) at deliver.c:540
540 env_tz = getenv("TZ");

Cool, the program is now getting ready to execute this line of code, let's run some commands to get the value and type of 'home' variable both before and after the line is executed.

First, we need to skip this line of code and get to the next one:

(gdb) n
541 home = getenv("HOME");

Now we can examine the variable:

(gdb) pt home
type = const char *

(gdb) p home
$4 = 0x3f1cd8 "ä[\023"


As you can see, 'pt' prints the type of the variable, and 'p' prints the actual value. The variable is currently pointing to some garbage in memory. Let's execute the line of code and find the value of the variable after it's execution:

(gdb) n
542 env_clean();
(gdb) p home
$5 = 0xbfffdf44 "/root"

This was a very simple example of how to use gdb to debug your applications, next time we will discuss a more complex example on how to debug a program that uses forks and threads. I'm sure you can't wait... :)

Friday, June 09, 2006

Review of 2006 Goals

Today I benched 200 pounds one time. This was my goal for the entire year, so I am very happy with my progress thus far. Now I'm going for 225 by the end of the year.

Here are the goals I blogged about in January for this year and the results thus far:

1. Blog More
I'm definitely doing this, probably boring everyone a bit. :)

2. Go to the Doctor
Hmm...still six months left.

3. Get my weight to 175 lbs
I'm at 167 (I probably started the year around 163), only 8 pounds to go.

4. Oversee the creation the fastest webmail client on the market.
We are very close...

5. Become the best hitter on my baseball team.
We've only had 2 games so far, but our team captain is kicking butt. I'm hitting .375 with 2 RBI and 2 runs scored. That puts me tied for third (with my brother) on the team. We are 1-1 as a team. Mannin, our captain, is hitting .778 with a homer and 4 RBI. I have a long way to go, perhaps second best hitter on the team will be a good goal.

6. Shoot a round of golf under 110.
I've only played 3 rounds of golf this year. All at The Hill in Blacksburg. I have only had time to play 9 holes each time, but I think my scores have been fairly good, 51, 55, and 60. They are actually terrible scores, but I've made a lot of progress since last year. I think I definitely have a shot of getting a round under 110 if I ever find time to play a full 18 holes.

7. Take my family to Disney World.
Done, we had a great time.

8. Successfully manage our ever growing group of developers.
Ask Pat.

9. Bench Press 200 lbs one time
Today I did this, next up 225.

10. Find more time to spend alone with my wife.
Still working on this, but my favorite memory so far this year has to be the great time we had at Bill and Beth's wedding. We also had a great 5th year anniversary weekend. It's amazing how hard it is to find time alone when you have two kids. I'll keep working on it though.

Windows Vista Tips

Thanks to "Rick H." for posting this tips for Vista:

--------
Just a quick note to say that you can change the default search provider in IE7. Click the drop-down arrow at the edge of the search box in the upper right. Select "Find more providers" and see if your favorite is available (most are). Then click the drop-down again and choose Change Search Defaults.

For the admin popups (It's called UAP or UAC depending who you ask), run secpol.msc. Navigate to Local Policies->Security Options. Set UAC:Behavior of Elevation Prompt for Admins to "No Prompt" and UAC:Run All Admins in Approval Mode to disabled. (Use at your own risk!)
---------

I'm not sure how you figured out the UAC issue but what a life-saver that is for me. Thanks! Now if someone could figure out how to get my transparent windows back I'll be all set.

Thursday, June 08, 2006

Windows Vista Beta 2 Review

After my computer decided to stop booting Windows XP Pro I decided to download and install Windows Vista Beta 2. It's been an interesting trial and error experience but I finally have it installed and running. I first had to ditch my old computer (Vista doesn't support the IDE driver that was built in 2001). I was able to install it on a brand new Dell, but I wasn't able to upgrade Windows XP Pro, I had to install from scratch. I'll probably be better off in the long run, I still hate how Dell insists on installing four million applications by default.

My first impression of Vista: "Cool Stuff". I like the visual changes they have made, I like the fading, the new transparency, the new start menu, the new explorer. I also like the gadget side bar, the new task bar, and the new task switcher. I also like IE 7, but there is no way to change the default search bar. It looks like it just hasn't been added yet, I am sure they will let me change it to Google eventually.

It comes with a few new apps like Windows Calendar and Windows Contact Manager, but they don't seem very useful. There is also some sort of collaboration app built-in, but that doesn't seem very useful either. Perhaps I need a few other users to help me test. The built-in search bar in the start menu seems to be a Google Desktop wannabe, but it appears to work rather well (I don't have a lot of data to test with yet).

I have only needed to install the following applications so far (they all work great w/ Vista):
Firefox
OpenOffice (Site doesn't work w/ IE 7 yet)
GAIM
Putty
Programmers Notepad
TurtoiseSVN
Thunderbird (for testing)

One really annoying "feature" that took two days to figure out is how to make my fonts look like crisp and clear. I was about to give up when I stumbled upon this setting:

Start->Control Panel->System and Maintenance->System->Advanced System Settings->Performance Settings->Click "Custom"->Uncheck "Smooth edges of screen fonts"

That setting has made all the difference in the world. Maybe today I can go home without a headache from my monitor.

Another issue is that I upgraded the video driver to the recommended version and I lost all of the cool transparency and window effects that I had on day one. :( I also haven't figured out how to turn off the annoying security pop-ups that open anytime I try to access an "Admin-Only" section of Windows.

I am finding the new control panel very complicated. It reminds me a lot of Perl in the fact that there are 50 ways to accomplish the same task. Navigating around the new control panel is frustrating, especially with the annoying "Admin Only, Are you really sure you want to do this?" pop-ups everywhere.

Overall, it looks like Microsoft has security as a top priority for this release. They are also finally catching up to Apple's cool windowing effects. And it won't take much more before they are caught up to Google on their desktop search capabilities. I can't think of any real compelling reason for anyone to upgrade besides the improved security features. Perhaps I'll find more in the coming days...

Thursday, June 01, 2006

A Semi-Solution for My Internet Based File System Problem

After thinking about this more today, I have come up with the following solution that will almost work perfectly. First, we have an existing Subversion repository, so I created a "personal/kminnick" project for my own use. Second, we have an existing network drive called the "u" drive which can be used to store personal files. So, I installed a Windows based svn client called Tortoise SVN which is actually a shell extension that integrates directly into Windows Explorer. This basically means I can use Windows explorer to check out / check in files into Subversion, awesome stuff. I checked out my personal project onto our network drive (which is going to be available over a VPN eventually) so that nothing is stored on a local hard disk. To access the files from any location I can use a simple web browser to access the Subversion repository directly. If traveling, I could checkout all the files to my local hard disk, then check them back in whenever I have Internet access. Cool stuff. So with all of this setup and working I can:

1. All of my files are backed up automatically.
2. Have a history of every revision for all of my documents.
3. Access my files from any location that has a web browser.
4. Check out files locally for airplane trips, vacations, etc.

Not a bad solution, it was easy to setup and doesn't require a local hard disk for file storage.

Dead Computer

Today my desktop computer at work died. My dad gave me this computer about 2.5 years ago because he bought a new one, I'm guessing it was only about 2 years old when he gave it me. I was very attached to it, but all good things must come to an end. I was constantly asked by my co-workers why I never upgraded to a new machine. Well, for one, I never really “needed” a new machine. I spend most of my day inside of an SSH session, so memory, CPU, etc isn't that necessary. I'm curious though how many Word documents I'll need to re-type because I saved them to my local hard drive instead of on the network. I usually back everything up to the network or have them somewhere in my email for safekeeping, but I'm sure I missed a few. I use webmail exclusively for email, contacts, calendar, and tasks, so nothing will be lost there. I'm going to try to put in a new hard drive into my old machine and re-install Windows. But this time I'm going to try hard to move everything onto the network, the way it should be stored. In fact, I'm going to try to find a solution that will allow me to store everything I need onto the Internet.

This thought has really made the wheels turn this morning. I really need the following:

1. A web-based central repository that I can put / get files (any file type).

2. It should work like any good version control system (subversion), meaning it keeps track automatically of revision numbers, and I can view old revisions easily.

3. It would be great if I could open the files using “File, Open” from any application such as Open Office, Photoshop, etc. I shouldn't need to download the file first locally. I imagine this would require some sort of add-on to Windows Explorer. It needs to be fast though, so the plug-in should pull it down to a local temporary location, but automatically save it on the Internet when I click “File, Save”.

4. It should support sharing, so instead of having to email a document to everyone, I could simply email a note to everyone I want to share the document with and they can pull it down after entering an email / password. The link should always point to the latest revision so that timing isn't an issue. You should also be able to setup who can make modifications to the document.

5. You should have the ability to setup notifications so that you know when the document has been changed / modified.

6. It should be integrated inside of webmail.

For now, all I really need is #1. I think to solve that I'll setup a subversion server for myself and put all of my documents into subversion. I will be able to access it from home and work, the only downside is that I won't be able to open / save documents without the help of a subversion client. But at least everything will be backed up automatically and all of my documents will be accessible from the Internet.

If anyone knows of a solution that will meet the first five requirements above please let me know. I honestly haven't researched it yet.

Thursday, May 25, 2006

Trac

Yesterday I setup and installed Trac for our development team. Trac is one of those tools that once a developer sees it, they gotta have it. If you havent heard of Trac, it allows developers to easily browse Subversion repositories. Subversion is the tool we use to track code changes. Subversion doesnt have a good web-based GUI for browsing the repository, which is the primary reason we wanted to install Trac. Trac is easy to navigate, provides timelines, and makes browsing differences between revisions easy-easy-easy. The developers at Edgewall really have a winner with this product.

Trac also has a bug ticketing system, wiki, report generation tool, and milestone tracking tools built-in. The other tools need a bit more flushing out, especially the bug-tracker, but the SVN browser makes it worth the trouble to install. The installation process for Trac iswelltough. Ive installed hundreds of applications by source and RPM over the years and Trac was definitely one of the hardest Ive ever encountered. Not only are there a ton of dependencies, but you have to be very careful that you have the right versions of each dependency. Thank goodness Google was able to help quite a bit.

I can see the benefits of Trac already paying off; just this morning, Mike sent me a link to the change-set in Trac for the next upload. Clicking on the link takes me to a web page with a description of the changes, a list of all files that were changed, and a color-coded side-by-side diff for each file.great stuff.

Thanks to Joel, one of our latest hires, for directing us to this application.

Sunday, May 21, 2006

5th Wedding Anniversary

When I was in the basement today I saw a picture album from our wedding night and decided to reminiscence a little. First, it's amazing how much changes in five years. Second, it amazing how much stays the same. Velvet, Bill, Jason, and Velvet's parents all look the same today as they did five years ago. Pat, my parents, and my brother look totally different. I look like another person and wonder what Velvet was thinking...I'll bring the pictures to work if anyone is interested in looking.

To celebrate our anniversary we dined at The Summit restaurant in Christiansburg for the first time. Velvet suggested it and I heard three reviews all raving about the great food. I have to admit the food was really good (even though my steak came out a little too done). If you are looking to impress your date with some great food and a nice atmosphere I definitely recommend The Summit. :)

I'm more of a “look to the future” guy than “dwell on the past” guy; so I'll give you some insight on what will be going on five years from now, instead of what has transpired over the last five....In five years Drew will be in kindergarten and Jordan will be in 4th grade. I'll be working on my 11th year at Webmail.us. I'll only have 20 more years to pay on my house (haha). Maybe I'll have the driveway paved by then so I can put up a basketball goal for Jordan. Hopefully I'll still have my Jeep (gas will only be $6.00/gallon but that still beats having a car payment). Maybe my golf game will be in the 90s (OK, so I'm dreaming a little). Jordan won't remember a time when TV wasn't in high definition. Wireless Internet access will be available practically everywhere for free. Jordan will be using this laptop and think it's archaic. I might be caught up on the Sopranos. I hope Jordan can teach Drew how to play chess, read, and create his own blog. Jordan will still be in gymnastics (and will keep requesting a trip to Sonic after each session). Velvet and I will be celebrating our 10th Wedding Anniversary.


Monday, May 15, 2006

OpenLDAP Errors

I ran across the following error in OpenLDAP two weeks ago when deleting an entry:

"Additional info: entry index delete failed"

Huh? We've been using OpenLDAP since 2003 and I've never seen this problem before, but looking at the OpenLDAP code showed it was trying to delete an index. We use the BDB backend and apparently a BDB index was corrupt. I haven't found an easy way to repair these indexes besides dumping all the data and re-importing. Maybe a BDB expert out there has some suggestions but luckily we replicate all of our data across multiple machines so I just had to copy the data (zip up the /var/openldap-data/ directory) from another machine. Googling for the error above didn't return many results so I thought I would go ahead and post my findings for the next person who searches for this error.