Careers in BIT Panel

I had the privilege today to be a panelist for the Pamplin College of Business "Careers in Business Information Technology Night" at Virginia Tech. Anytime I have the opportunity to meet students at Virginia Tech I take it, but honestly, I know nothing about "Careers in BIT", haha. But I'm glad Pat hooked me up with the opportunity because I enjoy going on campus and meeting new people. Gary Kinder is the Director of Undergraduate Career Services and he did a great job of moderating the event. Here were some of the questions and my answers:

1. What was the one thing that "sold you" on the field of information technology?

I knew going into college that computer science was the major for me because of the excellent computer related classes I took in high school. Our high school was small, but I had the opportunity to learn how to program from a really good teacher and I had no doubts this was what I wanted to do as a profession.

2. What's the two or three most helpful classes you had in college and why?

Any CS class was very helpful, but outside of CS I really enjoyed my two economics classes. Getting an "A" in econ was really exciting for me because there were a lot of business majors in my class who struggled. I guess I liked proving that I could handle more than programming.

3. When looking at applicants for positions within your company, what are the skill sets you find to be best suited for the positions?

I look for people who have two basic characteristics; they are smart and can get things done.

4. What types of things do you read, as far as professional journals, trade papers, industry material, etc.?

I follow the blogs of individuals from my industry that I believe are smart, on the cutting edge technology, and have similar jobs / goals as I do.


There were many other questions but those are the ones that stand out in my memory. A general theme of the night was how to decide on a career path. My advice was to do as much research as possible as early as possible. Co-op, intern, get a part-time job, talk to employers at career fairs, talk to graduating seniors, get a feel for the types of jobs that are available and how they line up with your interests.

The best part of the night was receiving a VT coffee mug as a gift for being on the panel!

Thanks again to Gary and Pat for the opportunity to participate! Gary, if you need help with the "Careers in Marketing" panel on Thursday just let me know. ;)

2006 Black Sox Team Picture

2006blacksox
Originally uploaded by kevin_minnick.

Our team picture has finally been published. I'm on the front row, third from the left ( or right ).

Other team members (back left to front right):

Ricky Smith, Adam Harber, Mike Mayo, Scott Minnick, Mannin Dodd, Zac Adelman, Brad Mullins, Paul Bielema, Sam Wood, Edwin Whitelaw, Me, Jason Ferritto, Tom King

Not Pictured:
Frank Bartin

Web Sox Win !!!

Last night we won our first softball game. Our softball team is composed of primarily Webmail.us employees and several of our close friends. Until last night we were yet to play a full game because our team is usually losing so bad that they call it early (known as the "run rule"). We came through last night though and won an exciting extra inning affair. Bill scored the winning run off of a nice hit by Brian Hartsock. It was a close play at the play but Bill was way too speedy and determined to get caught at home.

It was a true team effort and it felt like a thousand pounds were lifted off of our shoulders after that win. We are gaining confidence every week and are starting to gel as a team. I'm looking forward to our remaining games. So far this certainly has been a draining experience and it takes a lot to keep your head up after being killed game after game. But our team showed that we are not quitters and we can play in this league. Thanks to everyone who gave such a great effort, I'm looking forward to our next win and celebratory dinner at PK's.

Black Sox: 2006 NRV MSBL Champions !!!!

It was a beautiful day for a baseball game. We won 15-7. I think everyone on our team is a bit shocked that we were able to rip through the playoffs and win the championship. This is the third year of the league and we have played for the championship twice. The first year there were only 4 teams, we won the first playoff game, lost the championship game. The second year the league had 6 teams and we lost in the first round. This year the league had 8 teams, and we won 3 straight games to take home the championship. After finishing the regular season in 5th place I don't think many of us believed we would be able to make it to the championship, much less win it. But, like most single elimination tournaments, anything can and will happen.

There were three key events that won today's game:

1. We scored 11 runs in the first inning, giving our pitcher tons of breathing room, after that we just had to play good defense and hang on.

2. We have played this team so many times we know how to play each batter defensively. Most importantly, after the butt kicking they handed us the last game of the year, we knew not to pitch to their fourth batter. He is an amazing hitter, it's better to just walk him and pitch to their other batters. This kid was the Babe Ruth of our league.

3. Ricky Smith. Ricky has been on our team all three years. He is a good player but had a really rough year at the plate. Ricky was put into the game in the 5th inning to play left field. In the 7th inning we were in a tight spot. The Warriors had managed to load the bases with two outs when Babe Ruth steps into the box. I was thinking to myself, let's just walk him (scoring a run) and pitch to the next guy. Our pitcher threw a pitch about 3 feet outside and Gerald (his real name) drives it deep to left field. I start yelling "Back! Back! You got room! Back! Back!". Ricky is running hard straight back. I'm sure he's thinking he will hit the fence at any point but snags it 2 feet in front of the wall. We all scream in excitement, understanding that Ricky just saved the game. If that ball isn't caught, 3 runs score and all of a sudden our win is in doubt. Our pitcher gave Ricky a big hug in the middle of the field as we trot off to the dugout.

I'm sure our team wasn't the best team in the league. We were probably the 4th best team talent-wise. But, the most talented team doesn't always win the big game. Our team is a good example of why we actually play the games. String together several hits, play good defense, and don't make mistakes and anything is possible.

Great job Black Sox! Just think, every time we take the field next year we take it as "Defending League Champions". ;)

Bench Press

I'm stuck at 200 pounds. That was my original goal for this year, but I reached it so early in the year I feel like I should easily hit 225 by the end of the year. Hmm, so how do I get over this hump? Well, I did some Googling and found some good tips and a workout that I'm going to try:

1. Improved Form

I have a tendency to pick up my feet and let my elbows get away from my sides when the weight gets heavy. I'm going to try to correct these problems ASAP.

2. Squats

I haven't worked my legs all summer. I give the pitiful excuse that since I play baseball I don't need to do them, the reality is that I am a lazy fool when it comes to lifting legs. So today I did my first leg workout in months and I'm forcing myself to do squats. Now, how can doing leg exercises help your bench press? Well, it all comes back to the level of testosterone in your body. After reading this article I was sold and I'm going to force myself to do squats every week, no more excuses.

3. Start Over

I'm going to completely change my workout. I typically do

135 x 12
155 x 8
175 x 5
185 x 2
195 x 1

Starting next week I'm going to do this workout over a 12 week period:

Week	Weight	Sets	Reps
1	135	3	6
2	145	3	6
3	155	3	6
4	165	3	3
5	175	3	3
6	185	3	3
7	155	3	6
8	165	3	6
9	175	3	6
10	185	3	3
11	195	3	3
12	205	3	3

This means that by Dec. 22nd I should be lifting 205 three times, which means I should be able to bench 225 one time easily (according to this chart, which seem correct). The above workout should be easy until week 6, I'm a little scared after that. :)

I'll be doing this the rest of the week:

Monday - Light Bench, Curls, Calves
Tuesday - Squats, Leg Extensions
Wednesday - Pull-Ups, Hamstring Curls, Shoulders
Thursday - Off
Friday - Bench, auxiliary Chest

I'm putting my faith in the workout, wish me luck.

Rain Rain Go Away

Our championship game today was suspended in the 2nd inning due to rain. The good news is that we were winning 12-1 when it was called. The bad news is that there are 7 innings left to play....

My college football poll remains unchanged, but look out for Florida, they are really starting to impress me. They should fire their AD for scheduling Alabama, LSU, @Auburn, and Georgia consecutively. If they can make it through that stretch they will be #2 in my poll.

At the beginning of the year I picked Va Tech to beat Georgia Tech...boy was that stupid. I am looking forward to the game, but the Yellow Jackets have an above average defense and the best wide receiver in the country. Not to mention a quarterback that can scramble which is something the Hokies have always had a hard time stopping.

Andrew turns one year old this Thursday. His walking skills are really impressive. I enjoy watching him go from object to object and laughing at himself when he makes it without falling. We are going to have a birthday breakfast for him Saturday before we head over to the tailgate.

VT Recruiting

It's recruiting season at Virginia Tech. I spent a few hours on Monday and Tuesday on the Virginia Tech campus hanging out with computer science students. On Monday through our membership in the Computer Science Resource Consortium at Virginia Tech I attended a luncheon for all of the CS majors who won scholarships over the last year. I'm not sure how I didn't know this, but it turns out the number of CS majors is dropping every year. And the percentage of CS majors that are women is dropping every year too (I actually knew this part). Women only account for 6% of all CS majors. I'm sure that means that in many CS classes you will be the only female in the class, not good.

On Tuesday, I had the opportunity (again via our membership in the CSRC) to join a panel that would attend a junior level CS class and answer questions from the students. It's funny how I remember doing this as a student back in 1999. Back then our class size was triple the size of the class I attended on Tuesday. It's pretty scary how few CS majors are out there and how huge the demand is going to be over the next few years.

The other members of the panel were from:
Vanguard (11,500 employees)
MITRE (6,000 employees)
Microsoft (61,000 employees)
Northrop Grumman (24,000 employees)
Lockheed Martin (135,000 employees)
SWIFT (1577 employees)

Webmail.us has 40-ish employees.

Needless to say I had an unfair advantage over all of these companies. One, I was the highest ranking person on the panel. Two, our company is local, so they can stop by and see me whenever they want. Three, poking shots at Microsoft is almost too easy. Four, start talking about time tracking at a defense contractor and you can almost get the recruiter to start looking for a new job. :)

Perhaps the most interesting question was "What don't you like about your job?". I was the only one who said "Nothing." And it's true, I love my job. We were also asked:

1. What's your opinion on the job market?

Crisis. The U.S. over the next five years will see the largest shortest of (wo)manpower in it's history. Good for students, bad for employers.

2. Define computer science.

The other members gave good definitions, I just added: CS is a discipline that allows you to solve problems in a creative and artistic manner. Beauty is in the eye of the code reviewer, tester, and user. (I just thought of that last part, but it sums up my long winded answer nicely.)

3. How do I keep my skills sharp and stay on top of the latest technologies?

Find the right job. Find a job that allows you to do this type of stuff during work. Wouldn't that be awesome? (I can see the defense contractors squirming in their chairs, hahaha). But, if you do accept a job with one of these other companies, stay on top of things during your free time, it will be well worth it when it comes time to find a new job.

4. When looking at resumes, do you favor students with master degrees over bachelor degress?

NOPE. We hire smart people who like to get things done.

We Won ?!?!!!

I was simply shocked when I found out yesterday that the Black Sox destroyed the NRV Boyz on Sunday 23 - 7. Wish I could of been there for that butt kicking. NRV Boyz were the #1 ranked team going into the playoffs. I thought we had a shot at winning, but 23 - 7 ? Wow, nice job guys. The rumor at the beginning of the year was that the championship game would be played at English Field on the Virginia Tech campus. I've never played on that field, it should be fun. Unfortunately we play the Warriors again, the team that crushed us the game before the playoffs started. I'm sure we will be able to give them a better game this time around.

A Long Drive

After the Tech football game on Saturday Velvet and I packed up the kids and headed down to VA Beach for my niece's 5th birthday party. We get down there in good time (about 5 hours flat) and hung out with my sister for about two hours before heading over to the hotel. On the way to the hotel it became more than apparent that my son was not feeling good. After a couple hours of cleaning up and doing some laundry I finally make it to bed around midnight. The kids wake up around 7:00am and Andrew is not any better. I called my sister to inform her we wouldn't be able to stick around for the party and by 11:00am we were on the road back to Blacksburg. We made it home around 4:30pm. So I wasn't able to watch very much college football on Saturday, but based on the highlights and scores here is my new "top 5 teams most likely to play in the national championship" game list:

1. Ohio State

No change here, but with Michigan's unexpected win over Notre Dame it makes their season finale really interesting.

2. West Virginia

Only one opponent on their schedule can beat them.

3. Michigan

A very weak schedule with the exception of Notre Dame and Ohio State. If they can beat Ohio State in Columbus they will probably be #2 in the country.

4. Auburn

A nice win over LSU, if they can win out they are in, but a tough road ahead.

5. USC

A nice win over Nebraska, but was anyone else unimpressed with the game overall? I'm losing faith...

Honorable Mention: Louisville

And for you Hokie fans out there, I hope you paid attention to that Clemson game. It's going to be an interesting Thursday night when Clemson comes to town.

Amazon EC2

Today, with the help of Wenjie, I was able to provision my first 4 development servers at EC2. It was a really easy process. I had no problems installing the software needed for a MySQL cluster. Although I was unable to figure out the sql-node part of the cluster, I was able to get everything else running in under an hour. EC2 seems like a perfect environment for a test system. Having the ability throw away or create servers with one command is amazing. Setting up firewall rules was really easy too, I definitely recommend checking out EC2.

National Championship Predictions (Week 2)

My updated list of the most likely teams to play for the National Championship (not necessarily the best teams in the country):

The top 3 are the same:
1. Ohio State
2. USC
3. West Virginia

I guess I screwed up with Miami and Penn State, here are my replacements:

4. Notre Dame

An impressive home victory over Penn State. If they can beat Michigan at home next week it's smooth sailing until the USC game the last week of the year.

5. An SEC team

OK, so this is the easy way out, but my goodness look at all of the SEC teams in the top 15: Auburn, LSU, Florida, Georgia, and Tennesse. I don't see how any team in the SEC could come out with zero losses, but if one does, they deserve to play in the big game. I need to watch a few more games before I pick a winner here.

Honorable Mention: Louisville

Black Sox Win !!!

The first round of the NRV MSBL is over and the Black Sox have advanced to the second round by pounding their first round opponent 22-10. We took it to them and never let up and it was an easy win. I went 2-3 with a triple and 4 RBI. A triple? Amazingly enough that was my first extra base hit of the year but it was by far my best hit of the year. I crushed the ball over the left fielder's head and just kept running. Next week we face the NRV Boyz. The first place team in the regular season. Win that game and we are in the finals. It will be a tough opponent but I hope we can pull through.

One problem for me though is that my niece is turning five next Sunday and is having a birthday party in Va Beach that I'm suppose to attend. Hmm, what to do, what to do...

Black Sox Regular Season Finale

Tonight we played our final regular season game at Calfee Park in Pulaski. For those of you who don't know the area that well, Pulaski is a small town about 20 miles south of Blacksburg. Calfee Park is home of the Pulaski Blue Jays, a rookie affiliate of the Toronto Blue Jays. The park was built in 1935 and is simply an amazing place to play a baseball game. We rarely have an oppurtunity to play there, but since the Blue Jays season is over, we had the oppurtunity to make up one of our rained out games there tonight.

We typically play all of our games on high school fields in the area. It's amazing the difference between a high school field and one that is used for minor league baseball. The field doesn't raise or lower an inch from the backstop to the warning track. The infield is so smooth you just want to run around on it all day because it makes you feel fast. The infield grass is cut so close you would think it was the green of the 18th hole of a golf course. The pitching mound is the proper height and there are no big holes to slip and fall into. The batters boxes don't have huge divots from people standing in the same spot all year. The fence is so far away you couldn't imagine ever hitting a homer. The dugouts are actual dugouts (like lower than the field). The stands and bleachers look nice and comfortable. There are actual bullpens to warmup relief pitchers in between innings. I love playing on this field, you feel like a professional the moment you step on the field.

Unfortunately, we got our butt kicked tonight. We barely made it out of the first inning. I think the final score was 26-15 or something crazy. (Although scoring 15 runs is pretty good for our team.) I was 0-3, I guess I've been playing too much softball. Although we lost, I had as much fun playing as I have had in awhile. First, I know most of the guys on the opposing team and if you are going to get hammered I'd rather it be by a team that is a good group of guys rather than a bunch of a-holes. Second, I got to play catcher for three innings. I normally play centerfield, and I got to play second base several times this year, but I really don't like being the catcher. But, my opinion of the position changed dramatically tonight. I really had fun back there behind the plate tonight. One reason is that the pitcher did a good job of keeping the ball out of the dirt. That's the hardest part of being a catcher is trying to stop all the balls from going to the backstop on those low pitches. Second the pitcher didn't throw particularly hard or have nasty stuff. I would say he had decent stuff, but it's stuff that is very easy for me to catch. Last year I had to catch for Mannin and he throws pitches that are unbelievably hard to catch. Just really nasty pitches that break hard, late, and are thrown really fast. Last summer I caught an entire game for him on the hottest day of the summer and I thought I was going to die. Tonight, totally different. It was a blast and I'd definitely do it again, but not for Mannin, haha.

Our first playoff game is this Sunday at 10:00am. Yep, 10:00am. We get the first game of the quadruple header for our league at Calfee Park. It should be a blast. We finished the season in 5th place, which means we get to play the 4th place team in the first round. A game that is very winnable (we have already beat them 2 times this year). But nothing is ever easy for the Black Sox so wish us luck, we will need all of the help we can get.

MySQL Cluster 5.1

I've been researching MySQL cluster today and my brain is on a bit of information overload so I thought it would be wise to jot down some key points I have learned thus far.

MySQL cluster differs from a traditional MySQL installment because it is designed to work across multiple machines. In the traditional setup you install MySQL on one node and have several slaves that replicate the data and serve as backup servers in case of failure. In the cluster scenario there are a minimal of 4 machines (technically 3, but if you care at all about your data you will need 4). Machines fall into one of three categories:

1. Management Nodes
Management nodes are important for defining the roles of each of the other nodes in your cluster. Here you setup config files that the other servers read to understand how the cluster is setup. You would need at most two of these, one just sitting there in case the primary fails. If the primary does fail, it will not cause the cluster to stop working.

2. MySQL Client Nodes
Client nodes are the machines that actually perform all of the queries received from clients. You can setup as many of these as you load demands, preferably at least two in case of failure. In MySQL 5.1 these nodes do perform caching, but there are some gotchas that you should read in the documentation.

3. Data Nodes
The nodes use the NDB storage engine to replicate your data across multiple machines. This is layer I have the most questions about and I'm trying to find out more. Prior to 5.1 all database data, including indexes was stored in memory. So if you have large tables you need to make sure your data (and indexes) will fit into the available memory. A new feature of 5.1 is the ability to store only the primary key information in memory and store the rest of the data on disk. This will help a lot of really large data sets.

So if you had 2 management nodes, 2 client nodes, and 2 data nodes any of the nodes in the cluster could be shut down and there would be no loss of uptime or data.

A nice feature added to 5.1 is the ability to replicate clusters. So if you are paranoid and afraid that an entire cluster could go down then you could fail over to the secondary cluster. I didn't read the entire doc on this topic, but it seems complicated and probably overkill for most situations.

It's important to understand how indexes are implemented in the cluster scenario. Primary keys are implemented as a hash index type. They do this so they can partition your data based on the hash. This brings up two key points that must be understood when looking into the clustering:

The following queries will work great and be very fast (from the performance pdf available online):

SELECT * FROM tbl WHERE pk = 5;
SELECT * FROM tbl WHERE pk = 'ABC' or pk = 'XYZ';

The following queries will not use the primary key index:

SELECT * FROM tbl WHERE pk < 100;
SELECT * FROM tbl WHERE pk_part1 = 'ABC';

The last example is especially important to understand. If you have a primary key that uses multiple columns (pk_part1, pk_part2) you must specify the value of both columns in order for the index to be used.

Ordered indexes are implemented using T-tree's instead of B-tree's. This is for memory saving purposes and the need to not have to worry about disk-seek times (everything is in memory). This really should make any difference to the client but I mention here because I thought it was a great idea.

Data retrieval in a clustered scenario is very important to understand. If you do a query based on the primary key, the client node can go directly to the data node that has your row and retrieve it. If you do a query which results in the client having to use an ordered index, the client must query in parallel every data node in the cluster. This may be really fast, but would result in much more network traffic than a primary key lookup. The worst kind of query would be one where the client node must do a full table scan. In this scenario the query must be sent to all nodes. Prior to version 5.0 the "where" clause was not sent to the data nodes so that meant that the entire data set was returned to the client for scanning. If you had 2 million rows, all 2 million would be sent over the network for scanning. Since 5.0 you can enable an option to perform the "where" on each data node and only return the rows that match the where clause.

I think picking a primary key is especially important when implementing a table that will reside in the cluster. Pick the wrong key and I think your queries will be much slower than the traditional database. On that note, it's not uncommon for a clustered database to be slower than a traditional database. If you look at everything that must happen to perform a query in the clustered scenario that makes sense. But I think the clustered database kicks but in the ability to maximize throughput.

As I mentioned earlier, my main questions so far are in regards to the data nodes. I haven't found any documentation that covers how data is spread across the nodes. For instance, if I had a configuration with 50 data nodes, and I have a database with 100 million rows, how much of the data would be on each node? Is the data triple replicated? Double replicated? Which data nodes are safe to shut down at the same time? Does anyone out there know or could point me in the right direction?

Cool GUI Tools for MySQL 5.0

I just found these tools for 5.0, awesome stuff, but I'll probably still do most tasks the old fashion way. :)

http://dev.mysql.com/downloads/gui-tools/5.0.html

MySQL Partitioning

Bill asked me to look into the new paritioning features with MySQL 5.1. I have to say this has to be one of the best features I've seen added to MySQL in a long time. Check it out:

http://dev.mysql.com/tech-resources/articles/mysql_5.1_partitions.html


Finally the admin has some kick-ass control over how the data is split across files and disks. And it makes pruning old data super-easy.

College Football is broken, here is how I would fix it

I love college football. It's infinitely more interesting than pro football (unless you play fantasy football, which I don't). But it could be even better, here is what I see as college football's biggest problems:

1. Rankings
2. Every game matters too much
3. Out of conference scheduling
4. No playoffs

Rankings matter too much, we shouldn't care about Coaches Polls, AP Polls, computer polls, etc. There are too many polls and none of them should matter when deciding a national champion.

It's really tough for a one loss team to win a national championship, which means colleges that are smart will schedule as many "Northeasterns" as possible. Ideally, colleges should want a tough out of conference schedule to get them ready for the post season.

Without a playoff system, the teams that are extremely lucky and highly thought of in the polls play in the big game. Nothing is decided on the field and there will always be the one or two teams that make tremendous strides during the regular season but have no chance of proving they are the top team in the country.

Here is how we fix this mess:

1. The winner of each major conference (there are 7) gets an automatic bid into the playoffs. This does several things right out of the gate: we no longer care about polls, we can schedule tough out of conference games, and losing one or two games doesn't really matter.

2. The press still gets to pick the 8th and final playoff spot from the remaining conferences and independent teams. (So yes Notre Dame fans, it's still fixed for you.)

3. Force each team to only play 10 games (plus 1 for a conference championship if needed). This will mean that the football season won't last any longer than it currently does (don't forget that they currently take over a month off before the bowl game). And only 2 teams will play an additional 3 games max, so we are talking 14 games maximum, and some teams currently play 13.

4. We can still have several bowls to satisfy the other teams that didn't make the playoffs but had good years. I think the fans would still go to these games and they would still generate lots of money. But let's get real and reduce the number of bowls in half.

Amazingly enough, college basketball has everything right. Polls don't matter that much, they have a great playoff system, and teams schedule tough opponents to HELP their chances of making the playoffs. What an amazing concept!

MySQL: InnoDB vs MyISAM tables

MySQL gives the user several different storage engine options when creating a new table in a database. The two primary options for traditional databases are InnoDB and MyISAM. I won't go into the history of each engine, but I'd like to discuss when you would choose one versus the other. InnoDB has several features that MyISAM does not provide:

http://dev.mysql.com/doc/refman/5.0/en/innodb-overview.html

So why would anyone ever use MyISAM? First, it's been around longer and is very reliable. People are also more comfortable with the one file per table concept. And maybe most importantly it is the default engine when you create a new table. I've used both and like both, and when I was testing MySQL 5.0 over the last few days I decided to put both to several speed tests on a rather small table (45,000 rows). Each row contained about 470 bytes of data. Every test I ran showed that each database had similar performance. I was hoping that one would be superior to other but it just wasn't the case. On the other hand, my table size was pretty small. And I did discover that with large text based primary keys, MyISAM was much faster. But when the key size was reasonably small InnoDB had slightly better performance. I'm going to try to insert a tera-byte of data into each database and see which one performs better. I love all of the different options available for InnoDB storage that are available so it should be fun to try out different parameters and see which ones perform best.

National Championship Predictions

OK, after one weekend in college football I've seen enough to know who has a legitimate shot at playing for the national championship.

1. Ohio State

A great team with two Heisman candidates. A tough opening schedule, but if they survive at Texas and can hold off Penn State in week 4 there will be no stopping this team.

2. USC

All of their "tough" games are at home (Nebraska, Arizona St, Oregon, California, Notre Dame.) Plus they don't play a conference championship game.

3. West Virginia

Their schedule is laughable and no conference championship game.

4. Miami

Underrated and an easy schedule.

5. Penn State

A dark horse candidate, but if they manage to somehow beat both Notre Dame and Ohio State they deserve to play in the big game.

Teams that will NOT be contending, but are good teams:

1. Texas

Their schedule is tough, Ohio State, Oklahoma, and Nebraska. If they do somehow manage to beat all of those teams they deserve a shot in the big game.

2. Notre Dame

I can't see anyone beating USC, especially @ USC.

3. Any SEC team.

SEc schedule too tough, plus they play a championship.

4. Florida State.

Overrated.

Football Season Already?

Awesome.

Tailgating starts tomorrow, I’m REALLY looking forward to this, I’ve missed tailgating. But enough about that, here is my prediction for the 2006 Hokie football season, go ahead and call Vegas:

Northeastern – Win
North Carolina – Win
Duke – Win
Cincinnati – Win
Georgia Tech – Win
Boston College – Win
Southern Miss – Win
Clemson – Lose
Miami – Lose
Kent State – Win
Wake Forest – Win
Virginia – Win

Gator Bowl here we come!

I spent zero time on research and analysis but without an experienced quarterback our defense and running game won’t be able to overcome Miami nor will it keep pace with a good Clemson team.

The sad part about the VT schedule is the mediocre opponents out of conference. This schedule is garbage and the AD should be ashamed of himself. The good part about a weak schedule is that everyone stays in good spirits after the game and makes the post-game tailgate better than the pre-game tailgate. I can’t wait!!!

Our first softball game

Tuesday night we played our first co-ed softball game. Our team consists of Webmail.us employees and our friends. It was Bill’s idea to start the team and we wanted to play so that we would have something to do together and have some fun. Of the 19 people on our roster, I’m pretty sure only 5 or 6 have ever played in a real softball game before Tuesday night. To say we are inexperienced would be a gross understatement. However, our team is a very competitive and eager to learn group of people. I have volunteered to coach the team to the best of my abilities.

We have been practicing for several weeks now and the improvement from week 1 to now has been dramatic. Our team is much better at all aspects of the game and they understand terms such as “cut-off”, “backup”, “calling for the ball”, etc.

So game time rolls around on a dreary and rainy Tuesday night at Tom’s Creek Park. As soon as the game before us finishes they are calling for lineups for our game. I decided to keep myself out of the lineup for several reasons, but mainly I wanted everyone to get a lot of game time experience. I hand our lineup in and we proceed to warming up. I’ve never really seen such a short warm up period but in what had to be under 5 minutes the game was starting. We were the away team so we batted first. We quickly went down 1-2-3 and were in the field for the first time.

For anyone who hasn’t been to a softball game lately the pace of the game is much, much quicker than the baseball you see on TV. Almost every pitch is hit and the bases are very short so doubles are very common. There also isn’t a lot of down time between pitches. So at the start the bottom of the 1^st inning their leadoff batter grounds out to second for an easy out. Cool, 1 down 2 to go. Unfortunately the next five hitters get hits and before you can spin around twice in your chair we are down five runs. We finally get the last out right before they bat around.

As the team comes off the field I can see the shell shock on everybody’s face. So I tell everyone to huddle up and listen. I pause for a second or two and everything is real quiet. What do you say in this situation? I decide to crack a joke and say “Welcome to Softball.” Everybody laughs and I think we all relaxed at that point and the rest of the game went fairly well. Like I told my team, if you take out the first inning and the 4 run inning they had because of a two out error we would have won 5-4. :)

As far as coaching goes I think I did an OK job. There will always be things I can improve on, but overall I really enjoyed it. My favorite parts were watching people backup each other, just like I told them to do time after time in practice. Watching people line up cutoffs and actually hitting those cutoffs, just like I taught them to do. And hitting fly ball after fly ball, just like I told them NOT to do. (We’ll keep working on it.)

New (to me) UNIX Commands

I often have the need to compare two files on different servers, now I can do so without having to scp the file and then run diff:

vimdiff scp://webmail1//home/kminnick/tmp.txt scp://webmail2//home/kminnick/tmp.txt

I also often have the need to take a file and split it into several small pieces:

man split

Query Optimization

One of the more enjoyable parts of my job is finding and eliminating bottlenecks in our system. Yesterday I was faced with a problem of a moderate load average problem on our RSS servers. I could tell that our system has been growing and that there must be a query that just wasn't performing as optimally as when we first launched. I know the RSS code fairly well, so I dove in and starting looking for queries that could possibly be the culprit. It didn't take long to find a query that when executed on the live system took over 5 seconds to finish (table names and columns changed):

SELECT s.id, s.u, s.d
FROM s, i
LEFT JOIN p ON s.u = p.u
AND s.d = p.d
AND p.pr = 'RSS_E'
WHERE (p.val IS NULL OR p.val != '0')
AND s.fId = i.fId
AND s.lDItem < i.rTime LIMIT 1;

OK, so I looked at the columns of the tables and two columns (lDItem and rTime) were not indexed. So I added indexes to both columns and ran the query again. This time it actually took 10 seconds to execute! As my 4 year old daughter would say, "What in the hecka in the world?!?" I've never really seen MySQL slow down after adding indexes, but I know the query optimizer logic is very complex, so I decided to help it out a bit by giving it a hint as to which index to use:

SELECT s.id, s.u, s.d
FROM s, i use index (rTime)
LEFT JOIN p ON s.u = p.u
AND s.d = p.d
AND p.pr = 'RSS_E'
WHERE (p.val IS NULL OR p.val != '0')
AND s.fId = i.fId
AND s.lDItem < i.rTime LIMIT 1;

Now the query only takes .48 seconds to execute! Excellent. I also dropped the lDItem index and the query speed stayed the same, so I only really needed the one new index and a simple little addition to the query to achieve a 10x improvement in the query. This little fixed has eliminated all of the CPU bottlenecks on the RSS servers today. This stuff makes me so happy. :)

The point is that query optimization is not always a matter of just adding an index (many times it is, but not always). I had to experiment several times before I got the results that I expected.

MySQL Chain Replication

This took me awhile to figure out, but here is the option you need to configure chain replication w/ MySQL. This is useful for geographically dispersed MySQL servers. From the MySQL documentation:

--log-slave-updates

Normally, a slave does not log to its own binary log any updates that are received from a master server. This option tells the slave to log the updates performed by its SQL thread to its own binary log. For this option to have any effect, the slave must also be started with the --log-bin option to enable binary logging. --log-slave-updates is used when you want to chain replication servers. For example, you might want to set up replication servers using this arrangement:

A -> B -> C

Here, A serves as the master for the slave B, and B serves as the master for the slave C. For this to work, B must be both a master and a slave. You must start both A and B with --log-bin to enable binary logging, and B with the --log-slave-updates option so that updates received from A are logged by B to its binary log.

Little League Dilemma

I've coached little league ( a long time ago), it was fun, competitive, and more than anything my kids all had fun win or lose (but we never lost). Which brings us to a story from Salt Lake that just blew my mind. Read the full details here, but here's a quick synopsis:

Salt Lake, little league championship. Last inning. The best hitter on the opposing team is up with two outs. The kid has already hit a homer and a triple. Any rational coach would think about intentionally walking the best hitter so that he doesn't hit another homer, makes perfect sense to me. One problem though, the kid that hits after him is a cancer survivor. So they walk the best hitter, and the cancer kids starts crying after he gets two strikes. Oh boy. What were they thinking? The kid strikes out, crying his eyes out. The opposing coach is furious.

So, if you were the coach would you have walked the best hitter to get to the kid w/ cancer? I'm not sure I'd be able to do much celebrating if I did that. Maybe the bigger question is what in the world is the cancer kid doing batting behind the best hitter. You don't put Mario Mendoza behind Babe Ruth...Idiot.

How many of you would play for the win in that situation?

Amazon S3 Part 2

OK, so I got a few complaints (you know who you are) about part 1 of my Amazon S3 post being too long and boring. So....I'll keep part 2 short and sweet. Basically, I wrote a backup / restore script for my linux files using PHP and REST (instead of SOAP). The bottom line is that I love the Amazon REST interface. It's simple and easy to use, forget SOAP, use REST.

The only other issue w/ S3 that I've noticed is that it will fail 1 out of every 2000 or so requests. Based on the forums, you simply have to code around this issue. It's easy to do, but I'm not sure why it fails so frequently. But I've never seen it fail twice in a row, so simply re-trying the request after a failure seems to always work.

Blacksox Turning It Around

My baseball team won this past Sunday 20-2 over a pretty good team. I was amazed. Overall we are only 4-8, in 5th place (8 teams total). But, we have improved dramatically over the last 3 weeks. I'm really looking forward to the remaining games. I'm hitting .310, which isn't bad, but it only is 6th best on the team. My brother is hitting .462 with 17 RBI.

I haven't picked up a golf club in a couple of months, I'm just too busy, so I'll probably wait until the fall to pick it up again.

Our company softball team has formed, I've had fun at the two practices I've been able to attend. It should be an interesting first year, I believe our games should be starting sometime soon.

Amazon S3 Part 1

Two days ago I began looking into Amazon's S3 web service. For those of you who don't know, S3 is described as follows by Amazon:

"Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers."

Hmm...web-scale computing made easy...as a geek there's no way I'm not checking this out, so I went ahead and signed up for an account. The signup process was really easy, I just had to break out my Discover card and they promised to bill me monthly, excellent deal. Once I was signed in I had access to all of the documentation and example code. The cost by the way are as follows:

Pricing
* Pay only for what you use. There is no minimum fee, and no start-up cost.
* $0.15 per GB-Month of storage used.
* $0.20 per GB of data transferred.

I figure I'll owe them a few pennies each month, no big deal. I really just want to play around and figure out what kind of cool stuff I could build if I had the time.

I downloaded the .NET C# SOAP example. Installed it, put in my super secret key, recompiled and kaboom, I was up and running. The example showed me how to program the core concepts with S3 via a command prompt. Here are the technical concepts a developer would need to understand:

1. Buckets

A bucket is simply a container for objects. Each user can have up to 100 buckets. This may sound low, but in reality you only need 1 bucket.

2. Objects

Objects are like files, but they have meta-data around them. Meta-data is data about the objects, key/value pairs. You also have to setup ACL's (Access Control Lists) for each object. You can have an unlimited number of objects. At first glance you may be wondering how to organize all of the objects in a bucket. What is really cool is that you can can use any type of delimeter you want to group objects. UNIX people are use to the "/" seperator, Windows uses the "\" seperator, but you can use whatever fits your application.

3. Keys

Every object has a unique key.

4. Operations

Example operations:

a. Create a bucket
b. Write an object
c. Read an object
d. Delete an object
e. List Keys

That's it. Pretty easy concepts to understand, but it's pretty powerful. So the example project showed me the basic concepts but I wanted to build something useful. So I decided to improve "My Internet Based File System" by creating a program that will allow me to

1. Upload a folder to S3
2. View a list of objects in my S3 account.
3. Download the object to my local hard disk.
4. Delete objects from S3.

After about an hour I had a working version. The hardest part was fixing their example code to handle binary files as well as text files. Once I got that it was just a matter of hammering out the code.

Here is a screenshot of the working version (and yes I do design work on the side):


The main problem with this code is that it reads the entire file into memory before sending the object to S3. Not a problem with small files, but if you ever wanted to upload a big mp3 or something it wouldn't work. In order to get this to work, you really need to "stream" the object to S3. But, doing this via SOAP is rather hard. The basic problem is that SOAP is primarily XML going back and forth. You would need to either dig deep and format the objects yourself (not recommended) or use a concept called "DIME Attachments". Good luck finding example code. And a bigger problem for me was that Microsoft switch their "Web Services Enhancements (WSE)" to use MTOM instead of DIME between versions 2.0 and 3.0. I don't really have the time to try to get this mess working. But I have some other ideas I'm going to play with first to see if I can get this working in a simpler manner, stay tuned.

If you are wondering how S3 works on the backend, the API docs give some clues, here is one:

"If the object already exists in the bucket, the new object overwrites the existing object. S3 orders all of the requests that it receives. It is possible that if you send two requests nearly simultaneously, we will receive them in a different order than they were sent. The last request received is the one which is stored in S3. Note that this means if multiple parties are simultaneously writing to the same object, they may all get a successful response even though only one of them wins in the end. This is because S3 is a distributed system and it may take a few seconds for one part of the system to realize that another part has received an object update. In this release of Amazon S3, there is no ability to lock an object for writing -- such functionality, if required, should be provided at the application layer."

Some helpful hints when using mysqldump

I always have to go to the mysql reference manual for this, so I'm going to blog about it so that I'm fewer clicks away and I don't have to re-read the mysql manual every 3 days.

First, mysqldump is useful for a variety of reasons, primarily backup purposes, but it can also be very useful for dumping a very small subset of data. We do this commonly to dump live data to test databases.

Here is an example:

Say you have a database called 'webmail' with a table called 'address' and you want to dump all of the address data for a particular user ('owner') from the live system to the test system to debug a problem. You would run mysqldump in the following manner:

mysqldump --skip-opt --quick --extended-insert --no-create-info --user=testing -p --where="owner = 'kevin@domain.com'" webmail address > tmp.txt

mysqldump by default uses the parameter "--opt" which means:
--add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset

Most likely you don't want most of those in this scenario, so that's what all of the options are about.

And to import the data run:

mysql --user=testuser -p webmail < tmp.txt

The most important tip is to make sure mysqldump doesn't add the "DROP TABLE" commands!

Perl

Sometimes perl is a little too forgiving, this little bug has existed for over two years in one of my programs and I just now spotted it...

if(length($hostname < 5)) {
#precaution
return 1;
}

You throw like a girl

Yesterday I tried to give Jil and Velvet some pointers on how to throw a softball. Both of them had issues turning their shoulders properly, which isn't a big deal if throwing small distances, but if you really want to throw like the boys you have to turn those shoulders. My other big tip is to get your elbow up. By the end of the practice they were both throwing much better.

I think girls can throw just as well as boys, but I think they don't get enough practice. You won't throw like a girl for very long if you keep practicing because it's just not going to go the distance you want nor have the accuracy you want until your mechanics are good. Eventually you body will train itself to put the arm, legs, and shoulders in the proper positions. So my biggest piece of advice is to keep practicing. Be sure to start slowly though, throwing a ball overhand is not a natural motion for the human arm. It is very easy to get a sore elbow or shoulder so be sure to stretch and warm up properly.

I'm looking forward to our softball season, but I hope it cools down soon, yesterday my Jeep told me the temperature was 105....

Photos From My Phone

It's been over a year since I cleaned up the photos on my phone, here are some good ones...

Colin Cowherd UVA Rant

If you haven't heard this clip and are a VT Hokie fan, check it out... UVA fans probably shouldn't listen.

Local News

Webmail.us announced today that we are now staffed in our customer care division 24 x 7 x 365. And yes, all of our customer care staff works from our Blacksburg, VA office. I'm very excited about this announcement because it is an important milestone for our company.

In somewhat related local news, it seems a lot of local communities are really pushing forward with providing wireless Internet access for all of their citizens. Check out this Roanoke Times article. I'm amazed that the smaller communities such as Bland, Hillsville, Radford, and Pulaski are actually leading the way in this area. At the same time I can't help but wonder how they plan on supporting their users.

I have a good friend who actually started a wireless ISP two years ago that primarily services the rural parts of Montgomery county plus some of Christiansburg. He's been very successful and I'm sure he's earned a lot of loyalty due to the excellent customer service he provides. I'm sure his business will continue to grow as long as he is able to continue providing great customer service.

More Debugging Tools

Here are a couple of cool debugging tools that will help you accomplish more in less time:

strace
------

strace allows you to attach to a process and watch every system call that the process executes. It is very helpful for tracking down exactly what a program is doing without having to use gdb. It is run like this (use the -f option to strace forks):

strace -tt -o log.txt -p

For an example run this:

strace -tt "uptime"

lsof
---
Another cool command is the "lsof" command. It will give you a list of open files (including sockets) for a process.

lsof -p 3387

Using GDB to Debug Linux Applications

I had to refresh my memory over the last two week on how to use the debugging application called 'gdb'. I have needed to step through both Pound and Dovecot's delivery applications to help me nail down a couple of tough problems. Debugging Dovecot's delivery application is very simple, so let's walk through it really quickly.

First, understand how the application is normally executed; normally the MTA will pipe in the email content to the deliver process, but we can fake it by using simple re-direction.

>deliver -d kevin@boebel.us < email.txt

But we want to step through the code, so we need to run it through gdb.

>gdb deliver

Before we issue the 'run' command, we need to set a breakpoint, so let's say we are trying to debug this portion of code in deliver.c (lines 540-546)

env_tz = getenv("TZ");
home = getenv("HOME");
env_clean();
if (env_tz != NULL)
env_put(t_strconcat("TZ=", env_tz, NULL));
if (home != NULL)
env_put(t_strconcat("HOME=", home, NULL));


(gdb) break deliver.c:540
Breakpoint 1 at 0x8056e72: file deliver.c, line 540.

Ok, the breakpoint is set, let's run the program, giving it the correct command line arguments:

(gdb) run -d kevin@boebel.us < email.txt
Starting program: /var/home/kminnick/dovecot/dovecot-lda-0.1/src/deliver -d kevin@boebel.us < email.txt

Breakpoint 1, main (argc=3, argv=0xbfffb3b4) at deliver.c:540
540 env_tz = getenv("TZ");

Cool, the program is now getting ready to execute this line of code, let's run some commands to get the value and type of 'home' variable both before and after the line is executed.

First, we need to skip this line of code and get to the next one:

(gdb) n
541 home = getenv("HOME");

Now we can examine the variable:

(gdb) pt home
type = const char *

(gdb) p home
$4 = 0x3f1cd8 "ä[\023"


As you can see, 'pt' prints the type of the variable, and 'p' prints the actual value. The variable is currently pointing to some garbage in memory. Let's execute the line of code and find the value of the variable after it's execution:

(gdb) n
542 env_clean();
(gdb) p home
$5 = 0xbfffdf44 "/root"

This was a very simple example of how to use gdb to debug your applications, next time we will discuss a more complex example on how to debug a program that uses forks and threads. I'm sure you can't wait... :)

Review of 2006 Goals

Today I benched 200 pounds one time. This was my goal for the entire year, so I am very happy with my progress thus far. Now I'm going for 225 by the end of the year.

Here are the goals I blogged about in January for this year and the results thus far:

1. Blog More
I'm definitely doing this, probably boring everyone a bit. :)

2. Go to the Doctor
Hmm...still six months left.

3. Get my weight to 175 lbs
I'm at 167 (I probably started the year around 163), only 8 pounds to go.

4. Oversee the creation the fastest webmail client on the market.
We are very close...

5. Become the best hitter on my baseball team.
We've only had 2 games so far, but our team captain is kicking butt. I'm hitting .375 with 2 RBI and 2 runs scored. That puts me tied for third (with my brother) on the team. We are 1-1 as a team. Mannin, our captain, is hitting .778 with a homer and 4 RBI. I have a long way to go, perhaps second best hitter on the team will be a good goal.

6. Shoot a round of golf under 110.
I've only played 3 rounds of golf this year. All at The Hill in Blacksburg. I have only had time to play 9 holes each time, but I think my scores have been fairly good, 51, 55, and 60. They are actually terrible scores, but I've made a lot of progress since last year. I think I definitely have a shot of getting a round under 110 if I ever find time to play a full 18 holes.

7. Take my family to Disney World.
Done, we had a great time.

8. Successfully manage our ever growing group of developers.
Ask Pat.

9. Bench Press 200 lbs one time
Today I did this, next up 225.

10. Find more time to spend alone with my wife.
Still working on this, but my favorite memory so far this year has to be the great time we had at Bill and Beth's wedding. We also had a great 5th year anniversary weekend. It's amazing how hard it is to find time alone when you have two kids. I'll keep working on it though.

Windows Vista Tips

Thanks to "Rick H." for posting this tips for Vista:

--------
Just a quick note to say that you can change the default search provider in IE7. Click the drop-down arrow at the edge of the search box in the upper right. Select "Find more providers" and see if your favorite is available (most are). Then click the drop-down again and choose Change Search Defaults.

For the admin popups (It's called UAP or UAC depending who you ask), run secpol.msc. Navigate to Local Policies->Security Options. Set UAC:Behavior of Elevation Prompt for Admins to "No Prompt" and UAC:Run All Admins in Approval Mode to disabled. (Use at your own risk!)
---------

I'm not sure how you figured out the UAC issue but what a life-saver that is for me. Thanks! Now if someone could figure out how to get my transparent windows back I'll be all set.

Windows Vista Beta 2 Review

After my computer decided to stop booting Windows XP Pro I decided to download and install Windows Vista Beta 2. It's been an interesting trial and error experience but I finally have it installed and running. I first had to ditch my old computer (Vista doesn't support the IDE driver that was built in 2001). I was able to install it on a brand new Dell, but I wasn't able to upgrade Windows XP Pro, I had to install from scratch. I'll probably be better off in the long run, I still hate how Dell insists on installing four million applications by default.

My first impression of Vista: "Cool Stuff". I like the visual changes they have made, I like the fading, the new transparency, the new start menu, the new explorer. I also like the gadget side bar, the new task bar, and the new task switcher. I also like IE 7, but there is no way to change the default search bar. It looks like it just hasn't been added yet, I am sure they will let me change it to Google eventually.

It comes with a few new apps like Windows Calendar and Windows Contact Manager, but they don't seem very useful. There is also some sort of collaboration app built-in, but that doesn't seem very useful either. Perhaps I need a few other users to help me test. The built-in search bar in the start menu seems to be a Google Desktop wannabe, but it appears to work rather well (I don't have a lot of data to test with yet).

I have only needed to install the following applications so far (they all work great w/ Vista):
Firefox
OpenOffice (Site doesn't work w/ IE 7 yet)
GAIM
Putty
Programmers Notepad
TurtoiseSVN
Thunderbird (for testing)

One really annoying "feature" that took two days to figure out is how to make my fonts look like crisp and clear. I was about to give up when I stumbled upon this setting:

Start->Control Panel->System and Maintenance->System->Advanced System Settings->Performance Settings->Click "Custom"->Uncheck "Smooth edges of screen fonts"

That setting has made all the difference in the world. Maybe today I can go home without a headache from my monitor.

Another issue is that I upgraded the video driver to the recommended version and I lost all of the cool transparency and window effects that I had on day one. :( I also haven't figured out how to turn off the annoying security pop-ups that open anytime I try to access an "Admin-Only" section of Windows.

I am finding the new control panel very complicated. It reminds me a lot of Perl in the fact that there are 50 ways to accomplish the same task. Navigating around the new control panel is frustrating, especially with the annoying "Admin Only, Are you really sure you want to do this?" pop-ups everywhere.

Overall, it looks like Microsoft has security as a top priority for this release. They are also finally catching up to Apple's cool windowing effects. And it won't take much more before they are caught up to Google on their desktop search capabilities. I can't think of any real compelling reason for anyone to upgrade besides the improved security features. Perhaps I'll find more in the coming days...

A Semi-Solution for My Internet Based File System Problem

After thinking about this more today, I have come up with the following solution that will almost work perfectly. First, we have an existing Subversion repository, so I created a "personal/kminnick" project for my own use. Second, we have an existing network drive called the "u" drive which can be used to store personal files. So, I installed a Windows based svn client called Tortoise SVN which is actually a shell extension that integrates directly into Windows Explorer. This basically means I can use Windows explorer to check out / check in files into Subversion, awesome stuff. I checked out my personal project onto our network drive (which is going to be available over a VPN eventually) so that nothing is stored on a local hard disk. To access the files from any location I can use a simple web browser to access the Subversion repository directly. If traveling, I could checkout all the files to my local hard disk, then check them back in whenever I have Internet access. Cool stuff. So with all of this setup and working I can:

1. All of my files are backed up automatically.
2. Have a history of every revision for all of my documents.
3. Access my files from any location that has a web browser.
4. Check out files locally for airplane trips, vacations, etc.

Not a bad solution, it was easy to setup and doesn't require a local hard disk for file storage.

Dead Computer

Today my desktop computer at work died. My dad gave me this computer about 2.5 years ago because he bought a new one, I'm guessing it was only about 2 years old when he gave it me. I was very attached to it, but all good things must come to an end. I was constantly asked by my co-workers why I never upgraded to a new machine. Well, for one, I never really “needed” a new machine. I spend most of my day inside of an SSH session, so memory, CPU, etc isn't that necessary. I'm curious though how many Word documents I'll need to re-type because I saved them to my local hard drive instead of on the network. I usually back everything up to the network or have them somewhere in my email for safekeeping, but I'm sure I missed a few. I use webmail exclusively for email, contacts, calendar, and tasks, so nothing will be lost there. I'm going to try to put in a new hard drive into my old machine and re-install Windows. But this time I'm going to try hard to move everything onto the network, the way it should be stored. In fact, I'm going to try to find a solution that will allow me to store everything I need onto the Internet.

This thought has really made the wheels turn this morning. I really need the following:

1. A web-based central repository that I can put / get files (any file type).

2. It should work like any good version control system (subversion), meaning it keeps track automatically of revision numbers, and I can view old revisions easily.

3. It would be great if I could open the files using “File, Open” from any application such as Open Office, Photoshop, etc. I shouldn't need to download the file first locally. I imagine this would require some sort of add-on to Windows Explorer. It needs to be fast though, so the plug-in should pull it down to a local temporary location, but automatically save it on the Internet when I click “File, Save”.

4. It should support sharing, so instead of having to email a document to everyone, I could simply email a note to everyone I want to share the document with and they can pull it down after entering an email / password. The link should always point to the latest revision so that timing isn't an issue. You should also be able to setup who can make modifications to the document.

5. You should have the ability to setup notifications so that you know when the document has been changed / modified.

6. It should be integrated inside of webmail.

For now, all I really need is #1. I think to solve that I'll setup a subversion server for myself and put all of my documents into subversion. I will be able to access it from home and work, the only downside is that I won't be able to open / save documents without the help of a subversion client. But at least everything will be backed up automatically and all of my documents will be accessible from the Internet.

If anyone knows of a solution that will meet the first five requirements above please let me know. I honestly haven't researched it yet.

Trac

Yesterday I setup and installed Trac for our development team. Trac is one of those tools that once a developer sees it, they gotta have it. If you haven’t heard of Trac, it allows developers to easily browse Subversion repositories. Subversion is the tool we use to track code changes. Subversion doesn’t have a good web-based GUI for browsing the repository, which is the primary reason we wanted to install Trac. Trac is easy to navigate, provides timelines, and makes browsing differences between revisions easy-easy-easy. The developers at Edgewall really have a winner with this product.

Trac also has a bug ticketing system, wiki, report generation tool, and milestone tracking tools built-in. The other tools need a bit more flushing out, especially the bug-tracker, but the SVN browser makes it worth the trouble to install. The installation process for Trac is…well…tough. I’ve installed hundreds of applications by source and RPM over the years and Trac was definitely one of the hardest I’ve ever encountered. Not only are there a ton of dependencies, but you have to be very careful that you have the right versions of each dependency. Thank goodness Google was able to help quite a bit.

I can see the benefits of Trac already paying off; just this morning, Mike sent me a link to the change-set in Trac for the next upload. Clicking on the link takes me to a web page with a description of the changes, a list of all files that were changed, and a color-coded side-by-side diff for each file….great stuff.

Thanks to Joel, one of our latest hires, for directing us to this application.

5th Wedding Anniversary

When I was in the basement today I saw a picture album from our wedding night and decided to reminiscence a little. First, it's amazing how much changes in five years. Second, it amazing how much stays the same. Velvet, Bill, Jason, and Velvet's parents all look the same today as they did five years ago. Pat, my parents, and my brother look totally different. I look like another person and wonder what Velvet was thinking...I'll bring the pictures to work if anyone is interested in looking.

To celebrate our anniversary we dined at The Summit restaurant in Christiansburg for the first time. Velvet suggested it and I heard three reviews all raving about the great food. I have to admit the food was really good (even though my steak came out a little too done). If you are looking to impress your date with some great food and a nice atmosphere I definitely recommend The Summit. :)

I'm more of a “look to the future” guy than “dwell on the past” guy; so I'll give you some insight on what will be going on five years from now, instead of what has transpired over the last five....In five years Drew will be in kindergarten and Jordan will be in 4^th grade. I'll be working on my 11^th year at Webmail.us. I'll only have 20 more years to pay on my house (haha). Maybe I'll have the driveway paved by then so I can put up a basketball goal for Jordan. Hopefully I'll still have my Jeep (gas will only be $6.00/gallon but that still beats having a car payment). Maybe my golf game will be in the 90s (OK, so I'm dreaming a little). Jordan won't remember a time when TV wasn't in high definition. Wireless Internet access will be available practically everywhere for free. Jordan will be using this laptop and think it's archaic. I might be caught up on the Sopranos. I hope Jordan can teach Drew how to play chess, read, and create his own blog. Jordan will still be in gymnastics (and will keep requesting a trip to Sonic after each session). Velvet and I will be celebrating our 10^th Wedding Anniversary.

OpenLDAP Errors

I ran across the following error in OpenLDAP two weeks ago when deleting an entry:

"Additional info: entry index delete failed"

Huh? We've been using OpenLDAP since 2003 and I've never seen this problem before, but looking at the OpenLDAP code showed it was trying to delete an index. We use the BDB backend and apparently a BDB index was corrupt. I haven't found an easy way to repair these indexes besides dumping all the data and re-importing. Maybe a BDB expert out there has some suggestions but luckily we replicate all of our data across multiple machines so I just had to copy the data (zip up the /var/openldap-data/ directory) from another machine. Googling for the error above didn't return many results so I thought I would go ahead and post my findings for the next person who searches for this error.

Harbinger Beta Available for Public Testing

A beta of the latest version of our webmail client is available to any current Webmail.us customer. Simply log in with your email address and password at http://beta.webmail.us.

You will notice only a few minor interface changes as our primary focus for this version is speed and performance. We no longer use frames and we make heavy use of AJAX and caching to improve speed. The first time you log in to the application will take a bit longer than normal because the browser is busy downloading JavaScript code that will be cached for future sessions.

Switching between folders, calendar, contacts, and tasks is significantly faster than previous versions of our application. We believe that a web based email client has significant advantages over a desktop based client but only if the web based client can match the speed and performance of the desktop client.

I've been using Harbinger for a few months now and have been really impressed with the speed thus far. We believe this is only the tip of the iceberg though and will continue to push the limits of the browser to maximize the speed of every aspect of the application.

Disney World Wrap Up

We had a great time at Disney World and I'm glad we were able to have a family vacation. I think that's the first family vacation we have taken since Drew was born. The kids did great on the plane flights, I'm so proud of them. Disney amazes me each time I go with how big they think. Nothing they do is too much, too big, too hard. They have big shows, big parks, big fireworks, and thus big crowds. They really do a great job and make you want to come back for more. We might head to Va Beach one weekend this summer, but otherwise I think we are done with family vacations for awhile. I'm glad I got to spend so much time with Drew, Jordan, and Velvet. If we do go back next year I'll have to remember these facts:

1. Go back to Magic Kingdom, Jordan loves it.
2. Don't go back to MGM Studios, it was fun, but not good enough to do two years in a row.
3. Make reservations at Cinderella Castle for dinner at least a month in advance.
4. Rent from a car rental place at the airport, not Hertz or Thrifty. (Hertz was great, but too hard to get on and off the bus.)
5. Caribe Royal is a great hotel, easy check-in, easy check-out, great pool, easy parking, but don't order the room service, very expensive and the food just wasn't that great.
6. Eat at Smokey Bones again.
7. Buy souvenirs at Wal-Mart in Orlando, nobody will know the difference and they are 1/10th the price.
8. Don't try to stay until close for the fireworks, the kids could probably make it, but I can't.
9. The "park hopper" option isn't worth it.
10. 10:30am is a perfect flight time, both ways. Connect in ATL at 1:30.
11. Mother's Day weekend is a good weekend to go because there is no baseball.
12. Jordan's favorite ride is the tram from the parking lot to the park, so ride it around a few times.
13. Jordan's favorite part of the trip is the plane ride and the pool, so take a trip to the beach with a Cinderella DVD and save some money. :)
14. Bring your own map of Orlando, the one from Hertz might as well been created by Jordan on the airplane ride down there.
15. Bring your own band-aid's so that you don't get weird looks from the first-aid people when Jordan gets a scratch on her leg.
16. Don't leave the diaper bag unattended next to the Haunted Mansion again, the next time someone might think it's a bomb and you don't want to be the one responsible for shutting down Disney World.
17. Invite some family so that you can go golfing while they take the kids to Disney World.

Disney World

Tomorrow I am packing up the wife and kids and heading to Orlando for our family vacation ... man I am old. :) Should be fun though and I'm sure Jordan will have a good time. We are heading back on Sunday.

Update on MySQL Issues

Last week I wrote about a strange issue with MySQL replication and what I thought I wouldn't see again for a long time. Well, around 1:00am this morning another MySQL server (a different cluster than previous) had the exact same issue as the first cluster. This time, my first step was to reboot the machine to see if it would fix the problem and thus not require me to spend a lot of time exporting and importing data. The reboot fixed the problem with the constant restart of the mysql server, but the replication thread failed to start. In fact, it was failing to start the slave thread on both machines. Issuing ‘start slave’ produced this in the error log of the machine with the problem:

060509 1:32:09 [Note] Slave I/O thread: connected to master 'replxxx@xxx-slave1:3306', replication started in log 'xxx-slave1-bin.004981' at position 79
060509 1:32:09 [ERROR] Error reading packet from server: Could not find first log file name in binary log index file (server_errno=1236)
060509 1:32:09 [ERROR] Got fatal error 1236: 'Could not find first log file name in binary log index file' from master when reading data from binary log
060509 1:32:09 [ERROR] Slave I/O thread exiting, read up to log 'xxx-slave1-bin.004981', position 79

This time I was determined to fix the problem without dumping data so I Googled a bit for the error and found this great post:

http://archives.neohapsis.com/archives/mysql/2004-q1/2000.html

Ah, so after creating the missing file and issuing a ‘reset slave’ the problem was fixed. On the other master I simply had to run a ‘change master to…’ command and ‘start slave’ and now replication was working in both directions again.

I think there is definitely a bug in the MySQL code somewhere, but I’m not sure what triggers it. I don’t think it’s related to the number of days a server has been up, but it might be related to the number of connections that have been opened since it was last restarted. It may just be a corrupt relay log file.

NCTC TechNite 7.0

Tonight the NewVA Corridor Technology Council (NCTC) is hosting TechNite 7.0. TechNite is self described as "the premier event honoring the NewVa Region's most promising technology companies, leaders, and entrepreneurs." Awards will be handed out in the following categories:

High Tech Leadership
Excellence in High Tech Manufacturing
Excellence in Life Sciences
Excellence in IT Development
Excellence in IT Deployment

Webmail.us is a finalist for the "NCTC Excellence in IT Development" award. Pat, our CEO, is a finalist for the "High Tech Leadership" award. It's quite an honor to be a finalist, especially considering the competition:

NCTC Excellence in IT Development Award
Webmail.us, Inc.
Phoenix Integration, Inc.
Synchrony Industrial Controls, Inc.
Integrated Imaging

NCTC High Tech Leadership Award
Margaret "Elaine" Carver, County of Roanoke
Kenneth Ferris, LUNA Innovations Incorporated
Patrick M. Matthews, Webmail.us, Inc.
Thomas A. Dingus, Virginia Tech Transportation Institute
R.J. Kirk, Third Security, LLC
Joseph W. Meredith, Virginia Tech Corporate Research Cent

Thanks go out to Trevor who played a big role in getting us nominated for these awards.

If you see Pat today be sure to wish him luck and ask him if he has that speech ready.

MySQL Replication and Heartbeat

Velvet, Marisa, Joel, Robert, and Andrew all did great in their respective 1/2 marathons and full marathons on Saturday. I certainly have a lot of respect for anyone who can endure all of the pain and agony that is involved. After we got back to the hotel I laid down for a power nap before dinner (yes, it was tiring watching all of those runners). I woke up to a phone alert letting me know a MySQL server was acting up. I log into the system and I'm seeing all sorts of errors and continuous notices that MySQL is restarting for every new connection. I've never seen this problem before, so I stop heartbeat so the other master takes over the IP. We run most of our sql servers in a master-master configuration. Master-Master means either server can immediately take over as the master if one goes down.

Thanks to heartbeat and MySQL replication, it allowed me to take my time and debug the problem without any service interruption. As for the debugging, I didn't get very far. I spent about an hour looking through log files, moving directories around (including to a ramdisk to make sure it wasn't a hard drive acting weird), etc. I was about to give the order to re-image the machine and start over when I had Bill reboot the machine because sometimes that magically fixes things. Well guess what, it did. I didn't check the uptime before the reboot, but according the other master it had been up about 284 consecutive days without a reboot. Not bad.

After the reboot fixed the restart problem, I decided to take a few safety precautions and assume the data was corrupt. So I moved the data to a backup copy, dumped everything from the master, re-imported, ran several test, and fired heartbeat back up. Everything looked good, and the whole process took about 4 or 5 hours. The restore took awhile because we have some tables that have over 9 million rows.

So my vacation in Nashville turned into a work weekend thanks to a MySQL server going crazy. Luckily we have the proper monitoring, replication, and failover systems in place, thank you Bill and team!

Big Weekend

This weekend I'm heading to Nashville via Knoxville to "watch" Velvet run in the Country Music Marathon and 1/2 Marathon (she's doing the 1/2). She's been training non-stop (I rarely awaken when her alarm goes off at 5:45am or whatever god-awful time she has it set to). I can't imagine running that far, but I can imagine that I will have fun this weekend. Our first stop is in Knoxville to go country line dancing at the world famous "Cotton Eyed Joe's". I haven't been there, but Jil says it's great so I can't wait. Based on the pictures from the website it looks like a grand ole time to me. And no, I won't be doing any dancing.

From there we head over to Nashville (first time for me) to check in at the hotel an get settled in before the big race. I'm sure Velvet will be nervous as hell (hehe). And with 20,000 participants I hope I can at least find her in the pack.

I wish Velvet, Jil, Marisa, and everyone else we are going with the best of luck...

Webmail Load Balancing Improvements

Last night we completed a “behind the scenes” project for webmail that will help us improve the scalability of our webmail server farm, improve our ability to measure the performance of our applications, and improve the overall speed up our applications. I started this project way back February and I previewed what we were doing in this post. By utilizing a piece of open source software called “Pound” we will be able to accomplish all of the goals listed above.

The primary reason for starting this project was our need to differentiate static web pages from dynamic web pages and send them to different clusters depending on the type of traffic. This would allow us to better cache the static content and get a better measurement on how well the dynamic pages are performing. We also wanted an easier way to manage all of our SSL certificates. Bill’s team (Tony and Korey) created a set of web pages that our support team can use to install and configure new SSL certificates. Mike also chipped in any necessary code changes that were need for webmail. And finally, Vinny helped me re-configure the control panel application for the new cluster of static servers.

I appreciate everyone’s help in making this project a complete success.

Tech Talk

I benched a personal best of 155lbs 10 times morning (which means I can pretty much bench my weight 8-10 times which is pretty good), but that's not what inspired me to post a blog...

Last night Me, Mike, and Pat gave a 'Tech Talk' on the Virginia Tech campus to a small group of computer science students. Basically we talked about Webmail.us, our development environment, our projects, the technologies we use, and we answered a lot of questions. I think it went well and we had a good time. One question that I did not get to answer was from a student who wanted to know what it was like to start your own company in Blacksburg because he was pondering the idea. Pat gave a great answer, but I couldn't help but think about this on the drive home.

My advice to anyone who is coming out of college and thinking about starting their own company is simple. As long as you are ready to put in long hours and are not afraid of failure, I think you should go for it. I think you will fail. But that's OK. Life is short, if you take that high paying job at IBM and Microsoft, that is the safe thing to do. It takes absolutely no courage to be that person. It will also prove to be unrewarding and boring in two years. Even if it's exciting work, you haven't learned the tough lessons that go along with starting your own company. You haven't learned how to fail, and you won't appreciate what it means to really work hard. If you are 22 and coming out of college, there is no better time to learn life's tough lessons. You have no kids, no mortgage, no obligations. It doesn't take a lot of money to start a company, you basically need an apartment and a work ethic. There is no better time than the present.

If you have an idea, a passion for that idea, and are not afraid of failing, GO FOR IT!!!

Of course, if you are afraid of failing, or you just don't have a great idea, I suggest working for a small company as a first step towards starting your own company. At a small company you will see and experience what the owners of the business are challenged by on a daily basis. You will see it's not easy. But this is great experience. After you work for a small company you will quickly realize that either that's the life you want to live or it's not. Perhaps you will find out that you do belong at a big company. Whatever the case may be, don't be afraid to take chances every once in awhile. That's my advice for all you graduates, good luck with whatever you do decide.

Product Development Releases Last 60 Days

Looks like I forgot to post this last month, but here are some of the great things we released over the last 60 days:

* Website: Printable Pages
* Control Panel: Improved Alias Management
* Control Panel: Simultaneous Logins
* Control Panel: Redesigned Login Page
* Website: Email Traffic analyzer
* Webmail: Address Book Import / Export Improvements
* API: Added 3 new functions
* Website: Redesigned Home Page
* Control Panel: Improved Availability
* Webmail: Spanish Language Improvements
* Webmail: Ability to select default language, and html font

There's plenty of stuff I can't mention too, but overall we are really busy around here and are constantly pumping out bug fixes, new features, even a beta. ;)