MySQL Replication and Heartbeat

Velvet, Marisa, Joel, Robert, and Andrew all did great in their respective 1/2 marathons and full marathons on Saturday. I certainly have a lot of respect for anyone who can endure all of the pain and agony that is involved. After we got back to the hotel I laid down for a power nap before dinner (yes, it was tiring watching all of those runners). I woke up to a phone alert letting me know a MySQL server was acting up. I log into the system and I'm seeing all sorts of errors and continuous notices that MySQL is restarting for every new connection. I've never seen this problem before, so I stop heartbeat so the other master takes over the IP. We run most of our sql servers in a master-master configuration. Master-Master means either server can immediately take over as the master if one goes down.

Thanks to heartbeat and MySQL replication, it allowed me to take my time and debug the problem without any service interruption. As for the debugging, I didn't get very far. I spent about an hour looking through log files, moving directories around (including to a ramdisk to make sure it wasn't a hard drive acting weird), etc. I was about to give the order to re-image the machine and start over when I had Bill reboot the machine because sometimes that magically fixes things. Well guess what, it did. I didn't check the uptime before the reboot, but according the other master it had been up about 284 consecutive days without a reboot. Not bad.

After the reboot fixed the restart problem, I decided to take a few safety precautions and assume the data was corrupt. So I moved the data to a backup copy, dumped everything from the master, re-imported, ran several test, and fired heartbeat back up. Everything looked good, and the whole process took about 4 or 5 hours. The restore took awhile because we have some tables that have over 9 million rows.

So my vacation in Nashville turned into a work weekend thanks to a MySQL server going crazy. Luckily we have the proper monitoring, replication, and failover systems in place, thank you Bill and team!

Big Weekend

This weekend I'm heading to Nashville via Knoxville to "watch" Velvet run in the Country Music Marathon and 1/2 Marathon (she's doing the 1/2). She's been training non-stop (I rarely awaken when her alarm goes off at 5:45am or whatever god-awful time she has it set to). I can't imagine running that far, but I can imagine that I will have fun this weekend. Our first stop is in Knoxville to go country line dancing at the world famous "Cotton Eyed Joe's". I haven't been there, but Jil says it's great so I can't wait. Based on the pictures from the website it looks like a grand ole time to me. And no, I won't be doing any dancing.

From there we head over to Nashville (first time for me) to check in at the hotel an get settled in before the big race. I'm sure Velvet will be nervous as hell (hehe). And with 20,000 participants I hope I can at least find her in the pack.

I wish Velvet, Jil, Marisa, and everyone else we are going with the best of luck...

Webmail Load Balancing Improvements

Last night we completed a “behind the scenes” project for webmail that will help us improve the scalability of our webmail server farm, improve our ability to measure the performance of our applications, and improve the overall speed up our applications. I started this project way back February and I previewed what we were doing in this post. By utilizing a piece of open source software called “Pound” we will be able to accomplish all of the goals listed above.

The primary reason for starting this project was our need to differentiate static web pages from dynamic web pages and send them to different clusters depending on the type of traffic. This would allow us to better cache the static content and get a better measurement on how well the dynamic pages are performing. We also wanted an easier way to manage all of our SSL certificates. Bill’s team (Tony and Korey) created a set of web pages that our support team can use to install and configure new SSL certificates. Mike also chipped in any necessary code changes that were need for webmail. And finally, Vinny helped me re-configure the control panel application for the new cluster of static servers.

I appreciate everyone’s help in making this project a complete success.

Tech Talk

I benched a personal best of 155lbs 10 times morning (which means I can pretty much bench my weight 8-10 times which is pretty good), but that's not what inspired me to post a blog...

Last night Me, Mike, and Pat gave a 'Tech Talk' on the Virginia Tech campus to a small group of computer science students. Basically we talked about Webmail.us, our development environment, our projects, the technologies we use, and we answered a lot of questions. I think it went well and we had a good time. One question that I did not get to answer was from a student who wanted to know what it was like to start your own company in Blacksburg because he was pondering the idea. Pat gave a great answer, but I couldn't help but think about this on the drive home.

My advice to anyone who is coming out of college and thinking about starting their own company is simple. As long as you are ready to put in long hours and are not afraid of failure, I think you should go for it. I think you will fail. But that's OK. Life is short, if you take that high paying job at IBM and Microsoft, that is the safe thing to do. It takes absolutely no courage to be that person. It will also prove to be unrewarding and boring in two years. Even if it's exciting work, you haven't learned the tough lessons that go along with starting your own company. You haven't learned how to fail, and you won't appreciate what it means to really work hard. If you are 22 and coming out of college, there is no better time to learn life's tough lessons. You have no kids, no mortgage, no obligations. It doesn't take a lot of money to start a company, you basically need an apartment and a work ethic. There is no better time than the present.

If you have an idea, a passion for that idea, and are not afraid of failing, GO FOR IT!!!

Of course, if you are afraid of failing, or you just don't have a great idea, I suggest working for a small company as a first step towards starting your own company. At a small company you will see and experience what the owners of the business are challenged by on a daily basis. You will see it's not easy. But this is great experience. After you work for a small company you will quickly realize that either that's the life you want to live or it's not. Perhaps you will find out that you do belong at a big company. Whatever the case may be, don't be afraid to take chances every once in awhile. That's my advice for all you graduates, good luck with whatever you do decide.

Product Development Releases Last 60 Days

Looks like I forgot to post this last month, but here are some of the great things we released over the last 60 days:

* Website: Printable Pages
* Control Panel: Improved Alias Management
* Control Panel: Simultaneous Logins
* Control Panel: Redesigned Login Page
* Website: Email Traffic analyzer
* Webmail: Address Book Import / Export Improvements
* API: Added 3 new functions
* Website: Redesigned Home Page
* Control Panel: Improved Availability
* Webmail: Spanish Language Improvements
* Webmail: Ability to select default language, and html font

There's plenty of stuff I can't mention too, but overall we are really busy around here and are constantly pumping out bug fixes, new features, even a beta. ;)