Tech Talk

I benched a personal best of 155lbs 10 times morning (which means I can pretty much bench my weight 8-10 times which is pretty good), but that's not what inspired me to post a blog...

Last night Me, Mike, and Pat gave a 'Tech Talk' on the Virginia Tech campus to a small group of computer science students. Basically we talked about Webmail.us, our development environment, our projects, the technologies we use, and we answered a lot of questions. I think it went well and we had a good time. One question that I did not get to answer was from a student who wanted to know what it was like to start your own company in Blacksburg because he was pondering the idea. Pat gave a great answer, but I couldn't help but think about this on the drive home.

My advice to anyone who is coming out of college and thinking about starting their own company is simple. As long as you are ready to put in long hours and are not afraid of failure, I think you should go for it. I think you will fail. But that's OK. Life is short, if you take that high paying job at IBM and Microsoft, that is the safe thing to do. It takes absolutely no courage to be that person. It will also prove to be unrewarding and boring in two years. Even if it's exciting work, you haven't learned the tough lessons that go along with starting your own company. You haven't learned how to fail, and you won't appreciate what it means to really work hard. If you are 22 and coming out of college, there is no better time to learn life's tough lessons. You have no kids, no mortgage, no obligations. It doesn't take a lot of money to start a company, you basically need an apartment and a work ethic. There is no better time than the present.

If you have an idea, a passion for that idea, and are not afraid of failing, GO FOR IT!!!

Of course, if you are afraid of failing, or you just don't have a great idea, I suggest working for a small company as a first step towards starting your own company. At a small company you will see and experience what the owners of the business are challenged by on a daily basis. You will see it's not easy. But this is great experience. After you work for a small company you will quickly realize that either that's the life you want to live or it's not. Perhaps you will find out that you do belong at a big company. Whatever the case may be, don't be afraid to take chances every once in awhile. That's my advice for all you graduates, good luck with whatever you do decide.

Product Development Releases Last 60 Days

Looks like I forgot to post this last month, but here are some of the great things we released over the last 60 days:

* Website: Printable Pages
* Control Panel: Improved Alias Management
* Control Panel: Simultaneous Logins
* Control Panel: Redesigned Login Page
* Website: Email Traffic analyzer
* Webmail: Address Book Import / Export Improvements
* API: Added 3 new functions
* Website: Redesigned Home Page
* Control Panel: Improved Availability
* Webmail: Spanish Language Improvements
* Webmail: Ability to select default language, and html font

There's plenty of stuff I can't mention too, but overall we are really busy around here and are constantly pumping out bug fixes, new features, even a beta. ;)

March Bench Press Results

Technically I have another week in March, but I don't think next week will be better than today, so I'll go ahead and post this month's progress:

185 x 3
195 x 1 (2 with a spot)

My bully spotter made me try 205, but I needed a little help from him with that, maybe next month. I definitely should be able to hit my goal of 200 next month, so I might up the ante to 225 by the end of the year. The best part of all of this is that it's been the same two people in there spotting me every Friday since I started in January and they were saying today they can really tell a difference. Hopefully the progress will continue...oh, and I weighed in at 165 today.

Google is Yahoo 2.0

Jeremy posted this awhile back, but with the recent launch of Google finance, I totally see what he is talking about. I really enjoyed Jeremy's latest comments on how Yahoo! had many of these same ideas years ago, but due to many issues they were never able to act on them.

I think Yahoo! and Microsoft are very much in the same boat on this one. All of the competition from open-source in regards to the browser, the desktop, the office application suite, the server market, and the software as a service model is very similar to the Google vs. Yahoo! fight.

I certainly hope that our commitment to keeping our product teams small and focused will prevent our software from becoming stale. I want our teams to be able to release high quality software and then immediately be able to turn around and say "How can we make this even better?"

Congratulations Bill & Beth

Last weekend Bill & Beth were married in Alexandria, VA. I have known both of them it seems like forever and they are really great friends. They certainly know how to throw a great party too and their wedding was no exception. I think everyone had a great time and I was glad so many of our friends were able to attend.

So who's the next one in our company to tie the knot? Manny? Pat? Here are my odds:

Manny 3:2
Brian 11:5 (Vegas is only a plane-ride away!!!)
Pat 2:1
Mike 4:1
Steve 5:1
Huey 8:1
Matt 10:1
Ben 10:1
Vinny 12:1
A part timer 20:1


Any takers?

My Evaluation of Reverse Proxy Software

A proxy server takes outbound traffic and makes it appear as if it comes from a single IP address. This type of proxy is useful if you have a single access point to the Internet and want multiple computers behind that access point to be able to access the Internet. This is common in an office or corporate environment.

A reverse proxy takes incoming traffic and sends it to multiple backend computers. This is useful if you run a large website and want a single IP address to actually be served by multiple machines. You can also accomplish this with a load balancer, but load balancers typically lack several key features that a reverse proxy can provide.

Let's take a look at 4 reverse proxies that are currently available:

1. Apache's httpd
2. Squid
3. Perlbal
4. Pound
5. HAProxy

All of these are very good, but depending on your requirements some may not meet your needs. I'm interested in a reverse proxy that can handle the following requirements:

1. Easy to configure
2. Small memory footprint
3. Provides load balancing
4. Supports SSL to client, but non-ssl to the backend
5. Provides High Availability
6. Can proxy traffic based on URL pattern matching
7. Supports virtual hosts

Perlbal is really a terrific piece of software, but it has a lot of features that I don't need, plus it does not have one key feature that I do need, SSL. Although there is an option to configure Perlbal with SSL, don't do it. It's too easy to break because of this problem reported on their listserve. It appears this is not a problem with the Perlbal code, but a problem with the underlying libraries that Perlbal uses. I could stick stunnel in front of Perlbal and I think that would work great, but you have to setup one instance of stunnel per SSL site and that wouldn't work for me. Perlbal uses epoll, which would provide a big performance boost. Perbal doesn't support sticking users to backends (for session support), but that is not a requirement for me.

Squid has been around for a long time, but I think it's too bulky. First, the configuration is too hard. Second, it's primary goal is to serve as a proxy, but I do not need this, just the reverse proxy. Squid also does a lot of caching, I don't need this either, I would prefer to have something that never touches disk. Also, to my knowledge, Squid does not support high availability (checks to make sure a backend is not down).

Apache's latest release (2.2) of it's httpd server does have better support for load balancing and high availability than previous versions, but it's still too bulky for me. One thing that kills me about Apache is how hard it is to configure mod_rewrite. I actually was able to setup Apache 2.2 as a reverse proxy and it was working perfectly, but I realized that this was going to be too hard to maintain on a large scale system. It's great that Apache is so configurable and can handle just about any task, but that can also be looked at as a big negative too.

Bill showed me HAProxy and it looks great, very configurable. But it requires Apache for SSL support, and I don't think you can proxy traffic based on URL.

The final piece of software that I looked at as a possible reverse proxy that would meet all of my requirements was Pound. I had a little trouble compiling Pound at first on RHES3, but I finally figured out the right compiler flags. On RHES4 it compiled with no problems. Besides that Pound seems to work great. It's easily configurable, is designed to accomplish only one task, and is small in size.

So there you have it, my evaluation of reverse proxy software. If I missed anything or I am just dead wrong, feel free to leave a comment. I'll update this post if anything is inaccurate.

February Bench Press Results

Another month has come and gone and I know you are dying to know how much I benched pressed today, so here ya go:

155 x 8 times
175 x 3 times
185 x 2 times
190 x 1 time

190! Holy cow, only 10 more pounds to reach my year end goal! Not bad considering I weighed in at 163lbs today. I haven't gained or lost any weight since I started working out, that might be more amazing than the 190lbs I lifted today.

Cryptic Apache Error

Each night our webmail servers automatically restart httpd to clean up memory and start fresh. Last night one of our servers hit this error upon restart:

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed

I had never seen this error before and had no idea what it meant, but Google helped me out:

http://www.goldfisch.at/knowledge/224

After some testing, I found that Apache leaks a couple of semaphores after each restart and eventually w/o a server reboot will run out of resources. So I changed our restart script to clean up these semaphores each night.

ipcs -s grep nobody perl -e 'while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

In case you are wondering, it's not a big deal if a webmail server doesn't restart properly. Our load balancers quickly take a down server out of rotation so that customers are not affected. Thank you keepalived.

Development Releases Last 30 Days

Over the last 30 days we have released some really great additions to our platform. Here is the high level list I presented at our management meeting on Monday:

* Webmail Lite w/ Calendar / AJAX
* Control Panel Security Additions (IP Based Restrictions)
* 20+ Webmail Bug Fixes
* Webmail Import / Export Enhancements
* Company Directory
* Website Search
* 30 Day Free Trial

You can read the details on our blog.

OpenLDAP Replication

Today I setup a new LDAP slave machine, the process is fairly straight-forward, but a good thing to know about replication is how to do it without shutting down the master for an extended period of time. Let's take a look at how to do this:

First, let's verify we have a few things:

1. The LDAP ssl port is open from the master to the slave:
ldap-master> telnet ldap-slave 636

2. Make sure the slave has a valid SSL certificate. If you are using a self-signed cert you will need to make sure the master has the slave .crt file installed. If you have multiple slaves, you need to create a combined.crt file and concat all of your certs into one file. Then add an entry to the master ldap.conf (not slapd.conf) like so:

TLS_CACERT /usr/etc/openldap/certs/combined.crt

OK, so now we need to get all of the data from the master to the slave without losing anything during the transfer. Normally this would require the master to be taken offline for some period of time while you zip up the data files and scp them over. But, luckily I have another slave that is not being used for queries, so I can follow these steps to get the data moved without having to shut down the master for an extended period of time:

0. Stop the new slave.
1. Add the necessary

replica uri=ldaps...

lines to the master slapd.conf file.

2. Restart the master.
3. Stop the old slave.
4. Tar/Zip the data, scp to new slave.
5. Start the new slave, verify replication works to the new slave.
6. Bring back online the old slave, verify replication is still working.

That's it, really simply. One trick you should know about replication is that if you have multiple LDAP databases being replicated, slurpd will not replicate the second database to a machine with the same hostname as the first database. I have to create a fake host entry in /etc/hosts so that slurpd doesn't get confused. It took me awhile to figure this one out.

Bench Press Update

I've been working out consistently this month, pretty amazing, but I found that going in the morning is the only way I can do it. Today was the last chest workout for the month, so I thought I would log my progress here.

Today I did:
135 x 12 reps
155 x 6 reps
165 x 3 reps

165 is a long ways from 200, but I think I made good progress this first month. I started the month doing 115 x 12 times.

New Webmail Lite and Control Panel Features

We just finished releasing the next version of Webmail Lite. This version has a calendar, task list, and uses AJAX for speed improvements.

We also added a feature to our control panel that allows administrators to use IP addresses to restrict logins.

Read all about these these enhancements on our company blog.

2006 Goals

It's that time of year for resolution and goal setting, so here is my list for 2006 (in no particular order):

1. Blog More
I'd like to use this blog to record interesting things in my personal life and to document any research or technical details about the work I do at Webmail.us.

2. Go to the Doctor
Since we have great health coverage at Webmail.us, I better start using it. My dad has had a lot of health problems recently and let's just say it's been a wake up call for me.

3. Get my weight to 175 lbs
I think it's a perfect weight for me, and this goal will force me to attend the gym regularly.

4. Oversee the creation the fastest webmail client on the market.
Speed, speed, and more speed. We are working hard on this problem and it is the focus of our development efforts this year. I'm relying on the great developers we have to help me accomplish this goal.

5. Become the best hitter on my baseball team.
The past two years I have been the second best hitter on my team. This year I really want to take over that top spot.

6. Shoot a round of golf under 110.
I would say under 100, but that might be a bit unrealistic since I never have time to practice.

7. Take my family to Disney World.
Velvet and I work so much that we rarely take family vacations, so I would like to take the kids to Disney World this year. I know Jordan will love it almost as much as Velvet. :)

8. Successfully manage our ever growing group of developers.
I think this will be my biggest challenge this year. It's also the challenge I'm looking forward to the most.

9. Bench Press 200 lbs one time
I use to be able to do this, then I stopped working out. I'm probably at 165 right now (I use to do this 8 times w/o problems), but I honestly haven't tried a low number of reps in a long time.

10. Find more time to spend alone with my wife.
For obvious reasons... ;)

Understanding memcached memory management

memcached is a very fast program that can be used to cache data stored in a database for quick access and dramatically reduce the load on your database servers. It's one of those "must haves" if you develop a highly trafficked website that has a database backend. Understanding the internals of memcached may be a bit confusing at first, the details aren't documented well, and there are some tools that aren't that easy to understand what they do. I'll try to explain here by example.

Let's say you have a memcached setup and have allocated 256MB of RAM to the program. memcached works hard to avoid memory fragmentation, it does this by taking the memory and dividing it into classes, numbered 6 through 17 (don't ask). As you insert into memcached, it will take the size of the data you are inserting and inserts it into the class that is big enough to handle the data. See the chart below. The classes in the below memcached are of size 128B - 16KB. Ignore the Max_age column for now. Each class that has data is allocated at least one 1MB page. So, class 7 has 4 1MB pages, each data item in that class is using 128B and the class is full, so there are 32767 items in the class. (If you insert an item into memcached of say 100B, the class uses all 128B to avoid fragmentation.)

Now if we try adding to this cache, we won't necessarily get an "out of memory" error because memcached uses a LRU (least recently used) algorithm to kick out the item from the page that has been accessed the least. But, if we attempt to insert an item of say 15B, we will get an "out of memory" error because class 6 (the class would be of size 64B) has no items, and thus no pages have been assigned. But, all pages are currently being used, and memcached cannot allocate a 1MB page to class 6. In order to make this happen we would have to move a page from another class to class 6.

Let's examine how this is done:

First, let's get a snapshot of our current allocations:

[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 B 0 s 0 yes
7 128 B 742942 s 4 yes
8 256 B1175448 s 2 yes
9 512 B1279471 s 1 yes
10 1 kB6493931 s 1 yes
11 2 kB 434740 s 71 no
12 4 kB 917212 s 140 yes
13 8 kB 794005 s 30 no
14 16 kB1373021 s 7 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes

Now, let's go ahead and try to insert 2 bytes of data:

[root@sql-slave1 bin]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.
set key1 0 0 2
SERVER_ERROR out of memory

OK, just like I thought, memcached can't allocate a 1MB page, so let's help it out:

[root@sql-slave1 bin]# ./memcached-tool localhost move 14 6
Success.

Now, let's look at our pages again, as you can see 1 page has been moved from class 14 to class 6. The LRU items in class 14 go bye-bye.

[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 64 B 0 s 1 no
7 128 B 743011 s 4 yes
8 256 B1174812 s 2 yes
9 512 B1279618 s 1 yes
10 1 kB6494078 s 1 yes
11 2 kB 434887 s 71 no
12 4 kB 917208 s 140 yes
13 8 kB 794152 s 30 no
14 16 kB1373168 s 6 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes

Let's insert our data again:

[root@sql-slave1 bin]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.
set key1 0 0 2
aa
STORED
quit
Connection closed by foreign host.

Cool, it worked. Looking at our stats shows the item is aging:

[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 64 B 14 s 1 no
7 128 B 742970 s 4 yes
8 256 B1174852 s 2 yes
9 512 B1279658 s 1 yes
10 1 kB6494118 s 1 yes
11 2 kB 434918 s 71 yes
12 4 kB 917228 s 140 yes
13 8 kB 794192 s 30 no
14 16 kB1373208 s 6 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes

Conclusions, you better keep an eye on the data in your memcached, and knowing what type of data you are storing will help you maintain it. Writing scripts to automate the moving around of pages based on data should be fairly easy.

Also, there are some little known commands that I use a lot:
stats
stats items
stats slabs
stats sizes

memcached is a great piece of software, many thanks to the Danga team.

Newsworthy

Looks like my wish came true:

http://www.hokiesports.com/football/recaps/2006016aaa.html

and Google has bundled some common software and made it easy to download and install, I haven't tried it, but it looks pretty useful:

http://pack.google.com/

Maybe I'll wait for them to release their operating system. I'm very skeptical this will happen anytime soon. It would have to be built on some sort of Unix flavor and the bottom line is that most of the applications they have built don't have Unix versions. Of course, it is Google, so I'm sure it's on their drawing board somewhere.

529 Savings Plans

I just finished signing Andrew up for a 529 savings plan. For those of you who don't know, 529 plans are a great way to save for college. They invest in stocks, bonds, etc, just like normal mutual funds, but the money has to be used for college. The funds have to be sponsored by a state, so each state has a fund. My financial advisor, Clark Howard, has some great tips about 529 plans, including his "Dean's List" and "Honor Roll List" of states with the best plans. One piece of advice that I don't listen to, even though I agree with is

"But first remember my rule that you don't save a penny for college unless you are already saving the maximum you can for your own retirement. College can be paid for with grants, loans, scholarships and work. Retirement happens only if you have saved the dough."

I don't think the whole "saving the maximum" thing will happen until Jordan is about 33, a little too late to start saving for college. I guess I'll just have to work a few more years, but I like working so no big deal. When I enrolled Jordan, I picked the Missouri state plan because it was on his Honor Roll list ( I don't think he had a Dean's List 4 years ago) and you could sign up for the plan online. This time around I wanted to chose one of his Dean's List plans so I went with the New York plan. The plan is managed by Vangaurd and you can sign up online.

Both the Missouri and New York plans let you invest in "age based" options, meaning the plans become less aggressive as the beneficiary gets older. This is exactly what I want plus it requires zero maintenance on my end, a big-time bonus because I hardly have time to pay attention to it.
I looked into the Virginia 529 plan too (my home state), but their so-called website looks like something a 2 year old put together with Microsoft Frontpage '98.

Va Tech Football 2005 Year in Review

The Virginia Tech football season has come to end and it certainly was a season filled with ups and downs. I enjoy attending the games and have seen the football program grow from being just awful to exceptionally good year after year. Becoming a winning program does, apparently, come at a cost. If I had one wish for next year's team, I wish that Frank Beamer would get some guts and bench players that cause more trouble than what they are worth. Some players just behave in a manner that does not reflect well on my alma-mater. I guess discipline and sportsmanship are far less important than winning. I suspect Frank has so much riding on winning these days that he can't afford to bench players that are obviously a poor reflection on the entire program. I completely understand his position, and our program is certainly not the first program to have such issues.

Perhaps Va Tech can start offering a new class for all incoming recruits "How not to look like an immature spoiled kid on national television 101". I guess if you want a squeaky clean program, attend Vanderbilt...Which bowl did they go to this year? And when is my season ticket application due? :(

Update: Since writing this post (I saved it as a draft yesterday), the athletic department at Va Tech released this statement.

Java Schools

Joel has a great blog post about "Java Schools", it's a good read and I agree with every bit of it. The test we give programmers on their first interview has questions regarding both pointers and recursion. They are two of my favorite questions, most people think they are too easy, but the fact is if you solve the recursion problem using a "while loop" it tells me a lot. If you solve it using binary operations, you're hired. ;)

I don't use a test as the determining factor for any hire, but I like to see how people go about solving problems. Some struggle, some work fast, some love it, some hate it, some even give up and leave without even trying. If you can't handle 6 simple problem solving questions though, chances are you can't handle developing software applications that are critical to the success of tens of thousands of businesses.

MySQL Replication

This morning I setup MySQL replication for two of our user / spam databases that we use internally. This is about the 10th time I have done this, so I don't even have to look up the instructions anymore, but I have to admit that the MySQL team has made the replication process ever so easy. We setup our databases in a master-master configuration with multiple slaves. Slaves can span geographic locations, for instance, from Dulles, Va to Blacksburg, Va, some 300+ miles apart. It's a very nice setup and I always have so much fun setting it up, I don't know why, but for some reason that type of work is fun for me, especially when it works the very first time I try it (and the last two times it has worked right off the bat). Thinking back to the first time I setup MySQL replication it took a few tries to get it right, and so if you are having trouble getting it to work, here are a few suggestions:

1. Read High Performance MySQL by Jeremy Zawodny
2. Turn on logging (vi /etc/my.cnf, log=/var/lib/mysql/log.txt)
3. Make sure every server has a unique id (server-id = 1)
4. I had a pound symbol (#) as part of the password, for some reason this didn't work.
5. Make sure you can telnet to the mysql port on each machine.
6. Make sure the username/passwords are correct.
7. The "show slave status" and "show master status" commands are very helpful.
8. Only replicate the databases you need, make sure there is not a lot of unnecessary traffic.
9. Testing is easy to do, so do it.
10. The /etc/my.cnf file is not used after the first reload of mysql. You need to run the 'change master to' command from that point forward.

Jeremy's book was very helpful, so I highly recommend it for the first-timer.

The best tip I can give is to setup a slave server that does nothing but replicate from the master. This is important because some day you will want to setup another slave without bringing down a live master. This is nearly impossible if you are using all of your machines for queries. To setup a new slave without any downtime, do the following:

On the slave that is doing nothing but replication:

1. Run 'stop slave' from mysql prompt
2. Run 'show slave status\G' from the mysql prompt, record the following two lines:
Master_Log_File: rss-master1-bin.000082
Read_Master_Log_Pos: 993448778
3. Run a "mysqldump" to get all of the data to a text file.

On the brand new slave, do the following:
1. Setup the databases, users, and import the exported data.
2. Run the following command from mysql

CHANGE MASTER TO
MASTER_HOST='master-server',
MASTER_USER='repl',
MASTER_PASSWORD='xxxxxxxx',
MASTER_LOG_FILE='master-server-bin.000082',
MASTER_LOG_POS=934159359;

3. Run 'start slave' from mysql (on both slaves).
4. If you have logging turned on, you will see all of the queries coming through since you stopped the slave process on the first slave machine.

That's it, it should work fine, but testing is encouraged. :)

Webmail.us 2005 Development Year in Review

This past year, thanks to the hard work of our entire team, we were able to launch many improvements to our email hosting infrastructure. Below is a partial list of what we accomplished over the last year:

An improved billing system for our accounting team
Faster, more reliable statistics for our customers
Control Panel 2.0
Control Panel API 1.5, 1.6
Webmail API 1.0, 1.1
Centralized Logging System
Revamped Webmail.us Website
Webmail Lite
An Auto-Configure Tool for POP3 clients
An Outages Alert System for our customers
Improved Migration Process
Improved Anti-Fraud System
Webmail 3.1
Webmail w/ RSS Reader
Webmail w/ AJAX, Calendar, and Company Directory
Webmail Help Section

This list doesn't include all the work required to help get the new infrastructure up and running and any custom development work we did for our customers.

We were able to accomplish so much because of our great team. I thank everyone involved in the process.