Thursday, February 23, 2006
February Bench Press Results
155 x 8 times
175 x 3 times
185 x 2 times
190 x 1 time
190! Holy cow, only 10 more pounds to reach my year end goal! Not bad considering I weighed in at 163lbs today. I haven't gained or lost any weight since I started working out, that might be more amazing than the 190lbs I lifted today.
Wednesday, February 22, 2006
Cryptic Apache Error
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed
I had never seen this error before and had no idea what it meant, but Google helped me out:
http://www.goldfisch.at/knowledge/224
After some testing, I found that Apache leaks a couple of semaphores after each restart and eventually w/o a server reboot will run out of resources. So I changed our restart script to clean up these semaphores each night.
ipcs -s grep nobody perl -e 'while (
In case you are wondering, it's not a big deal if a webmail server doesn't restart properly. Our load balancers quickly take a down server out of rotation so that customers are not affected. Thank you keepalived.
Wednesday, February 15, 2006
Development Releases Last 30 Days
* Webmail Lite w/ Calendar / AJAX
* Control Panel Security Additions (IP Based Restrictions)
* 20+ Webmail Bug Fixes
* Webmail Import / Export Enhancements
* Company Directory
* Website Search
* 30 Day Free Trial
You can read the details on our blog.
Saturday, February 04, 2006
OpenLDAP Replication
First, let's verify we have a few things:
1. The LDAP ssl port is open from the master to the slave:
ldap-master> telnet ldap-slave 636
2. Make sure the slave has a valid SSL certificate. If you are using a self-signed cert you will need to make sure the master has the slave .crt file installed. If you have multiple slaves, you need to create a combined.crt file and concat all of your certs into one file. Then add an entry to the master ldap.conf (not slapd.conf) like so:
TLS_CACERT /usr/etc/openldap/certs/combined.crt
OK, so now we need to get all of the data from the master to the slave without losing anything during the transfer. Normally this would require the master to be taken offline for some period of time while you zip up the data files and scp them over. But, luckily I have another slave that is not being used for queries, so I can follow these steps to get the data moved without having to shut down the master for an extended period of time:
0. Stop the new slave.
1. Add the necessary
replica uri=ldaps...
lines to the master slapd.conf file.
2. Restart the master.
3. Stop the old slave.
4. Tar/Zip the data, scp to new slave.
5. Start the new slave, verify replication works to the new slave.
6. Bring back online the old slave, verify replication is still working.
That's it, really simply. One trick you should know about replication is that if you have multiple LDAP databases being replicated, slurpd will not replicate the second database to a machine with the same hostname as the first database. I have to create a fake host entry in /etc/hosts so that slurpd doesn't get confused. It took me awhile to figure this one out.
Wednesday, January 25, 2006
Bench Press Update
Today I did:
135 x 12 reps
155 x 6 reps
165 x 3 reps
165 is a long ways from 200, but I think I made good progress this first month. I started the month doing 115 x 12 times.
Tuesday, January 24, 2006
New Webmail Lite and Control Panel Features
We also added a feature to our control panel that allows administrators to use IP addresses to restrict logins.
Read all about these these enhancements on our company blog.
Thursday, January 12, 2006
2006 Goals
1. Blog More
I'd like to use this blog to record interesting things in my personal life and to document any research or technical details about the work I do at Webmail.us.
2. Go to the Doctor
Since we have great health coverage at Webmail.us, I better start using it. My dad has had a lot of health problems recently and let's just say it's been a wake up call for me.
3. Get my weight to 175 lbs
I think it's a perfect weight for me, and this goal will force me to attend the gym regularly.
4. Oversee the creation the fastest webmail client on the market.
Speed, speed, and more speed. We are working hard on this problem and it is the focus of our development efforts this year. I'm relying on the great developers we have to help me accomplish this goal.
5. Become the best hitter on my baseball team.
The past two years I have been the second best hitter on my team. This year I really want to take over that top spot.
6. Shoot a round of golf under 110.
I would say under 100, but that might be a bit unrealistic since I never have time to practice.
7. Take my family to Disney World.
Velvet and I work so much that we rarely take family vacations, so I would like to take the kids to Disney World this year. I know Jordan will love it almost as much as Velvet. :)
8. Successfully manage our ever growing group of developers.
I think this will be my biggest challenge this year. It's also the challenge I'm looking forward to the most.
9. Bench Press 200 lbs one time
I use to be able to do this, then I stopped working out. I'm probably at 165 right now (I use to do this 8 times w/o problems), but I honestly haven't tried a low number of reps in a long time.
10. Find more time to spend alone with my wife.
For obvious reasons... ;)
Sunday, January 08, 2006
Understanding memcached memory management
Let's say you have a memcached setup and have allocated 256MB of RAM to the program. memcached works hard to avoid memory fragmentation, it does this by taking the memory and dividing it into classes, numbered 6 through 17 (don't ask). As you insert into memcached, it will take the size of the data you are inserting and inserts it into the class that is big enough to handle the data. See the chart below. The classes in the below memcached are of size 128B - 16KB. Ignore the Max_age column for now. Each class that has data is allocated at least one 1MB page. So, class 7 has 4 1MB pages, each data item in that class is using 128B and the class is full, so there are 32767 items in the class. (If you insert an item into memcached of say 100B, the class uses all 128B to avoid fragmentation.)
Now if we try adding to this cache, we won't necessarily get an "out of memory" error because memcached uses a LRU (least recently used) algorithm to kick out the item from the page that has been accessed the least. But, if we attempt to insert an item of say 15B, we will get an "out of memory" error because class 6 (the class would be of size 64B) has no items, and thus no pages have been assigned. But, all pages are currently being used, and memcached cannot allocate a 1MB page to class 6. In order to make this happen we would have to move a page from another class to class 6.
Let's examine how this is done:
First, let's get a snapshot of our current allocations:
[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 B 0 s 0 yes
7 128 B 742942 s 4 yes
8 256 B1175448 s 2 yes
9 512 B1279471 s 1 yes
10 1 kB6493931 s 1 yes
11 2 kB 434740 s 71 no
12 4 kB 917212 s 140 yes
13 8 kB 794005 s 30 no
14 16 kB1373021 s 7 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes
Now, let's go ahead and try to insert 2 bytes of data:
[root@sql-slave1 bin]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.
set key1 0 0 2
SERVER_ERROR out of memory
OK, just like I thought, memcached can't allocate a 1MB page, so let's help it out:
[root@sql-slave1 bin]# ./memcached-tool localhost move 14 6
Success.
Now, let's look at our pages again, as you can see 1 page has been moved from class 14 to class 6. The LRU items in class 14 go bye-bye.
[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 64 B 0 s 1 no
7 128 B 743011 s 4 yes
8 256 B1174812 s 2 yes
9 512 B1279618 s 1 yes
10 1 kB6494078 s 1 yes
11 2 kB 434887 s 71 no
12 4 kB 917208 s 140 yes
13 8 kB 794152 s 30 no
14 16 kB1373168 s 6 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes
Let's insert our data again:
[root@sql-slave1 bin]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.
set key1 0 0 2
aa
STORED
quit
Connection closed by foreign host.
Cool, it worked. Looking at our stats shows the item is aging:
[root@sql-slave1 bin]# ./memcached-tool localhost
# Item_Size Max_age 1MB_pages Full?
6 64 B 14 s 1 no
7 128 B 742970 s 4 yes
8 256 B1174852 s 2 yes
9 512 B1279658 s 1 yes
10 1 kB6494118 s 1 yes
11 2 kB 434918 s 71 yes
12 4 kB 917228 s 140 yes
13 8 kB 794192 s 30 no
14 16 kB1373208 s 6 yes
15 B 0 s 0 yes
16 B 0 s 0 yes
17 B 0 s 0 yes
Conclusions, you better keep an eye on the data in your memcached, and knowing what type of data you are storing will help you maintain it. Writing scripts to automate the moving around of pages based on data should be fairly easy.
Also, there are some little known commands that I use a lot:
stats
stats items
stats slabs
stats sizes
memcached is a great piece of software, many thanks to the Danga team.
Saturday, January 07, 2006
Newsworthy
http://www.hokiesports.com/football/recaps/2006016aaa.html
and Google has bundled some common software and made it easy to download and install, I haven't tried it, but it looks pretty useful:
http://pack.google.com/
Maybe I'll wait for them to release their operating system. I'm very skeptical this will happen anytime soon. It would have to be built on some sort of Unix flavor and the bottom line is that most of the applications they have built don't have Unix versions. Of course, it is Google, so I'm sure it's on their drawing board somewhere.
Tuesday, January 03, 2006
529 Savings Plans
"But first remember my rule that you don't save a penny for college unless you are already saving the maximum you can for your own retirement. College can be paid for with grants, loans, scholarships and work. Retirement happens only if you have saved the dough."
I don't think the whole "saving the maximum" thing will happen until Jordan is about 33, a little too late to start saving for college. I guess I'll just have to work a few more years, but I like working so no big deal. When I enrolled Jordan, I picked the Missouri state plan because it was on his Honor Roll list ( I don't think he had a Dean's List 4 years ago) and you could sign up for the plan online. This time around I wanted to chose one of his Dean's List plans so I went with the New York plan. The plan is managed by Vangaurd and you can sign up online.
Both the Missouri and New York plans let you invest in "age based" options, meaning the plans become less aggressive as the beneficiary gets older. This is exactly what I want plus it requires zero maintenance on my end, a big-time bonus because I hardly have time to pay attention to it.
I looked into the Virginia 529 plan too (my home state), but their so-called website looks like something a 2 year old put together with Microsoft Frontpage '98.
Va Tech Football 2005 Year in Review
Perhaps Va Tech can start offering a new class for all incoming recruits "How not to look like an immature spoiled kid on national television 101". I guess if you want a squeaky clean program, attend Vanderbilt...Which bowl did they go to this year? And when is my season ticket application due? :(
Update: Since writing this post (I saved it as a draft yesterday), the athletic department at Va Tech released this statement.
Java Schools
I don't use a test as the determining factor for any hire, but I like to see how people go about solving problems. Some struggle, some work fast, some love it, some hate it, some even give up and leave without even trying. If you can't handle 6 simple problem solving questions though, chances are you can't handle developing software applications that are critical to the success of tens of thousands of businesses.
MySQL Replication
1. Read High Performance MySQL by Jeremy Zawodny
2. Turn on logging (vi /etc/my.cnf, log=/var/lib/mysql/log.txt)
3. Make sure every server has a unique id (server-id = 1)
4. I had a pound symbol (#) as part of the password, for some reason this didn't work.
5. Make sure you can telnet to the mysql port on each machine.
6. Make sure the username/passwords are correct.
7. The "show slave status" and "show master status" commands are very helpful.
8. Only replicate the databases you need, make sure there is not a lot of unnecessary traffic.
9. Testing is easy to do, so do it.
10. The /etc/my.cnf file is not used after the first reload of mysql. You need to run the 'change master to' command from that point forward.
Jeremy's book was very helpful, so I highly recommend it for the first-timer.
The best tip I can give is to setup a slave server that does nothing but replicate from the master. This is important because some day you will want to setup another slave without bringing down a live master. This is nearly impossible if you are using all of your machines for queries. To setup a new slave without any downtime, do the following:
On the slave that is doing nothing but replication:
1. Run 'stop slave' from mysql prompt
2. Run 'show slave status\G' from the mysql prompt, record the following two lines:
Master_Log_File: rss-master1-bin.000082
Read_Master_Log_Pos: 993448778
3. Run a "mysqldump" to get all of the data to a text file.
On the brand new slave, do the following:
1. Setup the databases, users, and import the exported data.
2. Run the following command from mysql
CHANGE MASTER TO
MASTER_HOST='master-server',
MASTER_USER='repl',
MASTER_PASSWORD='xxxxxxxx',
MASTER_LOG_FILE='master-server-bin.000082',
MASTER_LOG_POS=934159359;
3. Run 'start slave' from mysql (on both slaves).
4. If you have logging turned on, you will see all of the queries coming through since you stopped the slave process on the first slave machine.
That's it, it should work fine, but testing is encouraged. :)
Friday, December 23, 2005
Webmail.us 2005 Development Year in Review
This past year, thanks to the hard work of our entire team, we were able to launch many improvements to our email hosting infrastructure. Below is a partial list of what we accomplished over the last year:
An improved billing system for our accounting team
Faster, more reliable statistics for our customers
Control Panel 2.0
Control Panel API 1.5, 1.6
Webmail API 1.0, 1.1
Centralized Logging System
Revamped Webmail.us Website
Webmail Lite
An Auto-Configure Tool for POP3 clients
An Outages Alert System for our customers
Improved Migration Process
Improved Anti-Fraud System
Webmail 3.1
Webmail w/ RSS Reader
Webmail w/ AJAX, Calendar, and Company Directory
Webmail Help Section
This list doesn't include all the work required to help get the new infrastructure up and running and any custom development work we did for our customers.
We were able to accomplish so much because of our great team. I thank everyone involved in the process.
Sunday, November 27, 2005
Another Tailgating Season Completed
It's hard to believe that the home football season for the Hokies is over and in the books. I've never had so much fun tailgating and it's a shame the season is over. The main reason the tailgating was so great this year is the great group of friends and co-workers that come out to eat, drink their favorite beverage, play cornhole, and chat about an unimaginable number of topics. A big thanks goes out to David and Jenny Catalano. They are the ones who actually paid for the great spot and we owe them big-time. Also, a big thank you to Bill Boebel who let us use his truck all season, got up at 6:30A.M. to secure the spot, and probably paid for lots of food and drinks out of his own pocket. There are many other people that helped out with the tailgates each week and I'm sure I am going to forget someone (it's a long season, and I didn't exactly take notes) but here it goes:
Velvet, thanks for the awesome taco-dip and most importantly, putting up with me.
David and Megan Lawson, thanks for the great food, beverages, and putting up with all of us.
Gary Francis, thanks for introducing us to the great game of cornhole.
Matt, thanks for getting the food several times and bringing Velvet a birthday cake.
Mike, thanks for being a great cornhole partner and let's just agree never to wrestle again...
Brian Matthews, thanks for the great phone call at 2:30AM, sorry the girl never showed up. :)
Pat, thanks for being so supportive of our efforts and trying your best to make sure that everyone has a good time.
Jilian, thanks for being so sexy. ;)
Justina, thanks for ... well, you know. ;)
Brian Hartsock, thanks for putting up with Velvet and Justina :)
Beth, thanks for taking care of Bill.
Tammy, thanks for being a designated driver.
Vinny, thanks for being a designated driver.
Huey, thanks for calling Vinny.
Marissa, thanks for hanging out until 4:30AM (or later) twice.
Bill (again), thanks for leaving the truck running for 12 hours straight so we could listen to your great music. :)
David (again), thanks for the great wood-burner.
Totally random people, thanks for stopping by and hanging out.
Virginia Tech Police, thanks for not stopping by. :)
Hokie Football Team, thanks for winning so many games.
Everyone else, thanks for coming by and hanging out!
If I forgot about someone or something, I'm sorry in advance, and feel free to leave a comment. Me and Velvet will probably skip on the ACC Championship game, but are planning on attending the bowl game. Let us know if you are going too!
Tuesday, November 08, 2005
My Thoughts on Microsoft Live
Let’s take a look at some of the companies they have conquered in the past:
* Netscape (browser)
* Apple (Operating System)
* Borland (C++ Compiler)
* IBM (Operating System)
* Lotus (Spreadsheet)
* WordPerfect (Word Processor)
I think jumping into online services is a great idea for Microsoft. I think they need to do two things in order to really succeed at this strategy:
1. Integrate their services tightly with their applications
2. Develop API’s so that developers can build applications on top of these services
It sounds like they are going to do both, but I think #2 is especially important. If they provide access to a large web service API that would be a powerful move. It opens the door to a lot of possibilities for the development community out there and I think that’s important for wide-spread adoption. For those people who think they are getting into the web service game a little too late, think again. Being the first to do something has never been a part Microsoft’s strategy and I don’t think they will let their late arrival bother them this time around either.
How will this affect companies like Webmail.us? It's a good question but I don't think it will have much of an impact on what we do. Our company has been tremendously successful because we provide superior service and support, we continuously innovate our product, and we are focused on one thing - email. Our customers love us for our attention to detail, quick response times, and our passion.
If anything, I think the move by Microsoft into the web services space justifies our business model. We have known for a long time that offering software as a service makes complete sense and it's great to see that Microsoft is finally recognizing that fact too.
Sunday, November 06, 2005
My Daily Routine Part II
The only part about my day that I don't like is that I have no time to workout. I'm hoping that I can find some time to get back into the gym sometime soon. At work, it seems like I sit down at my desk in the morning and before I can blink it's time to leave and get Jordan. I hate the days that I forget to pack my lunch because I don't like having to go out and get food, it's such a waste of time.
Before we had Andrew everyone asked how I was going to handle the lack of sleep. Thus far, it's been easy, Velvet handles 99% of the work. She's been great and has really made it too easy for me. She starts work this week and I have a feeling the next six weeks will be harder than the first six. So stay tuned...
Wednesday, October 19, 2005
My New Laptop
I'm really stubborn when it comes to getting a new computer, I try to get everything I can out of a machine until it dies. I think it comes from my poor background. Hell, I used my Commodore 64 from the 2nd grade until my junior year of high school, no joke. That computer was definitely the best computer I've ever owned, but that's a different story.
Besides the Compaq being dog slow (which I really didn't mind because I use SSH for almost everything except email), it had the following issues:
1. The battery only lasted 23 minutes before it would shut off the computer, losing all work.
2. The computer would shut off randomly if the CPU got too hot. I never figured out what made it run so hot, but I did notice that when I had the AIM window open and an advertisement ran with animation the CPU would spike to 100% and shortly thereafter the machine would shut down. Needless to say, I now have a habit of keeping AIM always minimized.
3. The monitor part of the laptop would not stay up on it's own. It would fall back when you opened it, so I had to always prop it up with two pillows. Jordan even knew to put two pillows behind it before opening it, I hope she doesn't grow up thinking that's how *all* computers work.
4. The fan was so loud that I had to turn the TV up about 4 notches to hear it. I remember whenever we would watch movies I would turn off the laptop for a better movie experience.
5. The last straw for the Compaq came when the power cord would no longer stay in socket, so it was impossible to charge. This was the second adapter I bought for it, the first had a different problem that was replaceable, this problem was impossible to fix
Now, my new Sony Vaio is great. The screen is bright, you can actually put it on your lap without worrying about 2nd degree burns, it's quiet as can be, and it's extremely fast. The power cord is a little short, but I don't really need it since the battery last about 2.5 hours (not great, but beats 23 minutes). My only complaint was the crazy amount of software pre-installed. Software? Who uses software? All of the software I actually use, I had to download:
1. Putty
2. AIM
3. Open Office
4. Firefox
5. Programmer's Notepad
Oh, wow, somebody remind me again why Microsoft exists? OK, so I still use them for a window manager, but maybe I don't need to.....
Saturday, October 15, 2005
Andrew Ryan Minnick
Wednesday, September 21, 2005
My Daily Routine
My weekdays begin with the alarm going off at 7:00am sharp. Most mornings I lay there for a few minutes hoping that Velvet will jump into the shower first so I can grab an extra 15 minutes of sleep, it’s usually a 50/50 chance of that happening. Before showering, I open up the laptop, read my morning email, answer all the support questions that I can, read the blogs I follow, and check my Webmail Lite account. (All of this can be done in 15-20 minutes). After showering, shaving, and ironing; I grab a donut, pop-tart, or anything else that is quick and easy and gobble it down. Hopefully by this time Jordan is up, in a good mood, and requires only a moderate amount of teeth pulling to convince her to get dressed for school. (I won’t talk about the days that Jordan is in a BAD mood, LOL) Jordan generally has only two requests: Mickey Mouse pancakes (in a baggie so she can eat in the car on the way to school) and some chocolate milk.
Jordan’s school is only 5 minutes from work and right on the way, so it’s very convenient. I listen to my favorite sports talk show “Mike & Mike in the Morning” on the way to drop off Jordan. Jordan is generally very quiet during the ride to school, eating her pancakes and drinking her milk. But she does like to point out every school bus that she sees (and she has yet to miss one). I usually drop Jordan off at 8:30am and arrive at work shortly thereafter.
I eat lunch at my desk really quickly and try to leave the office by 6:00pm. (Although I have managed to find time to workout twice this week I would hardly consider it part of my “daily routine” at this point.) I come home (Velvet picks up Jordan from school), fix dinner (50/50 chance Velvet will handle this), read a book to Jordan, and give her bath. By this time it’s 8:00pm and I once again open the laptop, check my email, answer support questions, read my blogs, and begin working on whatever issues I didn’t quite have enough time for during “business hours”. After another 3-4 hours of working I close the laptop, grab a few late night snacks and head to bed. I usually toss and turn for 30 minutes or so thinking about servers, code, scalability, infrastructure, hiring, planning, or whatever other issues happen to be on my mind at the time. I force myself to sleep because I know 7:00am will come fast and furious.
Hopefully in a week or two I will have an update on the “routine” or lack thereof. I have a strong suspicion that this may be my last blog for awhile though.
