February Bench Press Results

Another month has come and gone and I know you are dying to know how much I benched pressed today, so here ya go:

155 x 8 times
175 x 3 times
185 x 2 times
190 x 1 time

190! Holy cow, only 10 more pounds to reach my year end goal! Not bad considering I weighed in at 163lbs today. I haven't gained or lost any weight since I started working out, that might be more amazing than the 190lbs I lifted today.

Cryptic Apache Error

Each night our webmail servers automatically restart httpd to clean up memory and start fresh. Last night one of our servers hit this error upon restart:

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed

I had never seen this error before and had no idea what it meant, but Google helped me out:

http://www.goldfisch.at/knowledge/224

After some testing, I found that Apache leaks a couple of semaphores after each restart and eventually w/o a server reboot will run out of resources. So I changed our restart script to clean up these semaphores each night.

ipcs -s grep nobody perl -e 'while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

In case you are wondering, it's not a big deal if a webmail server doesn't restart properly. Our load balancers quickly take a down server out of rotation so that customers are not affected. Thank you keepalived.

Development Releases Last 30 Days

Over the last 30 days we have released some really great additions to our platform. Here is the high level list I presented at our management meeting on Monday:

* Webmail Lite w/ Calendar / AJAX
* Control Panel Security Additions (IP Based Restrictions)
* 20+ Webmail Bug Fixes
* Webmail Import / Export Enhancements
* Company Directory
* Website Search
* 30 Day Free Trial

You can read the details on our blog.

OpenLDAP Replication

Today I setup a new LDAP slave machine, the process is fairly straight-forward, but a good thing to know about replication is how to do it without shutting down the master for an extended period of time. Let's take a look at how to do this:

First, let's verify we have a few things:

1. The LDAP ssl port is open from the master to the slave:
ldap-master> telnet ldap-slave 636

2. Make sure the slave has a valid SSL certificate. If you are using a self-signed cert you will need to make sure the master has the slave .crt file installed. If you have multiple slaves, you need to create a combined.crt file and concat all of your certs into one file. Then add an entry to the master ldap.conf (not slapd.conf) like so:

TLS_CACERT /usr/etc/openldap/certs/combined.crt

OK, so now we need to get all of the data from the master to the slave without losing anything during the transfer. Normally this would require the master to be taken offline for some period of time while you zip up the data files and scp them over. But, luckily I have another slave that is not being used for queries, so I can follow these steps to get the data moved without having to shut down the master for an extended period of time:

0. Stop the new slave.
1. Add the necessary

replica uri=ldaps...

lines to the master slapd.conf file.

2. Restart the master.
3. Stop the old slave.
4. Tar/Zip the data, scp to new slave.
5. Start the new slave, verify replication works to the new slave.
6. Bring back online the old slave, verify replication is still working.

That's it, really simply. One trick you should know about replication is that if you have multiple LDAP databases being replicated, slurpd will not replicate the second database to a machine with the same hostname as the first database. I have to create a fake host entry in /etc/hosts so that slurpd doesn't get confused. It took me awhile to figure this one out.