New (to me) UNIX Commands

I often have the need to compare two files on different servers, now I can do so without having to scp the file and then run diff:

vimdiff scp://webmail1//home/kminnick/tmp.txt scp://webmail2//home/kminnick/tmp.txt

I also often have the need to take a file and split it into several small pieces:

man split

Query Optimization

One of the more enjoyable parts of my job is finding and eliminating bottlenecks in our system. Yesterday I was faced with a problem of a moderate load average problem on our RSS servers. I could tell that our system has been growing and that there must be a query that just wasn't performing as optimally as when we first launched. I know the RSS code fairly well, so I dove in and starting looking for queries that could possibly be the culprit. It didn't take long to find a query that when executed on the live system took over 5 seconds to finish (table names and columns changed):

SELECT s.id, s.u, s.d
FROM s, i
LEFT JOIN p ON s.u = p.u
AND s.d = p.d
AND p.pr = 'RSS_E'
WHERE (p.val IS NULL OR p.val != '0')
AND s.fId = i.fId
AND s.lDItem < i.rTime LIMIT 1;

OK, so I looked at the columns of the tables and two columns (lDItem and rTime) were not indexed. So I added indexes to both columns and ran the query again. This time it actually took 10 seconds to execute! As my 4 year old daughter would say, "What in the hecka in the world?!?" I've never really seen MySQL slow down after adding indexes, but I know the query optimizer logic is very complex, so I decided to help it out a bit by giving it a hint as to which index to use:

SELECT s.id, s.u, s.d
FROM s, i use index (rTime)
LEFT JOIN p ON s.u = p.u
AND s.d = p.d
AND p.pr = 'RSS_E'
WHERE (p.val IS NULL OR p.val != '0')
AND s.fId = i.fId
AND s.lDItem < i.rTime LIMIT 1;

Now the query only takes .48 seconds to execute! Excellent. I also dropped the lDItem index and the query speed stayed the same, so I only really needed the one new index and a simple little addition to the query to achieve a 10x improvement in the query. This little fixed has eliminated all of the CPU bottlenecks on the RSS servers today. This stuff makes me so happy. :)

The point is that query optimization is not always a matter of just adding an index (many times it is, but not always). I had to experiment several times before I got the results that I expected.

MySQL Chain Replication

This took me awhile to figure out, but here is the option you need to configure chain replication w/ MySQL. This is useful for geographically dispersed MySQL servers. From the MySQL documentation:

--log-slave-updates

Normally, a slave does not log to its own binary log any updates that are received from a master server. This option tells the slave to log the updates performed by its SQL thread to its own binary log. For this option to have any effect, the slave must also be started with the --log-bin option to enable binary logging. --log-slave-updates is used when you want to chain replication servers. For example, you might want to set up replication servers using this arrangement:

A -> B -> C

Here, A serves as the master for the slave B, and B serves as the master for the slave C. For this to work, B must be both a master and a slave. You must start both A and B with --log-bin to enable binary logging, and B with the --log-slave-updates option so that updates received from A are logged by B to its binary log.

Little League Dilemma

I've coached little league ( a long time ago), it was fun, competitive, and more than anything my kids all had fun win or lose (but we never lost). Which brings us to a story from Salt Lake that just blew my mind. Read the full details here, but here's a quick synopsis:

Salt Lake, little league championship. Last inning. The best hitter on the opposing team is up with two outs. The kid has already hit a homer and a triple. Any rational coach would think about intentionally walking the best hitter so that he doesn't hit another homer, makes perfect sense to me. One problem though, the kid that hits after him is a cancer survivor. So they walk the best hitter, and the cancer kids starts crying after he gets two strikes. Oh boy. What were they thinking? The kid strikes out, crying his eyes out. The opposing coach is furious.

So, if you were the coach would you have walked the best hitter to get to the kid w/ cancer? I'm not sure I'd be able to do much celebrating if I did that. Maybe the bigger question is what in the world is the cancer kid doing batting behind the best hitter. You don't put Mario Mendoza behind Babe Ruth...Idiot.

How many of you would play for the win in that situation?

Amazon S3 Part 2

OK, so I got a few complaints (you know who you are) about part 1 of my Amazon S3 post being too long and boring. So....I'll keep part 2 short and sweet. Basically, I wrote a backup / restore script for my linux files using PHP and REST (instead of SOAP). The bottom line is that I love the Amazon REST interface. It's simple and easy to use, forget SOAP, use REST.

The only other issue w/ S3 that I've noticed is that it will fail 1 out of every 2000 or so requests. Based on the forums, you simply have to code around this issue. It's easy to do, but I'm not sure why it fails so frequently. But I've never seen it fail twice in a row, so simply re-trying the request after a failure seems to always work.

Blacksox Turning It Around

My baseball team won this past Sunday 20-2 over a pretty good team. I was amazed. Overall we are only 4-8, in 5th place (8 teams total). But, we have improved dramatically over the last 3 weeks. I'm really looking forward to the remaining games. I'm hitting .310, which isn't bad, but it only is 6th best on the team. My brother is hitting .462 with 17 RBI.

I haven't picked up a golf club in a couple of months, I'm just too busy, so I'll probably wait until the fall to pick it up again.

Our company softball team has formed, I've had fun at the two practices I've been able to attend. It should be an interesting first year, I believe our games should be starting sometime soon.