I still remember The Art of Assembly book by Randall Hyde that I read during college and his qoute

The Best Optimizer is Between Your Ears

The idea simply was revolving about the fact that if you focus too much on details and try to keep optimizing something that works really slow by just trying to get the assembly code more efficient or by using higher processing power, then you are just wasting your time… 20% or 30% performance improvement in an application that runs in 40 or 50 seconds isn’t really that much… What you should be looking at is a totally novel solution.

Well he was right, but this time the optimizer was not between my ears, it was between the ears of the people who developed the Sphinx Full Text Search Engine… Using Sphinx our team was able to make our ypanalytics tool much faster, we use this tool to analyse the user search trends and usage of Yellow Pages sites…

The tool now became 60 times faster! a mysql query that took 2 minutes to execute using an optimized mysql query now takes 2 seconds to execute using sphinx! Perhaps it is worth mentioning that the query originally took perhaps 4 or 5 minutes to execute before query optimization but still normal optimization is not enough by any means.. you just need a new solution to solve a new problem, solutions which you already know and have dealt with might not be sufficient any more…