What Kernel are you guys using?

The one I compiled. The benefits are not that obvious. If you are not into that sort of stuff, I would not recommend it.

But yes, if you don't support as wide of a range of hardware, you can usually squeeze a bit more performance out of your kernel.

Although, most processing time is spent not in kernel code. There will be a difference of course, especially if you know where to look for it, but nothing groundbreaking most of the time.