"I also found sleeping on it often magically produces a solution in the morning."
I think this is a great reason to take a nap in the middle of the day if your environment is conducive to it. It works for me as long as the code is the top thing on my mind when I fall asleep.
I think doing yoga poses, walking up a steep hill for 10 minutes, or practicing guitar have similar/related brain benefits: Neurologists: is it epinephrine or endorphine?
This has happened to me on the past two consecutive days (and nights). A problem that I couldn't figure out at 11pm, one that I spent at least 30 minutes pouring through Google Groups, searching through APIs, trial and error, etc...After sleeping on it the solution revealed itself to me in the morning in less than five minutes. Truly phenomenal and hugely rewarding.
The one I remember vividly was a bug in a student project - I was writing a set of tools to "compile" lambda calculus expressions to to various sets of combinators and gather statistics about the effectiveness of different sets of combinators (starting with SK).
For some reason any attempt to implement a Y-combinator for recursion was crashing and for a couple of days I had no idea what could be wrong - then sitting on a bus on the way to visit my sister I was looking at a cinema from the bus and the thought came into my head "it's your aggressive re-use of applicative nodes".
Sure enough next day I removed this attempt at an optimization and everything worked!
I've had a few experiences like that and I now know that if I find a really tricky bug the best strategy isn't to sit there all night looking at it but to go and get a decent nights sleep - there is a very good chance you will know the answer by the next morning.
I've done this and actually find it a bit disturbing.
It's great when I can't solve a problem in the evening, get a good nights sleep, and then I am able to quickly solve the problem in the morning. Sometimes though I actually have dreams about sitting at my desk writing the code that I couldn't come up with during the day. On the one hand it's great to wake up in the morning with the solution to my coding problem. But on the other hand, I'm not sure it's healthy to have my dreams filled with the same thing that I do all day while awake. Whenever that happens I definitely make some extra time for my hobbies that don't involve computers.
A story, perhaps apocryphal, about Thomas Edison. He would hold ball bearings in his hand as he fell asleep, and the noise of them falling to the floor would wake him immediately as he went to sleep, thus generating the shortest possible power nap.
There's a decent chance that story is actually lifted wholesale from a similar one told about Dali, except it was a spoon instead of ball bearings. I suppose it's the best you can do after melting your alarm clock.
Dali makes a big production out of the way he did it and describes it extensively in his book "50 Secrets of Magic Craftsmanship". It was a key rather than a spoon in that case, though no doubt the details changed in every telling.
During his day, Edison would take time out by himself and relax in a chair or on a sofa. Invariably he would be working on a new invention and seeking creative solutions to the problem he was dealing with. He knew that if her could get into that "twilight state" between being awake and being asleep, he could access the pure creative genius of his subconscious mind.
To prevent himself from crossing all the way over the "genius gap" into deep sleep, he would nap with his hand propped up on his elbow while he clutched a handful of ball-bearings. Then he would just drift off to sleep, knowing that his subconscious mind would take up the challenge of his problem and provide a solution. As soon as he went into too deep a sleep, his hand would drop and the ball-bearings would spill noisily on the floor, waking him up again. He'd then write down whatever was in his mind.
I think this is a great reason to take a nap in the middle of the day if your environment is conducive to it. It works for me as long as the code is the top thing on my mind when I fall asleep.