Rob, this is a great topic and thread. I feel it all the time, although I would say I would not feel it if I did something family related. Actually, I feel it whenever I am at the bench that I am in fact STEALING time away from other matter, both personal and professional in nature, but then I snap out of it and feel fine a few hours later.
I know I feel guilty if I am not building in particular when I order supplies or after market parts and then a week or two go by and I don't build. I ask myself what's it all for, anyway. Did I make a mistake.
I think this is all natural, and very much a part of the creative process, just like when I finish a build I tend to get depressed and then spend a week or so moping about uninspired.
It does help to talk about it, so thank you for bringing it up.