Personally, I record it all, but for a couple of different reasons.
The most important thing for me is to keep an eye on fatigue. I use Training Peaks as my log, so that makes it easier, because their metrics quantify fatigue. It’s not three-decimal-place accurate, but it’s better than what I would do on my own.
The second reason I track everything is to motivate me to do more of it. I’m a big believer in “what gets measured gets managed”. If I don’t track something, usually I stop paying attention to it.
In that spirit, I’ve also started tracking recovery activities that have no direct fitness impact. Although I don’t count these as training hours, I know I’ll do more of them if I can see recovery volume rising.
I hope that helps.