It sure does not sound like you are rest too much. If this is happening often enough that you notice a pattern then I take a look at the week leading into that rest week. It sounds to me like the load then is so high that it is really running you down and leaving your immune system depressed so you get sick a few days later.
In general light activity is going to speed recovery better than total rest. Unless you are completely trashed or sick/