This is common when adding Z4 intervals. The slowing can (probably) be due to local muscular fatigue. I’d suggest the following options as a course of action.
Shorten the ‘on’ time and try the same total volume so in stead of 5x4min try 7x3min. And/or Take longer recovery intervals. If you are slowing 10% that means that the higher force Fast Twitch fibers are dropping out of the recruitment pool. So they are not getting the desired training effect.
Another way, and our and Kilian’s preferred method of adding Z4 intervals is with 30-30s. I’ve used them for years with skiers and runners. Sam just wrote an article 30-30s and it should go live on the site in the next week or two.