Sure you can do your “fun” runs. Normally your longer runs will be done at a slower pace than shorter runs. Doing long hard runs, especially if pushing a lot in to Z3-4 will mean you need more recovery time. If your AeT is high then tossing a “fun” run into your program 1-2x/week is an important thing to do. However, if you are aerobically deficient doing these runs too often will derail the intended base building process. As your aerobic base becomes stronger your easy runs will become faster and faster. Then you’ll be doing your long fun runs at a fast pace.
In the end it is all up to you and what your goals are. When you look at the elite runners one common thing you see is that 80% of their training volume is done below AeT but this is still fast training. It took many thousands of km run below AeT to make them this fast.
Scott