You need to back off on the intensity or it will negate the benefit of your base work.
Think of it like a bank account. Zones 1 and 2 make deposits; Zones 3-5 make withdrawals. The bad news is that, minute for minute, the withdrawals are 20x more powerful than the deposits…
As a rule of thumb for long events, you want to keep your minutes of training above Zone 2 at 5% or less of your total training time. For example, although my main event is intense (skimo racing), in a 15-hour training week, I try to keep my training time above Zone 2 to 45′ or less.
Judging by your description, a long focus on base building is what you need. But if you’re patient, it will be worth it.
Although not as severe, I was in a similar situation when I started properly training six years ago. I had a long history of alpine climbing, but I still couldn’t jog at much less than 80% of maximum. And I usually lagged behind my friends. Now that I’ve built a good base, I can comfortably jog at 55-60% of maximum, and days out with friends are never over Zone 2 unless I intentionally pick up the pace.
Be patient! You’ll get there.