It will improve for sure, your body will adapt. For the most part I don’t notice a noticeable difference now with fasted vs non-fasted Z1 and Z2 workouts with one big exception: if I cut carbs from my diet I get that HR spike again.
Take this morning. I cut virtually all carbs out starting Monday (I’m doing the Maffettone carb intolerance test, which is almost keto) and did a 6 mile fasted Z2 workout. So I was pretty much completely glycogen deprived. It was painfully slow and my HR spiked easily, I had to stop a ton going uphill, whereas usually I can keep chugging along. I noticed I was breathing harder and my leg muscles would even feel it on occasion which I hadn’t noticed before.
I figure it’s the price to pay to become better at fat burning.