It sounds like you’re working too hard.
I started training for that, with something like 3 days of training per week + going out and trying my hardest on the weekends.
“Hardest” is for goal events, rarely training. GOing your hardest in training should happen maybe once in 20 workouts, most of which would be closer to a goal event. 95% of your training should be super easy, especially because it sounds like you’ve gone so hard you’ve destroyed your aerobic system. It will take some time and patience to repair it.
My problem is, nearly every-time I come back from a mini-expedition (4 days of intensive big-wall, or a 16h day of alpine mixed climbing with my mentors, stuff like that), I come back and I get ill in the next week. My body is just 100% crushed.
This is an indication that you don’t have the base you need to absorb the load. A training base has three functions:
1. It allows for increasing speeds at low training intensities, sometimes quite fast;
2. It lowers the metabolic cost of high-intensity efforts; and
3. It speeds the recovery from large loads, whether from high-intensity, long duration, or both.
It looks like my body is lacking some kind of background energy. Even when I workout during the week, I usually takes me 2 days to fully recover,
Yes, you’re going too hard. I suggest reading a lot of the articles on this site. Aerobic Deficiency Syndrome is self-inflicted but curable.