The answer depends on your fitness level and your goals. The fitter the athlete and the higher the goals the more specific must be the training. Typically xc skiers do a lot of trail running and ski-hiking in the their base period because these are not too far removed from xc skiing movements. Kayaking can be good upper body training but certainly not as close to skiing as double poling on roller skis or a ski erg. Skiers at fairly low levels of fitness as well as younger juniors will benefit from most training types because an increase in general work capacity will transfer over to specific. For the the variety of training types can be important to prevent boredom and injury. But for sure top level skier MUST train specifically if they hope to improve and use things like cycling and rowing as recovery workouts.
It definitely is best to do your high intensity training in more ski specific ways as this training models the sport more and so the transfer of training effect will be larger.
Scott