Tough to compete with Bruno’s detailed response, but I’ll offer two thoughts:
Reduce risk in appropriate ways and take the family with you.
1. Since my daughter’s birth, I’ve learned about the different components of risk in mountain travel and this has helped me dial down risk while keeping, for lack of a better term, the stoke high. I learned that while I used to spend a fair amount of time in areas of high exposure (skiing above large cliffs, e.g.), I no longer felt the need to. So, in my ski mountaineering or free riding, I have dramatically reduced the exposure but I have continued to ski steep terrain. This might not apply to alpinism, or be very palatable if it does. For me, the transformation in understanding and goal setting happened while listening to a conversation about the difference between steep skiing in Chamonix and Verbier. In the simplest terms, Chamonix was the town people went to test their ability to handle exposure in you-fall-you-die terrain, and Verbier was were you went to ski super steep couloirs with mellower run-outs and much less skiing about cliffs. The light went on. I can still ski steep stuff, I just have to avoid the exposure. A bystander might say, yeah, but what about the avalanche danger? I’m working on that one. I did back off of something in March and it burned for the next few hours, until I saw my daughter. Since then I’ve come to understand backing off as a learned skill. Who knows?
2. While you probably won’t be taking your toddler and wife to Baffin Island, you could certainly take them to the Alps. Expeditions or multi-day loops are much trickier with a family. But the Alps allow for lots and lots of single day efforts that are as burly as you want. And at the end of the day you’ll come back to food and architecture that will be a big step up over resort towns in the US. Costs are much more reasonable once you absorb the flights (those will be pricier). Day care in US resort towns can great, but in my experience in alpine Austria, Switzerland and north-eastern Italy, the ski schools and kindercare are dialed. They are also used to handling children from many different countries and languages.
Good luck and congrats.