It is a wonder that podcasts are generally open (and remained so) by convention. The concept itself, "periodical audio delivered digitally", does not impose openness. It could theoretically be executed by closed and proprietary platforms, if only they gain enough market power.
Right now, I suppose, the market is still fragmented enough so that all players (including big ones) benefit from general openness. This post made me curious, and I learned that podcast producers can generally move their podcast hosting provider ("publisher") because most of them 1) interoperate the open RSS standard for podcast sourcing & publishing, and 2) allow to configure a "HTTP 301 Moved Permanently" redirect that informs podcast subscribers/clients/directories of a new RSS feed location.
In that sense, the "portable audience" analogy with Mastodon holds up. But Mastodon goes further: it's possible to easily export your followee list, bookmarks, and more, and import those into your new server. This is true portability: from the producer's and from the consumer's perspective.
I'm personally using Spotify a lot, but I am worried that it does lock me down. They have to play along with open podcast sharing today, but they surely don't make it easy to move your subscriptions & listening history to a new app or service.