Both FitAndFlirtyHotwife and Freya de Vries have talked about how, after being married for several years, their husbands eventually opened up about a fantasy they wanted to explore: seeing their wives become intimate with another man.
In my view, the important point is the emotional aspect of this arrangement. If the same partner is involved repeatedly over a long period, there is a possibility that the connection could gradually become emotional rather than remaining purely physical. The wife might start seeing that person more like a boyfriend, and the relationship could eventually take on the dynamics of a live-in relationship.
That is why, if the goal is to keep the experience purely within the boundaries of a fantasy, constantly changing partners may prevent a strong emotional attachment from developing. If someone repeatedly becomes intimate with the same person, emotional bonding can naturally grow over time. With different partners, that same level of attachment is less likely to develop.