I'm not sure I quite get it, given being polyamorous, we're all bound by the same rules as far as relationships go, and thus equal. Those who choose not to express themselves sexually do so either because of their age or because they have qualms with our body's gender, but those who choose to have a partner (or more) can do so as long as they adhere to the same rules as everyone else (most of which concern safety.)
I fail to see how this arrangement doesn't provide equality, nor do I see how it could damage inner balance. If anything, your arrangement would, in our circumstances, cause harm as many within this system require physical intimacy for one reason or another (the most extreme of which being a "succubus" for lack of a better term.)