I am pretty sure the one headphone amp Presonus makes takes an A and a B input source, that gets you half the way there.
I have a Rolls headphone amp that has stereo inserts on each channel, effectively giving each channel its own input, which is the way I use it, so I have six different headphone monitor mixes used in this way from a single headphone amplifier. Of course the line in on each channel is not balanced, because the stereo insert/ input on each channel does not allow for it.
I am willing to bet there are other headphone amps available that have either a stereo insert or a direct stereo input on each channel, much like the Rolls headphone amp that I have. Do a search based upon that feature and see what you can come up with. If you want just one headphone amp, that is probably the way to go. Otherwise you'll be stuck with more than one headphone amp.