i think that there is another factor here to consider.
as we dive vertically or ascend there is a total body bouancy and total body weighted (how much weight you are wearing) this results in no physical compensation for the distribution of the weight and bouancy (it is all oriented in a straight line)
when swimming horizontally, there are different area of your body that are differently weighted (hence the need for neck weights or ankle weights). As such, i feel like there is more stabilizing that has to happen and thus more exertion that happens when swimming horizontally.
if one is able to find a perfect weight that included being balanced horizontally, there would have to be weights on the legs and arms and that would make the swimming harder.
just my thoughts