An analogy may be useful here. The position that Harris’ thought experiment thrusts us into is comparable to sailors adrift on a lifeboat. Clearly, all of these sailors would recognize that it would be a positive thing if everyone on the boat had water to drink, or as Harris inversely presents it: a lifeboat with no water is objectively worse than a lifeboat with water. Thus—Harris’ reasoning goes—there are objectively right and wrong answers to moral questions related to having water on a lifeboat. But where does this get us exactly? While all of these sailors may agree that having water is better than not having water, this does not bring us any closer to objective truth. There is in reality still no unity of value here, only the appearance of it. To underscore my point, let us assume that the Buddha and Idi Amin are among these sailors. Both the Buddha and Idi Amin would certainly agree that having no water on the lifeboat is categorically bad, but they would do so for entirely different reasons. While the Buddha may be adopting this position from the utilitarian’s ‘point of view of the universe’, having compassion for all people on the boat and viewing the situation from a perfectly objective stance, Idi Amin likely remains stuck in the subjective stance, thinking of himself. Yet the Buddha and Idi Amin appear to be in perfect moral agreement. What happened? The problem is that within Harris’s thought experiment, moral value is never actually identified, just preferences. Moral debate begins when interests start to collide—when there is some but not enough water for all the sailors—not when interests are comfortably aligned. Harris’s thought experiment does not bring us any closer to objective value—it merely achieves the illusion of it. With one cup of water, this illusion is shattered and the profound moral incongruity between the Buddha and Idi Amin that was present all along comes suddenly into view.
And here comes the truly difficult part. While we may find Idi Amin morally repugnant, we cannot provide a rational account of why Amin should adopt an objective view of his situation. The insight that value is reducible to brain states tells us nothing about why Idi Amin should care about the brain states of other people.
–Bryan Druzin, “Shouldering the Burden of Utilitarianism”