Home / more texts

HIDDEN VARIABLES IN QUANTUM THEORY:

THE HIDDEN CULTURAL VARIABLES OF THEIR REJECTION

By Miguel Montenegro

September 2006

This is an interpretation of what appears to me as a strange situation in physics: the rejection, by mainstream physicists, of hidden variables interpretations (in particular David Bohm’s) of quantum mechanics without any really solid scientific reason. The few physicists who have given the theme some attention acknowledge this. Others dismiss it with a hostile and characteristically brief remark. Most just don’t touch the subject at all.

I am not a physicist. What follows is written from an informed layman’s perspective. But the worldview that emerges from physics concerns us all. And I have reasons to believe that many mainstream physicists are driven by fears and cultural premises that I find highly questionable and of which they may be completely or, at least, partly unaware[1].

Why did the majority of physicists lock themselves up inside a hermetic understanding of quantum mechanics, the so-called “Copenhagen interpretation” or “standard interpretation,” conceived (and fiercely sustained) by Bohr in partnership with Heisenberg? The mathematical theory is good in that the equations allow good predictions, in spite of the lack of articulation with the other main physical theory of the twentieth century: relativity. But the mathematical theory must be distinguished from the Copenhagen interpretation which is often presented as the only possible one. The paradoxes usually associated with quantum theory are not directly derived from quantum mathematics but from the Copenhagen interpretation of quantum formalism.

If the successes of quantum mechanics explain physicist’s adherence to its mathematical core, they cannot account for their historical allegiance to the Copenhagen interpretation or for the shifts of attitude and worldview that it entailed:

1) the renunciation to the prospect of ever developing an intuitive grasp of quantum reality and the correlate conviction that quantum phenomena can only be described by mathematical formulae;

2) the acceptance of the “fact,” “imposed” by quantum mechanics (and, more specifically, implied by the postulates of the Copenhagen interpretation), that the deeper level of physical reality is the quantum level as it is depicted by current quantum theory and presently available experimental results;

3) the acceptance of the wave-particle ambiguity as an inescapable implication of physical experiments as they are explained by quantum equations coupled with the standard Copenhagen interpretation;

4) the acceptance of the probabilistic predictions allowed by quantum theory as the best possible approximation to quantum events.

In the subsequent history of physics through the twentieth century and after the formative years that began with Planck’s radiation formula (1900) and culminated with Von Newmann’s standard mathematical account and interpretation of quantum theory (The Mathematical Foundations of Quantum Mechanics, 1932), there has been an oscillation between, on the one hand, an epistemological view, strictly in line with the Copenhagen orthodoxy, and, on the other hand, more realistic approaches (but, nevertheless, Copenhagen-inspired) of two of the features presented above: the acceptance of the wave/particle ambiguity and the acceptance of probabilistic predictions as inherent features of the quantum world. While the epistemological approach ascribe these two features of quantum theory to intrinsic limitations of experimental situations (empirical interpretation of Heisenberg’s principle of uncertainty) and of the knowledge relationship (Bohr’s complementarity), the realistic approach implies that quantum “things” do have, in themselves, a probabilistic behaviour and a particle-wave duality that are “resolved” – the famous wave collapse – in and through measurement. (Particles are somehow supposed to be multi-located or diffusely located prior to measurement.)

The realistic (but Copenhagen-derived) standpoint has been fairly dominant up to now[2] and has led to most of the paradoxes, difficulties and eccentricities associated with quantum mechanics, ranging from the Schrödinger’s cat paradox to the quantum version of the “antropic principle,” according to which man was a necessary cosmological development since human observation and awareness of the universe allowed for the universe’s “quantum wave” to collapse and the cosmos to take a definite shape. Alien and, in a way, contrary to the “standard interpretation,” but nevertheless guided by the Copenhagen constraints, is the “many worlds theory” which, stated flatly, implies that each time a physicist makes quantum-related measurements in his laboratory, the universe splits into so many universes as there are possible results to the measurement[3].

Underlying the Copenhagen dogmas that shaped twentieth century physics is the refusal to acknowledge that quantum theory and quantum equations may only be an approximation of limited validity to the underlying reality. At the core of this refusal we find the rejection of what has been called the “hidden variables hypothesis.” According to those who sustain it, quantum theory is only an approximate description of atomic and sub-atomic phenomena. It yields probabilities that may be compared to the observation of crowds and mass phenomena, but that doesn’t mean there are not more fundamental levels where a statistical approach would give way to causal explanations, just like it happens with the so-called Brownian movement which can only be accurately described by statistical means at the macroscopic level, whereas at a microscopic level it allows a classical kinetic and causal description. Of course, in the quantum domain, non-locality implies that causality and determinism would not lead us back to anything resembling a classical view of the universe.

I believe that the hidden variables hypothesis is the real reason, the real “menace” to which most of the physics community reacted by sticking blindly and, one could almost say, “religiously” to a hard-boiled interpretation of quantum equations made of renunciations and of pill swallowing that led – and is still leading – to the most abstruse and far fetched interpretative consequences.

The ordinary substantive reasons put forward to dismiss the hidden variables approach appear fragile: hidden variables interpretations do not necessarily imply more assumptions or “arbitrary” constants than other more “orthodox” theories. And the current difficulties and apparent impossibility in assessing experimentally the validity of hidden variables theories is not an argument either since it works equally well against the standard interpretation. And there is another good counter-example: string theory is generally accepted as a good bet by the physics community without any empirical and experimental evidence to support it.

At this point I should also mention, very briefly, some historical reasons commonly used to explain the dominance of the Copenhagen interpretation and the general neglect of hidden variables interpretations. One is the fierce debate that opposed Bohr, the Copenhagen champion, and Einstein, who, in principle, favored a hidden variables approach. This confrontation would have pushed Bohr into his dogmatic and anti-realist final positions. Another reason would have been the neo-positivist climate in Europe at the time quantum physics was born; neo-positivism was anti-realist and, pushed to the limits, it was instrumentalist: theories are instruments that serve to organize and predict experimental data. They don’t necessarily reflect reality. Heisenberg shared this point of view. And, of course, there is the obvious if chocking factor that David Bohm, the major proponent of the hidden variables hypothesis, used to point out: the fact that the Copenhagen interpretation established itself first.

I don’t deny any of these factors. But I think the main reason why quantum mechanics was enshrouded in an orthodox and “tough” interpretation against any hidden variables interpretation was that the latter aroused the specter of determinism that was inherent to classical Newtonian physics. Quantum equations, with their mathematically grounded probabilities seemed to, if taken at face value – i.e. without considering the possibility of hidden variables – point to a fundamental level of reality (as far as it could be experimentally inferred) in which alea, that is, chance, seemed to play a fundamental role, liberating us from the “billiard ball” universe of Newton and allowing us into a cosmos where not all the cards are set from the beginning, but where, at the same time, a certain order seems to reign; an order that, at least locally, has been sufficient for life and consciousness to evolve.

As every anthropologist knows, different domains, areas and themes are, in any given culture, linked by different kinds of semantic and logical connections. Among the most important of these is what I call resonance, but which may also be thought of as “metaphorical” links. I believe the option between enshrining quantum mathematical and experimental theory in orthodoxy or taking it as a fair approximation and considering the possibility of hidden variables (which is nothing more than admitting that quantum uncertainty and ambiguities may be explained by quantum theory’s “telescopic” character and its consequent inability to describe “deeper” levels of reality) “resounded” with other dilemmas in other cultural domains of the twentieth century Western world, namely in the political and in the religious realms.

Quantum orthodoxy was a way out of the suffocating Newtonian model of the universe, which, according to Allan Watts, who called it the “fully-automatic model of the universe,” had already been a way out (with a price attached) of the “clay-model of the universe,” the biblical model in which an exclusivist, omniscient “God” made the cosmos, the same way as “He” made us all. The fully-automatic model was a haven to escape from the “clay-model” with its meddlesome all-seeing God.

Of course, the fully-automatic model also had its shortcomings. It had been the paradigm of the nineteenth century industrial world with all its political and social bleakness to which Romanticism had reacted in an idealistic and, as Colin Wilson pointed out, vain and self-defeating way. But its main drawback was the complete loss of liberty in a “cold mechanical world.”

The Newtonian deterministic world was also intimately linked with one of the two “totalitarianisms” that shaped the twentieth century: communism, whose dialectic materialism intended to extend the deterministic causal view of the universe from the physical realm into the social and historical realms, where it replaced the Hegelian idealistic determinism by turning it on its head.

In the physicist’s minds – and, probably, on a more or less subconscious level – letting chance in was undermining the Western mechanical civilization with its prophetic ideology of Progress (in the social and historical realms) and Evolution (which grounded Progress in the fundamental physical and natural realms) as well as refusing the physical confirmation – “resonance,” one could say – of dialectical materialism’s determinism, a confirmation upon which communist ideology depended for the validity of its prophecies and worldview[4].

The fact that the major proponent of the hidden variables hypothesis (after de Broglie’s initial proposals), David Bohm (1917-1992), was a victim of McCarthyism and publicly exposed as a communist in the beginning of his career (in 1949) was probably not without consequences for the fate of his theoretical option in the second half of the twentieth century. Bohm’s interest in communism was short-lived[5] and the fact that, from the early 1950’s until his death in 1992, he actively and consistently developed a hidden variables theory shows that his physical views had other grounds.

Finally, I should point out that standard quantum theory had another advantage over the “fully-automatic model.” Even if the latter was an escape from the “clay-model,” and even if God was, according to Laplace, a “superfluous hypothesis,” the “fully-automatic model” still allowed Him some backstage role (like setting-up the universe’s clockwork in the beginning of time). But the quantum alea was an entirely different matter because it implied, in Einstein’s words, that the capitalized God would be playing dice! One could, with some effort, endure Job’s paradox, i.e. the fact that God allows evil, but to know that His mysterious ways are nothing but a game of dice was to be confronted with an absolutely inconceivable and self-contradictory God. This turned quantum mechanics into a practical, mathematically and experimentally grounded ay of keeping Him out of the game!

Well then: with just one stroke, we were getting rid of the rude nineteenth century mechanical civilization, with its blind trust in Progress and Evolution, of communism and all its philosophical similes, and, at the same time, keeping God off premises!  And they want us to give it up? No way!

For the moment, at least, some of the implicit grounds on which the physics community rejected the hidden variables hypothesis are historical and I doubt the monotheistic God would use that door. I think there was a shift in the last quarter of the twentieth century, as a result of which the main specter associated with the hidden variables hypothesis was no longer God, communism or a mechanistic universe, but something else. Something that also expressed itself through that diffuse cultural movement known as “New Age,” a protean mixture of Oriental mysticism, Western occultism and multicultural patchwork, which inherited some of its features from the Beatniks, the American counter-culture and the Flower Power/Hippy trends.

David Bohm’s intellectual itinerary seems to have followed the flow, biographically speaking, as he associated with Krishnamurti and became interested in subjects such as parapsychology[6] and non-Western worldviews. But it is also clear from his work that his philosophical and existential positions were inseparable from his views in physics. These seem to have evolved considerably, but within a global orientation that goes back to his first doubts about the consistency of the standard interpretation following the publication of his Quantum Theory (1951) [7]. On the other hand, he had developed, from a very early age, a deep interest in the notion of wholeness, and, as a young man, he found in the philosophy of Hegel a way of dealing with it. It was this same wholeness that first appealed to him in Bohr’s interpretation of quantum mechanics in which the quantum phenomenon is inseparable from the measuring instrument and the whole act of observation. But this “indivisibility of the quantum of action” worked more like a localized lock preventing further inquiry than as hint to, or a preview of, the interconnectedness of the universe and its workings. An explorer rather that a ideologue, Bohm had no hesitations in abandoning the standard interpretation when he realized that quantum formalism authorized the more rational  and conceptually sound picture of localized particles evolving in a shifting network of waves (see footnote 7). This realization set his intuition of the universe as a whole on a much clearer and broader outlook, which, unlike the Copenhagen view, wasn’t closed to intellectual examination.

Indeed, Bohm’s version of the hidden variables theory – as well as his philosophical views – seems to “resonate” with the “holistic” dimensions of Oriental philosophies. This is obviously opposed to the historical ethos of Western science based on distinction and separation (which was inherited from Judaism via Christianism). I believe this “resonance” constitutes nowadays – particularly since the publication of Capra’s book The Tao of Physics, which publicly introduced the “Oriental connection” into physics – an important, if not the main underlying reason why the hidden variables hypothesis is still banned from mainstream physics.

I believe it is not just the single “deterministic possibility” embedded in the hidden variables hypothesis that scares the physics community but the combination of determinism[8] with non-locality. This combination yields a vision of a (possibly) deterministic “undivided whole,” as Bohm called it, that doubly deprives Westerners in general and physicists in particular of their separated and mysteriously free egos[9], and of their potentially controllable world made of distinguishable and divisible parts.

***

The evolution of physics is, in a very large measure, connected with cultural factors that physicists and “scientifically oriented people” tend to dismiss because it deeply contradicts their self-image. I’m not saying that cultural environment determines physicist’s choices and ideas regardless of scientific criterions[10], theory’s internal logic, experimental evidence, and so on. I’m rather saying that, whereas the latter factors structure deeply what we could call the “physicist’s field of choices” and are the main and, virtually, the only really determining factors they acknowledge when they account for their decisions, these decisions are also influenced by the cultural and historical environments and, namely, by socio-cultural domains that, in our reflexively compartmented cultural world, we construe and perceive as being largely unrelated to physical science.