Why do we think dark energy exists?

It has long been known that the Universe is expanding; in the late 1990s, the discovery was made that this expansion is speeding up, due to something called "dark energy". Why do we think dark energy exists?

By way of explanation, our goal is to understand this plot from the Supernova Cosmology Project:



and, in particular, the meanings of the lines drawn on the plot and the implications of the points as plotted.

The x-axis (horizontal) of this plot is fairly straightforward — z is the redshift. For the purposes of this discussion, I want you to think of redshift as indicating lookback time into the past, so a higher redshift means longer ago (and we're talking billions of years here, not last week Wednesday).

Let's motivate this. The Universe is expanding; what this means is that space itself is stretching, much as the surface of a balloon stretches as it expands. In terms of the current discussion, this says that a light wave emitted in the past has a longer wavelength now because the space between the wave crests has stretched since the light was emitted. A longer wavelength means redder light (hence, redshift), and the z value on this graph is a technical measure of that. The higher the redshift, the greater the stretching, and hence the further back in the past the light was emitted.

So just know that z = 1 is roughly 8 billion years ago, with a more exact value depending on a number of factors I'm not going to talk about here.

The y-axis (vertical) is somewhat obscurely labeled, but is simple to understand. The points on the plot represent Type Ia supernovae, whose absolute brightness is known from the physics of the cause of the explosion. Put simply, the y-axis represents the difference between the measured (apparent) brightness and the apparent brightness we'd expect at that redshift (z value) if the Universe was empty.

Consider that an empty expanding universe should expand at the same rate forever; there would be no gravity to slow it down. In such a universe, the apparent brightness of an object would be precisely the brightness you'd expect at its distance as computed from its z value (how long ago it was emitted) and the known constant rate of expansion (which we can measure). That exact linear relationship would be represented by a straight line across the graph at y = 0. Deviations from this straight line are therefore a measure of deviation from a constant rate of expansion (in other words, acceleration or deceleration of the expansion rate) as the universe ages.

Since the Universe is patently NOT empty, we would expect gravity to be slowing down (decelerating) the rate of expansion. The y-axis of this graph is computed such that, in a decelerating Universe, the difference between observed and expected brightness of objects (the "magnitude residual") due to this presumed slowdown falls below y = 0.

Thus, objects in a matter-filled, gravitationally decelerating universe would be, by definition, below the y = 0 horizontal line on the graph. The dashed line represents a universe with 1/4 of the matter required to stop the expansion (and no dark energy at all), and the dotted line represents a universe with exactly the amount of matter required for gravity to stop the expansion of the Universe at infinity (again without dark energy). Those lines indicate how fast the Universe would be decelerating at each z value; as you would expect, the dotted-line Universe, having 4 times as much matter as the dashed-line universe, would decelerate much more quickly. Both types of universe would decelerate faster as the z value increases (which makes sense, because an expanding universe would be smaller long ago, so galaxies were closer together, which means their mutual gravitational attraction was greater).

But look at the actual supernova data: when you get beyond z = 0.2, where the differences between the possibilities really begin to diverge, 18 out of 20 of the points fall above the empty-universe value of y = 0 — and the two that are below y = 0 have error bars that reach above it. Since a decelerating universe is depicted by objects falling below the y = 0 line, if objects instead fall above that line, it follows that the expansion of the Universe must actually be accelerating. Remarkably, the rate of expansion of the Universe is speeding up, not slowing down.

This was a completely unexpected result, made all the more solid by the fact that there were two competing teams of observers, using different methods, who got the same results. In fact, neither team believed it; they each spent a year or so trying to figure out where they'd gone wrong.

Nobody knows why this acceleration is occurring; it's as if there's some unknown type of energy out there that pushes things apart more strongly than gravity holds them together over cosmological distances. Incidentally, this particular graph was published in 2003; the case has only gotten stronger since then, as various alternative ways of explaining the data have been proposed and shown to be wrong, and as additional data has been added.

The difference isn't small, either; in order to generate an expansion-rate history line (the solid blue line) that rather approximates the supernova data points, you have to posit a universe that consists of three times as much dark energy by mass as it does matter.

You'll note that, in the plot, the solid blue line begins to turn back downward after z = 0.6 or so. This means that the rate of acceleration was slower in the past, and it is believed that, at even higher z values (much farther into the past), the Universe actually was decelerating as expected due to gravity. Eventually, however, the acceleration became stronger than the gravity-induced deceleration, and the Universe has been accelerating at an ever-increasing rate ever since.

The interpretation of all of that is that the acceleration is due to a property of space itself. In the early Universe, there wasn't much space between the galaxies, so gravity was stronger than the acceleration. But eventually, as the galaxies continued to move apart, enough space between the galaxies existed so that the force of the space-caused acceleration overcame the force of gravity.

We call the cause of this acceleration "dark energy", just to give it a name; its nature is a mystery. Perhaps it's general relativity's cosmological constant, perhaps it's some manifestation of the vacuum energy, or perhaps it's something else.