We begin this chapter with an overview in section 1 of how the scalar–tensor theory was conceived, how it has evolved, and also what issues we are going to discuss from the point of view of such cosmological subjects as the cosmological constant and time-variability of coupling constants. In section 2 we provide a simplified view of fundamental theories which are supposed to lie behind the scalar–tensor theory. Section 3 includes comments expected to be useful for a better understanding of the whole subject. This section will also summarize briefly what we have achieved.

In section 1 we emphasize that the scalar field in what is qualified to be called the scalar–tensor theory is not simply added to the tensor gravitational field, but comes into play through the nonminimal coupling term, which was invented by P. Jordan. Subsequently, however, a version that we call the prototype Brans–Dicke (BD) model has played the most influential role up to the present time. We also explain the notation and the system of units to be used in this book.

The list of the fundamental ideas sketched in section 2 includes the Kaluza–Klein (KK) theory, string theory, brane theory as the latest out-growth of string theory, and a conjecture on two-sheeted space-time.

After section 4.1 giving a brief history of the problem of the cosmological constant, we go up the ladder starting from the standard theory with Λ added (section 4.2), proceeding to the prototype BD model without Λ (section 4.3), and culminating in the prototype BD model with Λ included (section 4.4), where the discussion will concern both the J frame and the E frame. We will face some crucial aspects that Λ has brought into being for the first time. Most remarkable is that the attractor solution in the J frame represents a static universe. This conclusion turns out to be evaded in the E frame, but particle masses are shown to vary with time too much. We then propose in subsection 4.4.3 a remedy in the matter part of the Lagrangian, thus violating the WEP in a manner that, we hope, allows us to remain within the observational constraint. At this cost, however, we are rewarded with a successful implementation of the scenario of a decaying cosmological constant in the E frame, which is now considered to be (approximately) physical. Another point to be noticed is that a physical condition, positivity of the energy density of matter, requires that ∈ = –1, an apparently ghost nature of the scalar field in the J frame, unexpectedly in accordance with what string theory and KK theory suggest. This also entails the condition, and thus is in contradiction with the widely known constraint ω ≳ 3.6 × 103, or ξ ≳ 7.0 × 10-5. A reconciliation with the solar-system experiments will be made only with a nonzero mass of the scalar field.

During the last few decades of the twentieth century, we saw an almost triumphant success in establishing that Einstein's general relativity is correct, both experimentally and theoretically. We find nevertheless considerable efforts still being made in terms of “alternative theories.” This trend may be justified insofar as the scalar–tensor theory is concerned, as will be argued, not to mention one's hidden desire to see nature's simplest imaginable phenomenon, a scalar field, be a major player.

The success on the theoretical front prompted researchers to study theories with the aim of unifying gravitation and microscopic physics. Among them string theory appears to be the most promising. According to this theory, the graviton corresponding to the metric tensor has a scalar companion, called the dilaton. The interaction between these two fields is surprisingly similar to what Jordan foresaw nearly half a century ago, without sharing ideas that characterize the contemporary unification program. There seems to be, however, a crucial point that might constrain the original proposal through the value of the parameter ω, whose inverse measures the strength of the coupling of the scalar field.

More specifically, string theory predicts that ω = –1, which goes against the widely accepted constraint from observation, namely ω ≳ 103 ≫ 1. Although many more details have yet to be worked out in order for string theory to be compared with the real world, we point out that expecting the dilaton to be close to the limit of total decoupling is by no means obvious or natural.

Once we introduce the nonminimal coupling term, we face the issue of conformal transformations. By applying a conformal transformation, we can put a nonminimal coupling term into another form. This comes from the fact that Einstein's theory is not invariant under conformal transformations. In this sense this transformation has a feature different from the gauge transformation. In the literature, however, we sometimes find confusions. We wish to provide readers with a better understanding of the issue.

We are particularly interested in how we can eliminate the factor of the scalar field in a nonminimal coupling term by transforming it into a constant. We say that we move from one conformal frame to another by applying a conformal transformation. The questions are then those concerning what conformal frame we live in, and on what physical grounds we are able to select which. An explicit discussion of these questions will be given in Chapter 4 on cosmological applications.

Among infinitely many conformal frames, the J(ordan) frame and the E(instein) frame are those discussed most frequently. Generally speaking, physics looks different in two different conformal frames. In the limit of a weak gravitational field (including a diagonalization process), however, physical conclusions remain the same.

In section 1, the concept of conformal transformation is introduced in general terms but briefly. A fact of special importance in connection with the nonminimal coupling term is discussed in section 2.

We now move on to discuss the second face of the problem of the cosmological constant, which was highlighted recently by the discovery of the acceleration of the universe. This chapter will first review briefly how searching for “dark energy” has come finally to a spatially flat universe well described by a cosmological constant Λ of a size smaller than but nearly comparable to the critical density. For a number of reasons, we consider that this Λ is not a true constant but is mimicked most naturally by a scalar field.

In section 5.1, we sketch what the development has been like mainly on the observational front, culminating in the conclusion that we have an accelerating universe.

As a possible theoretical model discussed recently, we first review in section 5.2 the results of “quintessence,” a name mainly indicating a cosmological scalar field. Since this is a phenomenological approach that is not necessarily constrained rigorously by the scalar–tensor theory, our focus is mainly on the assumed inverse-power potential. A primary concern is the question of how naturally the initial conditions for the scalar field can be chosen. A relevant question is that of whether the scalar-field energy falls off in the same way as the ordinary matter density (“scaling”), or approaches the latter starting from different values (“tracking”).