The genetic consequences of inbreeding is a subject that has received thorough theoretical attention and has been of interest to empirical biologists since the time of Darwin. Particularly for species with genetically controlled mechanisms to promote outcrossing (self-incompatibility or SI systems), it is expected that high levels of genetic load should accumulate through sheltering of deleterious recessive mutations. Nevertheless, transitions to selfing are common across angiosperms, which suggests that the potentially negative consequences of reduced heterozygosity and genetic diversity are balanced by other factors, such as reproductive assurance. This mini-review focuses on empirical research in the Brassicaceae to emphasize some of the genetic consequences of shifts to inbreeding in terms of mechanisms for loss of SI, changes in genetic diversity following loss of SI, and inbreeding depression in relation to outcrossing history. Despite the long history of theoretical attention, there are still some surprisingly large gaps in our understanding in each of these areas. Rather than providing a complete overview, examples are drawn predominantly from published and emerging data from Arabidopsis thaliana and its relatives to highlight recent progress and remaining questions. We are currently on the brink of major breakthroughs in understanding due both to advances in sequencing technology and a shift in focus from crop plants to natural populations, where critical factors such as population structure, phylogeography, demographic history, partial compatibility and individual variation can be taken into account when investigating the nature of the selective forces regulating mating system evolution.

Inbreeding species of hermaphroditic animals practising copulation have been characterized by few copulations, no waiting time (the time that an isolated individual waits for a partner before initiating reproduction compared with paired individuals) and limited inbreeding (self-fertilization) depression. This syndrome, which has never been fully studied before in any species, is analysed here in the highly selfing freshwater snail Biomphalaria pfeifferi. We conducted an experiment under laboratory conditions over two generations (G1 and G2) using snails sampled from two populations (100 individuals per population). G1 individuals were either isolated or paired once a week (potentially allowing for crosses), and monitored during 29 weeks for growth, fecundity and survival. Very few copulations were observed in paired snails, and there was a positive correlation in copulatory activity (e.g. number of copulations) between the male and female sexual roles. The waiting time was either null or negative, meaning that isolated individuals initiated reproduction before paired ones. G2 offspring did not differ in hatching rate and survival (to 28 days) between treatments, but offspring from paired individuals grew faster than those from isolated individuals. On the whole, the self-fertilization depression was extremely low in both populations. Another important result is that paired G1 individuals began laying (selfed) eggs several weeks prior to initiating copulation: this is the first characterization of prior selfing (selfing initiated prior to any outcrossing) in a hermaphroditic animal. A significant population effect was observed on most traits studied. Our results are discussed with regard to the maintenance of low outcrossing rates in highly inbreeding species.

Androdioecy, where males co-occur with hermaphrodites, is a rare sexual system in plants and animals. It has a scattered phylogenetic distribution, but it is common and has persisted for long periods of evolutionary time in branchiopod crustaceans. An earlier model of the maintenance of males with hermaphrodites in this group, by Otto et al. (1993), considered the importance of male–hermaphrodite encounter rates, sperm limitation, male versus hermaphrodite viability and inbreeding depression suffered by selfed progeny. Here I advance this model in two ways: (1) by exploring the conditions that would allow the invasion of hermaphrodites into a dioecious population and that of females into an androdioecious population; and (2) by incorporating a term that accounts for the potential effects of genetic load linked to a dominant hermaphrodite-determining allele in androdioecious populations. The new model makes plausible sense of observations made in populations of the species Eulimnadia texana, one of a number of related species whose common ancestor evolved hermaphroditism (and androdioecy) from dioecy. In particular, it offers an explanation for the long evolutionary persistence of androdioecy in branchiopods and suggests reasons for why dioecy has not re-evolved in the clade. Finally, it provides a rather unusual illustration of the implications of the degeneration of loci linked to a sex-determining locus.

The most prominent feature of Y chromosomes is that they do not recombine and are usually genetically degenerate, containing only a few genes. White campion Silene latifolia has evolved sex chromosomes relatively recently, probably within the last 10–15 million years. Perhaps due to its recent origin, the Y chromosome in this species has not completely degenerated and most isolated X-linked genes have intact Y-linked homologues. A gene encoding a protein with strong homology to spermidine synthases, Slss, is the exception to this rule, as the Y-linked copy of this gene has apparently lost its function. Here I report evidence for a recent selective sweep in the X-linked copy of this gene (SlssX) that could reflect compensatory evolution in an X-linked gene that has lost a functional Y-linked homologue. The spread and fixation of an advantageous mutation in SlssX has resulted in a dramatic loss of genetic diversity and an excess of high-frequency derived polymorphisms in this gene. As the sweep has not affected the closely linked DD44X gene, the selective advantage of the mutation that has driven the sweep in the SlssX gene might have been less than 1%.

Mating systems and recombination are thought to have a deep impact on the organization and evolution of genomes. Because of the decline in effective population size and the interference between linked loci, the efficacy of selection is expected to be reduced in regions with low recombination rates and in the whole genome of self-fertilizing species. At the molecular level, relaxed selection is expected to result in changes in the rate of protein evolution and the pattern of codon bias. It is increasingly recognized that recombination also affects non-selective processes such as the biased gene conversion towards GC alleles (bGC). Like selection, this kind of meiotic drive in favour of GC over AT alleles is expected to be reduced in weakly recombining regions and genomes. Here, we investigated the effect of mating system and recombination on molecular evolution in four Triticeae species: two outcrossers (Secale cereale and Aegilops speltoides) and two selfers (Triticum urartu and Triticum monococcum). We found that GC content, possibly driven by bGC, is affected by mating system and recombination as theoretically predicted. Selection efficacy, however, is only weakly affected by mating system and recombination. We investigated the possible reasons for this discrepancy. A surprising one is that, in outcrossing lineages, selection efficacy could be reduced because of high substitution rates in favour of GC alleles. Outcrossers, but not selfers, would thus suffer from a ‘GC-induced’ genetic load. This result sheds new light on the evolution of mating systems.

One of the long-standing mysteries in genomic evolution is the observation that much of the genome is composed of repetitive DNA, resulting in inter- and intraspecific variation in nuclear DNA content. Our discovery of a negative correlation between nuclear DNA content and flower size in Silene latifolia has been supported by our subsequent investigation of changes in DNA content as a correlated response to selection on flower size. Moreover, we have observed a similar trend across a range of related dioecious species in Silene sect. Elisanthe. Given the presence of sex chromosomes in dioecious Silene species, and the tendency of sex chromosomes to accumulate repetitive DNA, it seems plausible that dioecious species undergo genomic evolution in ways that differ from what one might expect in hermaphroditic species. Specifically, we query whether the observed relationship between nuclear DNA content and flower size observed in dioecious Silene is a peculiarity of sex chromosome evolution. In the present study we investigated nuclear DNA content and flower size variation in hermaphroditic species of Silene sect. Siphonomorpha, as close relatives of the dioecious species studied previously. Although the nuclear DNA contents of these species were lower than those for species in sect. Elisanthe, there was still significant intra- as well as interspecific variation in nuclear DNA content. Flower size variation was found among species of sect. Siphonomorpha for petal claw and petal limb lengths, but not for calyx diameter. This last trait varies extensively in sect. Elisanthe, in part due to sex-specific selection. A negative correlation with nuclear DNA content was found across populations for petal limb length, but not for other floral dimensions. We conclude that impacts of nuclear DNA content on phenotypic evolution do manifest themselves in hermaphroditic species, so that the effects observed in sect. Elisanthe, and particularly in S. latifolia, while perhaps amplified by the genomic impacts of sex chromosomes, are not limited to dioecious taxa.

Cytoplasmic genomes typically lack recombination, implying that genetic hitch-hiking could be a predominant force structuring nucleotide polymorphism in the chloroplast and mitochondria. We test this hypothesis by analysing nucleotide polymorphism data at 28 loci across the chloroplast and mitochondria of the outcrossing plant Arabidopsis lyrata, and compare patterns with multiple nuclear loci, and the highly selfing Arabidopsis thaliana. The maximum likelihood estimate of the ratio of effective population size at cytoplasmic relative to nuclear genes in A. lyrata does not depart from the neutral expectation of 0·5. Similarly, the ratio of effective size in A. thaliana is close to unity, the neutral expectation for a highly selfing species. The results are thus consistent with neutral organelle polymorphism in these species or with comparable effects of hitch-hiking in both cytoplasmic and nuclear genes, in contrast to the results of recent studies on gynodioecious taxa. The four-gamete test and composite likelihood estimation provide evidence for very low levels of recombination in the organelles of A. lyrata, although permutation tests do not suggest that adjacent polymorphic sites are more closely linked than more distant sites across the two genomes, suggesting that mutation hotspots or very low rates of gene conversion could explain the data.

The problem of jointly estimating the intensity of past selection affecting an allele and the allele's age is formulated in a Bayesian framework. The prior distribution of allele age given its frequency is obtained from existing population genetics theory. The prior distribution of selection intensity is assumed to reflect the fact that positive selection on a new mutant is more likely to be weak than strong. The general approach is illustrated by the development of an importance sampling method applicable to low-frequency alleles. This method can be used either when the haplotypes of closely linked marker loci are known or when the lengths of linked ancestral chromosomal segments can be inferred. The method is illustrated with an application to the A− allele of G6PD in Africa. Because changes in allele frequency and recombination are both intrinsically stochastic, there are limits to the accuracy achievable with any method.

Explicit formulae are given for the effects of a barrier to gene flow on random fluctuations in allele frequency; these formulae can also be seen as generating functions for the distribution of coalescence times. The formulae are derived using a continuous diffusion approximation, which is accurate over all but very small spatial scales. The continuous approximation is confirmed by comparison with the exact solution to the stepping stone model. In both one and two spatial dimensions, the variance of fluctuations in allele frequencies increases near the barrier; when the barrier is very strong, the variance doubles. However, the effect on fluctuations close to the barrier is much greater when the population is spread over two spatial dimensions than when it occupies a linear, one-dimensional habitat: barriers of strength comparable with the dispersal range (B≈σ) can have an appreciable effect in two dimensions, whereas only barriers with strength comparable with the characteristic scale () are significant in one dimension (μ is the rate of mutation or long-range dispersal). Thus, in a two-dimensional population, barriers to gene flow can be detected through their effect on the spatial pattern of genetic marker alleles.