The Basics

DNA variants occur at different frequencies in different places across the world, and every marker has its own pattern of geographical distribution. The 23andMe Ancestry Composition algorithm combines information about these patterns with the unique set of DNA variants in your genome to estimate your genetic ancestry.

Here's an example of a haplogroup, a special kind of DNA marker, that illustrates the idea. This map shows the frequency of the maternal haplogroup H around the world. Haplogroup H is very common in Europe, is also found in Africa and Asia, and is rarely seen in people native to Australia or the Americas.

The association between this marker and geographic location works in two ways. If you know you have European ancestry, we know that there's a decent chance you have the H haplogroup. And if you have the H haplogroup, we know that your genetic history likely includes at least one European ancestor.

Although we can't locate your ancestry with much precision based on this one DNA marker, we measure hundreds of thousands of DNA markers on the 23andMe platform. If we combine the evidence from many markers, each of which offers a little bit of information about your ancestry, we can develop a clear overall picture.

Wrinkle #1: People Usually Have Multiple Ancestries

If all of your DNA came from one place in the world, figuring out where you're from would be easy. Recent research has suggested that, for a European person whose entire family comes from the same place, genetic analysis can locate their ancestral home within a range of around 100 miles!

But most people's ancestors come from many places. The technical word for this is admixture—the genetic mixing of previously separate populations. For example, it's common for people of European descent to have ancestry from all around Europe, and Latino people typically have ancestors from the Americas, Europe, and sometimes Africa.

Our Ancestry Composition algorithm handles the challenge of admixture by breaking your chromosomes into short adjacent windows, like boxcars in a train. These windows are small enough that it is generally safe to assume that you inherited all your DNA in any given window from a single ancestor many generations back.

Wrinkle #2: We Don't Know Which DNA Comes From Which Parent

Recall that for each of your 23 chromosome pairs, one chromosome in each pair comes from your mother and the other from your father. But genotyping chips don't capture information about which genetic variants reside on the same chromosome.

Here's a quick example to illustrate this point. Say, for a short stretch of Chromosome 1, you inherited the following genetic variants at three consecutive DNA markers:

from Dad: A-T-C

from Mom: G-T-A

When we look at your raw 23andMe data in this spot on Chromosome 1, we'll see the following:

You: A/G   - T/T - A/C

The sources from which you inherited different variants are jumbled up. There are two possible "haplotypes" that are consistent with the raw data, and we don't know which one is your actual DNA sequence. It could be:

A-T-A

G-T-C

which happens to be wrong, or it could be:

A-T-C

G-T-A

which is right. The technical term for determining which variants reside on the same chromosome together is phasing. DNA data like our raw data is called unphased.

So what? This matters because we can learn more from long runs of many DNA markers together than we can learn from individual DNA markers alone. In the above example, the combination A-T-C will generally say more about your ancestry than the A, T, and C say when they are considered separately. Luckily, we can use statistical methods to estimate the phasing of your chromosomes. After phasing your raw data, the Ancestry Composition algorithm calculates ancestry separately for each phased chromosome.

The Ancestry Composition Algorithm

Overview

The Ancestry Composition algorithm comprises four distinct steps.

First, we use a computational method to estimate the phasing of your chromosomes — that is, to determine the contribution to your genome by each of your parents. Second, we break up the chromosomes into short windows and compare your DNA sequence in each window to the corresponding DNA in our reference datasets, labeling your DNA with the ancestry whose reference DNA it is most similar. Third, we process those assignments to "smooth" them out, computing a probability distribution over ancestries for each segment of your genome. Fourth, we summarize these probabilities to “paint” your chromosomes with inferred ancestries. Each step in this process is described in more detail in the following sections.

Step 1: Phasing

Recall wrinkle #2 above. For each customer, we measure a set of genotypes (pairs of alleles). But what we really want is a pair of haplotypes for each chromosome. That is, we want to figure out the series of alleles present on each of your two copies of, for example, chromosome 7: one you received from your mother and one you received from your father. To do so, we built a very large "phasing reference panel" using data from more than ten million customers. We then use SHAPEIT (Hofmeister et al., 2023) to jointly phase these individuals. SHAPEIT uses sophisticated statistics and a very clever algorithm to do this. Once we have phased this large collection of customers, we can use the inferred information to efficiently phase new customers.

Step 2: Classifying Windows

After phasing your chromosomes, we segment them into consecutive windows containing ~300 genetic markers each. We measure between 7,300 and 45,000 markers per chromosome, which translates to 24 to 147 windows, depending on the chromosome's length. We consider each window in turn and compare your DNA to the reference datasets to determine which ancestry most closely corresponds to your DNA.

There are many ways to assign ancestry to DNA segments based on reference data, and we tried several. The best-performing option was a well-known classification tool called a support vector machine, or SVM. An SVM can "learn" different ancestry classifications based on a set of training examples and then assign new DNA segments to a learned category.

In the case of Ancestry Composition, we train the SVM with reference DNA sequences and tell it which ancestry population those sequences are from. Then, when we look at the DNA from a 23andMe customer with unknown ancestry (like you), we can ask the SVM to classify your DNA for us based on the reference datasets.

We use SVMs because they performed the best out of all the techniques we tried. SVMs are also very fast, which is critical for a large and growing database.

Step 3: Smoothing

The SVMs classify each window of your genome independently, creating a "first draft" version of your ancestry result. We use another computational process, called a smoother, to smooth this raw SVM output. The smoother uses a version of a well-known mathematical tool called a Hidden Markov Model to correct, or “smooth,” two kinds of errors. Hidden Markov Models are used to analyze sequential data, like biological sequences or recorded speech. As an example, suppose we had three ancestry populations: X, Y, and Z. An example of output from the SVMs might look like this:

Chromosome 1, Parent 1: X - X - X - Z - Z - Z - Y - Z

Chromosome 1, Parent 2: Z - Z - Z - X - X - X - X - X

The first kind of error the smoother corrects is an unusual assignment in the middle of a run of similar assignments. In the first line above, there's a run of Zs, interrupted by a single Y. It's possible that the lone Y was a close call between Y and Z that went the wrong way. If that were the case, the smoother could correct it to a run of just Zs.

The second kind of error the smoother corrects arises from the phasing step. Phasing algorithms can make mistakes, known as switch errors, where they mix up the DNA of one parent with that of another. The smoother can switch the ancestry assignments between your mother and your father if it detects one of these errors. In this example, there may be a switch error after the third window. If the switch were reversed, then the runs of Xs and the runs of Zs would stay together. In our simplified example, the smoother might output something like this:

Chromosome 1, Parent 1: Z - Z - Z - Z - Z - Z - Z - Z

Chromosome 1, Parent 2: X - X - X - X - X - X - X - X

This example illustrates the purpose of the smoother. But with real data the picture is much messier, and the correct inferences are rarely so clean. So instead of assigning a single ancestry to each window like we did in this example, the smoother estimates the probabilities of each Ancestry Composition population matching each window of DNA. The following picture shows a concrete example:

This is the output of the smoother analysis of one copy of chromosome 2. Starting on the left, there is a short run of pink, then a wider run of green, then another run of pink. In this chart, pink is the color for Sub-Saharan African ancestry, and green is the color for Native American ancestry. The y-axis runs from 0 to 100 percent, and it shows the probability that the DNA in that region of the chromosome comes from each Ancestry Composition population. These pink and green regions fill the entire vertical space of the graph, which means that we are 100 percent confident that the DNA in those regions has Sub-Saharan African and Native American genetic ancestry, respectively.

The next region to the right—between positions 50 and 100 on the x-axis—is a stretch of multi-colored blue. The thickest strip at the bottom is dark teal, which is the color for British & Irish. This segment of DNA has somewhere between a 50 percent chance and a 60 percent chance of reflecting British & Irish ancestry. The other shades of blue show that the same DNA segment also has a chance of reflecting Italian, Iberian, or French & German ancestry. If you think back to the haplogroup example above, this result makes sense: it is normal for a DNA marker to match reference DNA from lots of places, even if it matches some places better than others. In this example, the result shows that this DNA segment matches reference DNA from all over Europe. We can very confidently conclude that this stretch of DNA reflects European ancestry, but we are less confident in assigning it to one specific region of Europe.

Step 4: Painting

The last step is to summarize the smoother-computed probabilities to “paint” your chromosomes with an inferred ancestry for each segment of your genome. Once we’ve assigned an ancestry to each segment, we add up the lengths of all the assignments to compute your ancestry proportions. We use two different painting approaches, “most likely” and “threshold-based”, and display both in your Version History and DNA Painting Reports.

Mostly Likely Painting

In the “most likely” painting approach, we consider a genome segment’s ancestry probability distribution in the context of our population hierarchy. First, we add up probabilities across each continent and decide which continental assignment is most likely for the segment. Then, we add up probabilities across each region within the most likely continent and decide which region is the most likely assignment. Finally, we select the most likely sub-region within the most likely region and label the segment accordingly. Once we’ve painted all your chromosomes, we fill in any small anomalous segments with the ancestries of neighboring segments. This procedure gives the most specific population assignment to each segment of your genome—that is, there are no assignments to broader regional or continental populations.

Threshold-Based Painting

Another way to paint your chromosomes is to apply a threshold to the probability plot as in this figure:

The horizontal line in this image indicates a 70 percent confidence threshold, which we will use for this example. You can view your own DNA Painting at different confidence thresholds, ranging from 50 percent (speculative) to 90 percent (conservative).

In this approach, we ask, for each segment, whether any ancestry has an estimated probability exceeding the specified threshold (in this case 70 percent). In this example, with the exception of the blue European stretch, the ancestry estimates exceed 70 percent over the majority of the chromosome. Consider the green vertical band near the right-hand side of the plot. Even though there is some probability that the segment comes from a different population, the Native American proportion exceeds the 70 percent threshold, and so we label this segment Native American.

In the case of the European segment, no single ancestry exceeds the 70 percent threshold, so we don't assign that DNA to any fine-grained ancestries. Instead, we refer to our hierarchy of ancestries. When this figure was originally generated, there was a "Broadly Northern European" ancestry that included four fine-level ancestries: British & Irish, Scandinavian, Finnish, and French & German. If, when we add up the contributions of each of these subgroups, the total contribution toward Broadly Northern European exceeds the 70 percent threshold, then we will report the region as Broadly Northern European.

In this example, the Broadly Northern European reference populations still don't exceed the 70 percent threshold, but the combined probabilities of all the European populations do. So this region is assigned "Broadly European" ancestry.

When using threshold-based painting, we use broad Ancestry Composition categories to avoid making assumptions about your ancestry when your DNA matches several different country-level populations. In regions where no ancestry—including the broad ancestries—exceeds the specified threshold, we report "Unassigned" ancestry. You can see the entire ancestry hierarchy in your Ancestry Composition report by clicking "See all tested populations."

Testing & Validation

Ancestry Composition includes a lot of steps, and each step has to be tested. We've discussed a few of those tests already while explaining our algorithm. In this section, we want to share some test results to give a sense of how well Ancestry Composition works. This section focuses on the final test we run, because that integrates the performance of each of the steps into an overall picture.

This test looks at two classic measures of model performance, precision and recall. These are the standard measurements that researchers use to test how well a prediction system works. Precision answers the question "When the system predicts that a piece of DNA comes from population A, how often is the DNA actually from population A?" Recall answers the question "Of the pieces of DNA that actually are from population A, how often does the system correctly predict that they are from population A?"

There is a tradeoff between precision and recall, so we have to strike a balance between them. A high-precision, low-recall system will be extremely picky about assigning, say, Korean ancestry. The system would only assign DNA as Korean when it is very confident. That will yield high precision—since the assignment of Korean is almost always correct—but low recall, because a lot of true Korean ancestry is left unassigned.

With a low-precision, high-recall system the opposite problem exists. In this case, the system liberally assigns Korean ancestry. Any time a piece of DNA might be Korean, it is assigned that ancestry. This will yield high recall, as most genuine Korean DNA will be labeled accordingly, but low precision, because non-Korean DNA will often be incorrectly labeled Korean.

The ideal system has both high precision and high recall, but that may be impossible in real life. Let's see how Ancestry Composition performs on these metrics. For this quality-control test, we set apart 20 percent of the reference database, approximately 4,300 individuals of known ancestry. We trained and ran the entire Ancestry Composition pipeline on the other 80 percent of the reference individuals. Then we treated the "hold-out" 20 percent as though they were new 23andMe customers and used our Ancestry Composition pipeline to calculate their ancestries. Since we know these people's true ancestries, we can check to see how accurate their Ancestry Composition results are. We repeated this test four more times, with a different 20 percent held out each time, and then averaged across the five tests to give the following results, shown here for a minimum confidence threshold of 50%:

This table shows that our precision numbers are high across the board, mostly above 90 percent, and rarely dipping below 75 percent. That means that when the system assigns an ancestry to a piece of DNA, that assignment is very likely to be accurate. You can also see that as you move up from the sub-regional level (e.g., Irish) to the regional level (e.g., British & Irish) to the continental level (e.g., European), the precision approaches 100 percent.

Poor recall doesn't mean bad results. Some populations, like Russian, are hard to tell apart from others. When Ancestry Composition fails to assign Russian DNA, that DNA is almost always classified as "Belarusian, Polish & Ukrainian", a closely related population.

The precision and recall metrics reported above are for unadmixed individuals. We conducted similar experiments for admixed individuals by simulating individuals with known ancestry. As this is a harder problem, performance isn't quite as good as for unadmixed individuals, but it is still quite good.