Top 4 backlink API vendors compared (1 million domain study)

For a couple of years, SEO PowerSuite has been working on enhancing its backlink index, used by the SEO PowerSuite backlink API and the SEO SpyGlass backlink tool. In June 2021, a huge infrastructure update went live, increasing our crawling speed by three times.

To better understand our progress and see where we stand, we’ve decided to run an in-house analysis of the top backlink database providers — Ahrefs, SEO PowerSuite, Semrush, and Majestic.

We’ve described all the major findings of the comparison here, but more details can be found on this page.

Disclaimer: For this study, we’d purchased the API keys from each of the compared providers ($6,200 worth in total). In this report, SEO PowerSuite enclosed the raw data files for your reference. To verify that the data hasn’t been modified in any way, you can reach out to the mentioned companies and purchase similar stats from them to run your own comparison.

The methodology

To run this analysis, we decided to use the Majestic Million dataset.

Why this dataset? The Majestic Million is a list of the top million sites on the web, based on the number of citations (aka links) from other websites. That’s why each of these domains should have sufficient backlinks to make the comparison statistically significant.

So, we have taken this dataset and analyzed the results from the following backlink index providers:

SEO PowerSuite

SEO PowerSuite backlink index counts 3.6 trillion external backlinks and 264 million indexed domains.

The cost of checking backlink stats for 1M domains: $899.

Ahrefs

According to the official website, Ahrefs has 3.2 trillion external backlinks and 193 million indexed domains.

The cost of checking backlink stats for 1M domains: $2,000.

Semrush

Semrush contends to have over 43 trillion backlinks and 1.6 billion domains in its index. 

The cost of checking backlink stats for 1M domains: $2,500.

Majestic

We haven’t found any information on the number of backlinks and referring domains in the Majestic backlink index. The only stats that are available on the official website are that the Fresh backlink index has 434 billion URLs crawled and over 1 trillion URLs found.

The cost of checking backlink stats for 1M domains: $800.

After setting up the accounts and checking the data for one million domains, we then calculated

• the total wins by referring domains;
• the total wins by backlinks;
• how often each of the vendors occupied the 1^st, 2^nd, 3^rd and 4^th position in the comparison (for links and referring domains);
• and a few more metrics covered on this page.

The study was conducted in autumn 2021.

Findings                                  

Who wins by the number of referring domains?

First off, we looked at the number of referring domains returned by every provider, the metric that lets us evaluate the backlink index coverage.

When it came to the largest number of reported backlink domains, SEO PowerSuite led the way with 44.4% wins:

The backlink index by Semrush took second place with over 36.3% wins. Ahrefs received 14% wins, and Majestic – 5.3%.

Please keep in mind that the graph above highlights the number of wins. Here’s how the quantity of reported referring domains looked in absolute numbers:

What’s the difference between wins and absolute numbers?  Here’s a simple example – let’s imagine we’ve checked how many referring domains each of the analyzed databases reported for the same URL:

• SEO PowerSuite: 100 referring domains
• Ahrefs: 101 referring domains
• Semrush: 99 referring domains
• Majestic: 100 referring domains

In this case, Ahrefs would receive the “point” for their “win.” And then, we repeat the process for the remaining 999,999 domains.

The winner in the referring domains category: SEO PowerSuite

Who wins by the number of backlinks?

For the total number of backlinks, Semrush came out on top with 47.65% wins. It was followed by SEO PowerSuite (21.38% wins) and Ahrefs (16.25% wins). Majestic showed the lowest number of backlinks in this run and gained 14.73% wins.

Total backlinks in absolute numbers:

The winner in the referring backlinks category: Semrush

How positions in the comparison were distributed

Additionally, we have calculated how often each of the vendors occupied the 1^st, 2^nd, 3^rd and 4^th position in the comparison.

This is how the positions were distributed for referring domains:

And this is how the vendors scored for backlinks:

Important factors to keep in mind

All tests have limitations, and this one’s no exception. It’s quite challenging to run research of backlink databases and draw a totally unbiased conclusion.

Below, we’ve outlined a few important aspects that you should keep in mind when you read any backlink index study.

Different types of backlink indexes

Every backlink index is unique since every company uses a different approach to crawl, store and count backlink data. So, with any backlink index comparison, you have this big flaw of analyzing databases that are inherently not equal.

In our research, we’ve accessed the data via APIs that allegedly return the results from the following indexes:

• Ahrefs Live index (all links that are currently live, a link is considered live until recrawled and found otherwise)
• Semrush Fresh index (all links crawled in the last six months)
• Majestic Fresh index (all links crawled in the last four months)
• SEO PowerSuite backlink index includes all live links (similarly to Ahrefs, a link in our index is considered live until we recrawl it and see otherwise).

Unlike Ahrefs and SEO PowerSuite, Semrush and Majestic return both live and dead backlinks, and in most cases, it can make a huge difference in the backlink numbers.

For example, if you check the backlink profile of Amazon.com in Semrush, you’ll see that this domain has a total of over 18 billion backlinks (that’s what you get via the API as well). But if you switch to the Active links mode, you’ll see only seven million links:

Although Ahrefs index includes links that are currently live, you can still spot some inconsistencies. Ahrefs API returns 114 million backlinks for Ahrefs.com, this is close to the number we see in the Summary dashboard, but if we switch to the backlinks report, we’ll see just 35 million links.

There’s probably a good explanation behind this mismatch (low-quality or lost backlinks filtered out?), but it brings us to two important conclusions:

1. We can’t be 100% certain about the type of backlink index used by APIs.
2. We can’t eliminate this type of discrepancy from our research.

Different ways to count links

Yet another problem you’ll face when comparing the backlink indexes is that links are counted differently. For instance, you may calculate duplicate links on a page as a single link or as multiple links. The same’s true for redirects, canonicals, etc. All these things will affect the total calculation, and that’s why we have to take every backlink index comparison with a grain of salt.

Different ways to record IPs

Similarly, there’s a question of how backlink API providers record IP data. Most providers count only the most recent IP they’ve found for the backlink, which means they have one IP recorded per domain. But Semrush seems to record every IP they’ve detected in the last six months; thus, they may have a few dozen IPs recorded for the same domain.

The IP/subnet stats are included in the raw data file, and initially, we planned to use these numbers in the report. However, when we noticed this crucial difference in IP calculation, we considered that these stats could not be used to draw meaningful conclusions.

Different understanding of referring domains

A domain is a domain, right? Our analysis shows that things are a bit more complicated when we talk about backlink indexes.

According to Verisign, “the second quarter of 2021 closed with 367.3 million domains across all top-level domains.”

And while many data providers have a matching number of domains indexed (Ahrefs – 193.4 million referring domains, SEO PowerSuite – 264 million domains), Semrush claims to have a very dubious 1.6 billion referring domains.

Hence, what Semrush counts as a unique domain seem to be different from the widely accepted notion.

To sum it up, there’s probably no winner or loser in this type of comparison, due to all the mentioned factors and varying approaches.

But we believe that the more experiments we run, the more transparent backlink index providers will become about their data in the long run. And that means that enterprise and API users will better understand what they are paying for.

If you’re interested in testing our backlink API for your SEO services and apps (or running your own independent analysis), please drop us a note to api@link-assistant.com, and we’ll set you up with a free trial.

The post Top 4 backlink API vendors compared (1 million domain study) appeared first on Search Engine Land.