A Machine Learning Guide for Average Humans

Posted by alexis-sanders

https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js
//

Machine learning (ML) has grown consistently in worldwide prevalence. Its implications have stretched from small, seemingly inconsequential victories to groundbreaking discoveries. The SEO community is no exception. An understanding and intuition of machine learning can support our understanding of the challenges and solutions Google’s engineers are facing, while also opening our minds to ML’s broader implications.

The advantages of gaining an general understanding of machine learning include:

  • Gaining empathy for engineers, who are ultimately trying to establish the best results for users
  • Understanding what problems machines are solving for, their current capabilities and scientists’ goals
  • Understanding the competitive ecosystem and how companies are using machine learning to drive results
  • Preparing oneself for for what many industry leaders call a major shift in our society (Andrew Ng refers to AI as a “new electricity”)
  • Understanding basic concepts that often appear within research (it’s helped me with understanding certain concepts that appear within Google Brain’s Research)
  • Growing as an individual and expanding your horizons (you might really enjoy machine learning!)
  • When code works and data is produced, it’s a very fulfilling, empowering feeling (even if it’s a very humble result)

I spent a year taking online courses, reading books, and learning about learning (…as a machine). This post is the fruit borne of that labor — it covers 17 machine learning resources (including online courses, books, guides, conference presentations, etc.) comprising the most affordable and popular machine learning resources on the web (through the lens of a complete beginner). I’ve also added a summary of “If I were to start over again, how I would approach it.”

This article isn’t about credit or degrees. It’s about regular Joes and Joannas with an interest in machine learning, and who want to spend their learning time efficiently. Most of these resources will consume over 50 hours of commitment. Ain’t nobody got time for a painful waste of a work week (especially when this is probably completed during your personal time). The goal here is for you to find the resource that best suits your learning style. I genuinely hope you find this research useful, and I encourage comments on which materials prove most helpful (especially ones not included)! #HumanLearningMachineLearning


Executive summary:

Here’s everything you need to know in a chart:

Machine Learning Resource

Time (hours)

Cost ($)

Year

Credibility

Code

Math

Enjoyability

Jason Maye’s Machine Learning 101 slidedeck: 2 years of headbanging, so you don’t have to

2

$0

’17

Credibility level 3

Code level 1

Math level 1

Enjoyability level 5

{ML} Recipes with Josh Gordon Playlist

2

$0

’16

Credibility level 3

Code level 3

Math level 1

Enjoyability level 4

Machine Learning Crash Course

15

$0

’18

Credibility level 4

Code level 4

Math level 2

Enjoyability level 4

OCDevel Machine Learning Guide Podcast

30

$0

’17-

Credibility level 1

Code level 1

Math level 1

Enjoyability level 5

Kaggle’s Machine Learning Track (part 1)

6

$0

’17

Credibility level 3

Code level 5

Math level 1

Enjoyability level 4

Fast.ai (part 1)

70

$70*

’16

Credibility level 4

Code level 5

Math level 3

Enjoyability level 5

Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

20

$25

’17

Credibility level 4

Code level 4

Math level 2

Enjoyability level 3

Udacity’s Intro to Machine Learning (Kate/Sebastian)

60

$0

’15

Credibility level 4

Code level 4

Math level 3

Enjoyability level 3

Andrew Ng’s Coursera Machine Learning

55

$0

’11

Credibility level 5

Code level 2

Math level 4

Enjoyability level 1

iPullRank Machine Learning Guide

3

$0

’17

Credibility level 1

Code level 1

Math level 1

Enjoyability level 3

Review Google PhD

2

$0

’17

Credibility level 5

Code level 4

Math level 2

Enjoyability level 2

Caltech Machine Learning on iTunes

27

$0

’12

Credibility level 5

Code level 2

Math level 5

Enjoyability level 2

Pattern Recognition & Machine Learning by Christopher Bishop

150

$75

’06

Credibility level 5

Code level 2

Math level 5

N/A

Machine Learning: Hands-on for Developers and Technical Professionals

15

$50

’15

Credibility level 2

Code level 3

Math level 2

Enjoyability level 3

Introduction to Machine Learning with Python: A Guide for Data Scientists

15

$25

’16

Credibility level 3

Code level 3

Math level 3

Enjoyability level 2

Udacity’s Machine Learning by Georgia Tech

96

$0

’15

Credibility level 5

Code level 1

Math level 5

Enjoyability level 1

Machine Learning Stanford iTunes by Andrew Ng

25

$0

’08

Credibility level 5

Code level 1

Math level 5

N/A

*Free, but there is the cost of running an AWS EC2 instance (~$70 when I finished, but I did tinker a ton and made a Rick and Morty script generator, which I ran many epochs [rounds] of…)


Here’s my suggested program:

1. Starting out (estimated 60 hours)

Start with shorter content targeting beginners. This will allow you to get the gist of what’s going on with minimal time commitment.

2. Ready to commit (estimated 80 hours)

By this point, learners would understand their interest levels. Continue with content focused on applying relevant knowledge as fast as possible.

3. Broadening your horizons (estimated 115 hours)

If you’ve made it through the last section and are still hungry for more knowledge, move on to broadening your horizons. Read content focused on teaching the breadth of machine learning — building an intuition for what the algorithms are trying to accomplish (whether visual or mathematically).

Your next steps

By this point, you will already have AWS running instances, a mathematical foundation, and an overarching view of machine learning. This is your jumping-off point to determine what you want to do.

You should be able to determine your next step based on your interest, whether it’s entering Kaggle competitions; doing Fast.ai part two; diving deep into the mathematics with Pattern Recognition & Machine Learning by Christopher Bishop; giving Andrew Ng’s newer Deeplearning.ai course on Coursera; learning more about specific tech stacks (TensorFlow, Scikit-Learn, Keras, Pandas, Numpy, etc.); or applying machine learning to your own problems.


Why am I recommending these steps and resources?

I am not qualified to write an article on machine learning. I don’t have a PhD. I took one statistics class in college, which marked the first moment I truly understood “fight or flight” reactions. And to top it off, my coding skills are lackluster (at their best, they’re chunks of reverse-engineered code from Stack Overflow). Despite my many shortcomings, this piece had to be written by someone like me, an average person.

Statistically speaking, most of us are average (ah, the bell curve/Gaussian distribution always catches up to us). Since I’m not tied to any elitist sentiments, I can be real with you. Below contains a high-level summary of my reviews on all of the classes I took, along with a plan for how I would approach learning machine learning if I could start over. Click to expand each course for the full version with notes.


In-depth reviews of machine learning courses:

Starting out

Jason Maye’s Machine Learning 101 slidedeck: 2 years of head-banging, so you don’t have to ↓

{ML} Recipes with Josh Gordon ↓

Google’s Machine Learning Crash Course with TensorFlow APIs ↓

OCDevel’s Machine Learning Guide Podcast ↓

Kaggle Machine Learning Track (Lesson 1) ↓


Ready to commit

Fast.ai (part 1 of 2) ↓

Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems ↓


Broadening your horizons

Udacity: Intro to Machine Learning (Kate/Sebastian) ↓

Andrew Ng’s Coursera Machine Learning Course ↓


Additional machine learning opportunities

iPullRank Machine Learning Guide ↓

Review Google PhD ↓

Caltech Machine Learning iTunes ↓

“Pattern Recognition & Machine Learning” by Christopher Bishop ↓

Machine Learning: Hands-on for Developers and Technical Professionals ↓

Introduction to Machine Learning with Python: A Guide for Data Scientists ↓

Udacity: Machine Learning by Georgia Tech ↓

Andrew Ng’s Stanford’s Machine Learning iTunes ↓


Motivations and inspiration

If you’re wondering why I spent a year doing this, then I’m with you. I’m genuinely not sure why I set my sights on this project, much less why I followed through with it. I saw Mike King give a session on Machine Learning. I was caught off guard, since I knew nothing on the topic. It gave me a pesky, insatiable curiosity itch. It started with one course and then spiraled out of control. Eventually it transformed into an idea: a review guide on the most affordable and popular machine learning resources on the web (through the lens of a complete beginner). Hopefully you found it useful, or at least somewhat interesting. Be sure to share your thoughts or questions in the comments!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Advertisements

Monitoring Featured Snippets – Whiteboard Friday

Posted by BritneyMuller

We’ve covered finding featured snippet opportunities. We’ve covered the process of targeting featured snippets you want to win. Now it’s time for the third and final piece of the puzzle: how to monitor and measure the effectiveness of all your efforts thus far. In this episode of Whiteboard Friday, Britney shares three pro tips on how to make sure your featured snippet strategy is working.

https://fast.wistia.net/embed/iframe/rcgls1y8tz?seo=false&videoFoam=true

https://fast.wistia.net/assets/external/E-v1.js

Monitoring featured snippets

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hey, Moz fans. Welcome to another edition of Whiteboard Friday. Today we are going over part three of our three-part series all about featured snippets. So part one was about how to discover those featured snippet opportunities, part two was about how to target those, and this final one is how to properly monitor and measure the effectiveness of your targeting.

So we’ll jump right in. So there are a couple different steps and things you can do to go through this.

I. Manually resubmit URL and check SERP in incognito

First is just to manually resubmit a URL after you have tweaked that page to target that featured snippet. Super easy to do. All you do is go to Google and you type in “add URL to Google.” You will see a box pop up where you can submit that URL. You can also go through Search Console and submit it manually there. But this just sort of helps Google to crawl it a little faster and hopefully get it reprioritized to, potentially, a featured snippet.

From there, you can start to check for the keyword in an incognito window. So, in Chrome, you go to File > New Incognito. It tends to be a little bit more unbiased than your regular browser page when you’re doing a search. So this way, you’d start to get an idea of whether or not you’re moving up in that search result. So this can be anywhere from, I kid you not, a couple of minutes to months.

So Google tends to test different featured snippets over a long period of time, but occasionally I’ve had experience and I know a lot of you watching have had different experiences where you submit that URL to Google and boom — you’re in that featured snippet. So it really just depends, but you can keep an eye on things this way.

II. Track rankings for target keyword and Search Console data!

But you also want to keep in mind that you want to start also tracking for rankings for your target keyword as well as Search Console data. So what does that click-through rate look like? How are the impressions? Is there an upward trend in you trying to target that snippet?

So, in my test set, I have seen an average of around 80% increase in those keywords, just in rankings alone. So that’s a good sign that we’re improving these pages and hopefully helping to get us more featured snippets.

III. Check for other featured snippets

Then this last kind of pro tip here is to check for other instances of featured snippets. This is a really fun thing to do. So if you do just a basic search for “what are title tags,” you’re going to see Moz in the featured snippet. Then if you do “what are title tags” and then you do a -site:Moz.com, you’re going to see another featured snippet that Google is pulling is from a different page, that is not on Moz.com. So really interesting to sort of evaluate the types of content that they are testing and pulling for featured snippets.

Another trick that you can do is to append this ampersand, &num=1, &num=2 and so forth. What this is doing is you put this at the end of your Google URL for a search. So, typically, you do a search for “what are title tags,” and you’re going to see Google.com/search/? that typical markup. You can do a close-up on this, and then you’re just going to append it to pull in only three results, only two results, only four results, or else you can go longer and you can see if Google is pulling different featured snippets from that different quota of results. It’s really, really interesting, and you start to see what they’re testing and all that great stuff. So definitely play around with these two hacks right here.

Then lastly, you really just want to set the frequency of your monitoring to meet your needs. So hopefully, you have all of this information in a spreadsheet somewhere. You might have the keywords that you’re targeting as well as are they successful yet, yes or no. What’s the position? Is that going up or down?

Then you can start to prioritize. If you’re doing hundreds, you’re trying to target hundreds of featured snippets, maybe you check the really, really important ones once a week. Some of the others maybe are monthly checks.

From there, you really just need to keep track of, “Okay, well, what did I do to make that change? What was the improvement to that page to get it in the featured snippet?” That’s where you also want to keep detailed notes on what’s working for you and in your space and what’s not.

So I hope this helps. I look forward to hearing all of your featured snippet targeting stories. I’ve gotten some really awesome emails and look forward to hearing more about your journey down below in the comments. Feel free to ask me any questions and I look forward to seeing you on our next edition of Whiteboard Friday. Thanks.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

How to Discover and Monitor Bad Backlinks

Posted by rjonesx.

Identifying bad backlinks has become easier over the past few years with better tool sets, bigger link indexes, and increased knowledge, but for many in our industry it’s still crudely implemented. While the ideal scenario would be to have a professional poring over your link profile and combing each link one-by-one for concerns, for many webmasters that’s just too expensive (and, frankly, overkill).

I’m going to walk through a simple methodology using Link Explorer and Excel (although you could do this with Google Sheets just as easily) to combine together the power of Moz Link Explorer, Keyword Explorer Lists, and finally Link Lists to do a comprehensive link audit.

The basics

There are several components involved in determining whether a link is “bad” and should potentially be removed. Ultimately, we want to be able to measure the riskiness of the link (how likely is Google to flag the link as manipulative and how much do we depend on the link for value). Let me address three common factors used by SEOs to determine this score:

Trust metrics:

There are a handful of metrics in our industry that are readily available to help point out concerning backlinks. The two that come to mind most often are Moz Spam Score and Majestic Trust Flow (or, better yet, the difference between Citation Flow and Trust Flow). These two scores actually work quite differently. Moz’s Spam Score predicts the likelihood a domain is banned or penalized based on certain site features. Majestic Trust Flow determines the trustworthiness of a domain or page based on the quality of links pointing to it. While calculated quite differently, the goal is to help webmasters identify which sites are trustworthy and which are not. However, while these are a good starting point, they aren’t sufficient on their own to give you a clear picture of whether a link is good or bad.

Anchor text manipulation:

One of the first things an SEO learns is that using valuable anchor text can help increase your rankings. The very next thing they learn is that using valuable anchor text can bring on a penalty. The reason for this is pretty clear: the likelihood a webmaster will give you valuable anchor text out of the goodness of their heart is very rare, so over-optimization sticks out like a sore thumb. So, how do we measure anchor text manipulation? If we look at anchor text with our own eyes, this seems to be rather intuitive, but there’s a better way to do it in an automated, at-scale fashion that will allow us to better judge links.

Low authority:

Finally, low-authority links — especially when you would expect higher authority based on the domain — are concerning. A good link should come from an internally well-linked page on a site. If the difference between the Domain Authority and Page Authority is very high, it can be a concern. It isn’t a strong signal, but it is one worth looking at. This is especially obvious in certain types of spam, like paginated comment spam or forum profile spam.

So, let’s jump into how we can pull together a quick backlink analysis taking into account these various features of a bad backlink profile. If you’d like to follow along with this tutorial, hop into Link Explorer in another tab:

Follow along with Link Explorer

Step 1: Get the backlink data

The first and easiest step is just to get your backlink data from Link Explorer’s huge backlink index. With nearly 30 trillion links in our index, you can rest assured that we will find most of the bad backlinks with which you should be concerned. To begin, visit the Link Explorer > Inbound Links section and enter in the domain or page which you wish to analyze.

How to Find Bad Backlinks

Because we aren’t concerned with nofollow links, you will want to set the “follow” filter so that we only export followed links. We also aren’t concerned with deleted links, so we can set the Link Status to “Active.”

How to Find Bad Backlinks

Once you have set these filters, hit the “Export” button. You will have a couple of choices. If your site has fewer than 1,000 backlinks, go ahead and choose the immediate download. However, if your link profile is larger, choose the largest setting and be patient for the download to be prepared. We can keep going with other steps of the project in the meantime, but you don’t want to miss out on bad links, which means you need to export them all.

A lot of SEOs will stop at this point. With PA, DA, and Spam Score included in the standard export, you can do a damn good job of finding bad links. Link Explorer does all of that out-of-the-box for you. But for our purposes here, we wan’t to go a step further and do “anchor text qualification.” This is especially valuable for large link profiles.

Step 2: Get anchor text

Getting anchor text out of the new Link Explorer is incredibly simple. Just visit Link Explorer > Anchor Text and hit the Export button. No extra filters will be needed here.

How to Find Bad Backlinks

Step 3: Measure anchor text value

Now here is a quick trick where we can take advantage of Moz Keyword Explorer’s Keyword Lists to find anchor text that appears to be manipulated. First, we want to remove some of the extraneous anchor text which we know absolutely won’t be concerning, such as URLs as anchor text. This step isn’t completely necessary, but will save you some some credits in Moz Keyword Explorer, so it might be worth it.

How to Find Bad Backlinks

After you’ve removed the extraneous anchor text, we’ll just copy and paste our anchor text into a new keyword list for Keyword Explorer.

How to Find Bad Backlinks

By putting the anchor text into Keyword Explorer, we’ll be able to sort anchor text by search volume. It isn’t very common that anchor text happens to have a high search volume, but when webmasters are trying to manipulate search results they often use the keyword for which they’d like to rank in the anchor text. Thus, we can use the search volume of anchor text as a proxy for manipulated anchor text. In fact, when working with Remove’em before I joined Moz, we discovered the anchor text manipulation was the most predictive factor in link penalties.

Step 4: Merge, filter, sort, & model

We will now merge the data (backlinks export and keyword list export) to finally get that list of concerning backlinks. Let’s start with the backlink export. We’ll open it up in Excel and then remove duplicate domain-anchor text pairs.

I’ll start by showing you a quick trick to extract out the domains from a long list of URLs. I copied the list of URLs from the first column to the last column in Excel, and then chose Data > Text to Columns > Delimited > Other > /. This will cause the URLs to be split into different columns wherever the slash occurs, leaving you with the 4th new column being just the domain names.

How to Find Bad Backlinks

Once you have completed this step, we are going to remove duplicate domain-anchor text pairs. Notice that we aren’t going to limit ourselves to one link per domain, which is what many SEOs do. This would be a mistake, since there could be multiple concerning links on the site with different anchor text.

How to Find Bad Backlinks

After choosing Data > Remove Duplicates, I select the column of Anchor Text and the column of Domain. With the duplicates removed, we are now left with the links we want to judge as good or bad. We need one more thing, though. We need to merge in the search volume data we got from Keyword Explorer. Hit the export button on the keyword list you created from anchor text in Keyword Explorer:

How to Find Bad Backlinks

Open up the export and then copy and paste the data into a second sheet in Excel, next to the backlinks sheet you already created and filtered. In this case, I named the two sheets “Raw Data” and “Anchor Text Data”:

How to Find Bad Backlinks

You’ll then want to do a VLOOKUP on the backlinks spreadsheet to create a column with the search volume for the anchor text on each link. I’ve taken a screenshot of the VLOOKUP formula I used, but yours will look a little different depending upon the the names of the sheets and the exact columns you’ve created.

Excel formula: =IF(ISNA(VLOOKUP(C2,'Anchor Text Data'!$A$1:$I$402,3,FALSE)),0,VLOOKUP(C2,'Anchor Text Data'!$1:$I$402,3,FALSE))

=IF(ISNA(VLOOKUP(C2,’Anchor Text Data’!$A$1:$I$402,3,FALSE)),0,VLOOKUP(C2,’Anchor Text Data’!$1:$I$402,3,FALSE))

It looks a little complicated, but that’s simply because I’m using two VLOOKUPs simultaneously to replace N/A results with the number 0. You can always manually put in 0 wherever N/A shows up.

Now it’s time for the fun part: modeling. First, I recommend sorting by the volume column you just created just so you can see the most concerning anchor text at the top. It’s amazing to see links with anchor text like “ring” or “jewelry” automatically populate at the top of the list, since they’re also keywords with high search volume.

How to Find Bad Backlinks

Second, we’ll create a new column with a formula that takes into account the quality of the link, the riskiness of the anchor text, and the Spam Score:

Excel formula: =D11+(F11-E11)+(LOG(G11+1)*10)+(LOG(O11+1)*10)

=D11+(F11-E11)+(LOG(G11+1)*10)+(LOG(O11+1)*10)

Let’s break down that formula real quickly:

  • D11: This is simply the Spam Score
  • (F11-E11): This is the Domain Authority minus the Page Authority. (This is a bit debatable — some people might just prefer to choose 100-E11)
  • (Log(G11+1)*10): This is a fancy way of converting the number of times this anchor text link occurs into a consistent number for our equation. Without taking the log(), having a high number here could overcome the other signals.
  • (Log(O11+1)*10): This is a fancy way of converting the search volume to a number consistent for our equation. Without taking the log(), having a high search volume could also overcome other signals.

Once we run this equation and create a new column, we can sort by “Riskiness” and find the links with which we should be most concerned.

How to Find Bad Backlinks

As you can see, examples of comment spam and paid links popped to the top of the list because the formula gives a higher value to low-quality, spammy links with risky anchor text. But wait, there’s more!

Step 5: Build a Link List

Link Explorer doesn’t just leave you hanging after doing analysis. Our goal is to help you do SEO, not just analyze it. Your next step is to start a new Link List.

The Link List feature allows you to track whether certain links are alive. If you embark on a campaign to try and remove some of these spammier links, you can create a Link List and use it to monitor the status of those links. Just create a new list by naming it, adding your domain, and then copying and pasting the concerning links.

How to Find Bad Backlinks

You can now just monitor the Link List as you do your outreach to remove bad links. The Link List will track all the metrics, including whether the link has been removed.

How to Find Bad Backlinks

Wrapping up

Whether you want to do a cursory backlink audit by just looking at Spam Score and PA, or a deep-dive taking into account anchor text qualification, Link Explorer + Keyword Explorer and Link Lists make it possible. With our greatly improved backlink index, you can now rest assured that the data you need is right at your finger tips and, if you need to get down-and-dirty in Excel, you can readily export it to do deeper analysis.

Find your spammy links!

Good luck hunting bad backlinks!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

How to Optimize Your Google My Business Listing [Updated May 1, 2018]

Posted by sherrybonelli

Updated May 1, 2018

An important first step in any local SEO strategy is to claim and verify your local business’ Google My Business (GMB) listing. Getting on Google My Business can increase your chances of showing up in Google’s Local Pack, Local Finder, Google Maps, and organic rankings in general. Qualifying local businesses can claim this free listing on Google and include information about their company, like their address, phone number, business hours, and types of payments accepted.

Additionally, over the past several months, Google has added some great features to Google My Business that companies should take advantage of that enhances your Google My Business listing and helps to grab viewers’ attention — and can increase how you rank in local search results.

If you haven’t claimed and verified your Google My Business Listing yet, that’s the first step. To get started, visit https://www.google.com/business.

How to optimize your Google My Business listing

Many local businesses just claim their GMB listing and forget about it. What most businesses don’t realize is that there are a variety of other features Google gives you that you can use to optimize your Google My Business listing and several reasons why you should frequently check your business listing to ensure that its accuracy stays intact. Want to know more?

Complete all the information Google asks for

There are a variety of questions Google wants you to fill out to complete your Google My Business profile. When done, your listing will have valuable basic data that will make it easier for potential customers to find more information about your company. And if you don’t fill out that information, someone else could. Many business owners don’t realize that anyone can suggest a change (or “edit”) to your business listing — and that includes your competitors.

When a searcher clicks on your GMB listing they see a “Suggest an edit” option:

How to optimize your Google My Business listing

When someone clicks on that option they can literally edit your Google My Business listing (and make some pretty dramatic changes, too):

How to optimize your Google My Business listing

And these aren’t just “suggested” edits — these user-generated changes can actually be made live on your listing without you even being notified. This is just one reason why it’s very important that you log in to your Google My Business dashboard regularly to ensure that no one has made any unwanted changes to your listing.

Here’s how:

If you log in to Google My Business, you can switch back to the “Classic” dashboard here:

How to optimize your Google My Business listing

In the classic dashboard, you might see “Google Updates” notifications.

If you see updates, these are changes that Google made to your business listing because either their algorithm found new information about your business (perhaps from another directory/citation site or a change they found on your Google Map) or a Google user submitted an edit that was published. (Yes, when people make “suggested edits,” they are not really “suggestions” -– the changes are often made live without you ever getting a notification or the opportunity to dispute the change!)

When you click on “Google Updates,” you’ll see a box that allows you to “Review Updates.” It’s here where you’re given the opportunity to remove incorrect information that may have been made by a troublesome Google user.

Now, Google supposedly sends out emails to the owner and others managing your Google My Business account when changes are made, but oftentimes those people never receive notifications about changes to their listing. So beware: you may (or may not) be notified by Google if changes are made to your listing. (For example, your business category could be changed from “criminal attorney” to the generic “lawyer” category, which could negatively impact your search rankings.) That’s why it’s extra important for you to log in and check your listing frequently (especially when, quite literally, some businesses have had their address and website URLs changed in their GMB listing by nefarious users.)

If you see a change that is incorrect and you have difficulty changing it (like a bogus review, for instance), create a new post explaining the situation in detail in the Google My Business forum and reach out to one of the Google Top Contributor volunteers for help.

Also, it’s important to realize that Google encourages people who are familiar with your business to answer questions, so that Google can learn more information about your company. To do this they simply click on the “Know this place? Answer quick questions” link.

How to optimize your Google My Business listing

They’ll then be prompted to answer some questions about your business:

How to optimize your Google My Business listing

If the person knows the answer to the question, they can answer and then they’ll typically be asked another question. If not, they can decline.

Now, some business owners have cried foul, saying that competitors or others with malicious intent can wreak havoc on their Google My Business listings with these features. However, Google’s philosophy is that this type of user-generated content helps to build a community, more fully completes a business’ profile, and allows Google to experiment with different search strategies.

Just remember, after you get your Google My Business listing verified, continue to check your listing regularly to be on the safe side.

Once you have your GMB listing verified, now is the time to optimize your listing. (This is where you have a greater chance to outdo your competition!)

Google My Business Posts

Google Posts are almost like “mini-ads” or “social media posts” that show up in Google search in your Google My Business listing (in the Knowledge Panel and on Google Maps).

How to optimize your Google My Business listing

To get started with Posts, log in to your GMB dashboard and you’ll see the Posts option on the left-hand side:

How to optimize your Google My Business listing

You can have fun with your Google My Business Posts by adding an image, a call-to-action (CTA), and even including a link to another page or website. If you’re using Yext, you can create GMB Posts directly from your Yext dashboard.

Not sure what type of Post you should make? Here are just a few Post ideas:

  • If you’re having an event (like a webinar or a seminar about your chiropractic practice) you can set up an event Post with a date and time, then add a link to the registration page.
  • Do you have a sale going on during a specific time? Create a “sale” event Post.
  • Does your latest blog post rock? Add a short description and link to the post on your blog.
  • New product you want to feature? Show a picture of this cool gadget and link to where people can make the purchase.
  • Want to spread holiday joy? Give potential customers a holiday message Post.

The possibilities with Posts are endless! Posts show up prominently in your business’ Knowledge Panel, so don’t miss this opportunity to stand out.

TIP: To grab a searcher’s attention, you want to include an image in your Post, but on Google Maps the Post image can get cut off. You might have to test a few Post image sizes to make sure it’s sized appropriately for Maps and the Knowledge Panel on desktop and mobile devices.

Want to have even MORE fun and potentially help your local SEO? Try adding relevant emojis to your Post. Google is beginning to index emoji-relevant search results. (In fact, you can now search Google by “tweeting” an emoji at it!) Additionally, people — especially younger people — are beginning to search (typically on their mobile devices) with emojis! So if a person is searching for “[pizza emoji] + nearby” and you own a local pizza restaurant and use the [pizza emoji] somewhere on your Google My Business listing — like in a Post with a special offer on a pizza order — you might have an SEO edge over the other pizzeria competitors in your city.

Not sure how to add emojis? If you’re using a Windows computer, you can add emojis by pressing the Windows key + the “.” OR “;” key at the same time on your keyboard. The emoji list of characters will appear and you can select the emoji you’d like to include (but don’t get carried away — one emoji is enough):

How to optimize your Google My Business listing

When people search using Chrome on their smartphones with an “emoji + near me,” you might be surprised by what they find:

How to optimize your Google My Business listing

You got it! Google knew that I was looking for a great burger joint around my home! (Pretty cool, huh?)

Disclaimer: This strategy is still new and we’re not certain how adding emojis to your GMB listings impact these “emoji search results,” but if you have a related emoji that is pertinent to your business, you should definitely test it! (But don’t overdo the emojis — it gets obnoxious and doesn’t look professional if you go overboard.)

Posts stay live for seven days or “go dark” after the date of the event. (However, the old Posts still appear in your GMB listing — they’re just pushed down by the new Posts.)

How to optimize your Google My Business listing

If you’re forgetful, Google is great about sending you reminders when it’s time to create a new Post.

How to optimize your Google My Business listing

And remember, Posts show up prominently in mobile searches, so make your website stand out among search results by keeping your Posts “topped off.”

How to optimize your Google My Business listing

It’s important to note that at this time, hotels and B&Bs are not allowed to make Posts. That may change sometime in the future, so stay tuned!

Booking button feature

Google’s Booking button feature can really help your business stand out from the crowd. If you have any type of business that relies on customers making appointments and you’re using integrated scheduling software, people can now book an appointment with your business directly from your Google My Business listing. This can make it even easier to get new customers — they don’t have to leave Google to book an appointment with you!

How to optimize your Google My Business listing

If you have an account with one of Google’s supported scheduling providers, the booking button is automatically added to your Google My Business listing. Take advantage of this integrated Google My Business feature if you use the booking providers, it’ll make it super simple to get new clients or customers.

Messaging

Did you know that you customers — and potential customers — can send you text messages? This is a great way to connect directly with people interested in what you have to offer, and a great way to engage with people looking at your GMB listing (and you know that Google is always watching engagement.)

To get started with Messaging, log in to your GMB dashboard and click on “Messaging”:

How to optimize your Google My Business listing

You can then set up the message people will receive after they send you a message and your mobile phone number.

How to optimize your Google My Business listing

If you don’t want text messages sent to your personal phone number, you can download Google’s Allo app. When you set up your Allo account, use the same phone number connected to your Google My Business account. Now when someone messages you, the message will be sent to the Allo app instead of appearing alongside your personal text messages.

The Allo app is a great way to keep your personal and business text messages separate:

How to optimize your Google My Business listing

This feature is still in its infancy, though. Right now, messaging is only available to mobile web users and is not available to mobile app or desktop users. People also won’t see the Messaging option in the Knowledge Panel or on Google Maps.

The ONLY way someone can message your business is if they perform a mobile web search on Chrome. (I expect that Google will expand the Messaging feature once they work the kinks out.)

Questions & Answers

Questions & Answers is a great feature for Google local search. It’s very cool! Just like it sounds, Q&A allows people to ask questions about your business and you can answer those questions.

How to optimize your Google My Business listing

The Google My Business Questions & Answers feature is the perfect opportunity to hear directly from “the people” and you can respond to them. Win-win. However, according to a study done by Get Five Stars, 25 percent of locations on Google Maps have questions (and many of those questions are probably STILL unanswered).

Here are a few things to keep in mind about Questions & Answers:

  • On mobile devices, you can see, ask and answer questions on Google Maps on Android devices and when you search for your business on mobile browsers on both iPhone and Android devices. To use Google Maps on your Android device, download the Google Maps app and sign in with the email address you use for your GMB listing.

How to optimize your Google My Business listing

Ironically, you can’t see Questions and Answers on the Google My Business app.

  • No notifications of new questions show up in your GMB dashboard. To find out if you have new questions that need answering, you need to install Google Maps on your phone, log in, and check for questions/notifications. You can also go on a mobile browser, search for your business, and see if you have new questions that need to be answered.
  • Google has recently started sending out email notifications letting you know that a new question has been asked, but it’s possible that not everyone associated with your account receives these emails:

How to optimize your Google My Business listing

This email notification is a BIG improvement over the lack of notification we’ve experienced so far with Q&As.

One thing you should do is be proactive and create a Frequently Asked Questions list to preempt people’s GMB Q&As. Check with your sales reps and your customer service staff to identify the questions people most often ask, then put those Q&A questions on your GMB listing.

TIP: Google has said that upvoting questions can make them more visible. If someone has a particularly important question, go ahead and upvote it.

WARNING: It’s important to note that just like “Suggest an Edit” on GMB, anyone can answer questions asked of your business. Therefore, you want to keep an eye out and make sure you answer questions quickly and ensure that if someone else answers a question, that the answer is accurate. If you find that someone is abusing your GMB listing’s Q&A feature, reach out to the Google My Business support forums.

Still have questions about Google Questions & Answers? You can read Google’s Q&A guidelines.

Google My Business online reviews

Unlike Yelp, which vehemently discourages business owners to ask their customers for reviews, Google encourages business owners to ethically ask their customers or clients for online reviews. (Yelp takes it to the extreme, in my opinion.) Online reviews appear next to your listing in Google Maps and your business’ Knowledge Panel in search results. Online reviews can help your business stand out among a sea of search results.

Additionally, online reviews are known to impact search result rankings, consumer trust, and click-through rates. According to BrightLocal’s 2017 Consumer Review Survey:

  • 97% of consumers read online reviews for local businesses in 2017, with 12% looking for a local business online every day
  • 85% of consumers trust online reviews as much as personal recommendations
  • Positive reviews make 73% of consumers trust a local business more
  • 49% of consumers need at least a four-star rating before they choose to use a business
  • Responding to reviews is more important than ever, with 30% naming this as key when judging local businesses
  • 68% of consumers left a local business review when asked — with 74% having been asked for their feedback
  • 79% of consumers have read a fake review in the last year

If you follow Google’s guidelines for Google My Business reviews, you can ask your customers for reviews. (However, if you violate any of these policies, your reviews could be removed.)

Recently Google made some changes to their review guidelines. They have now changed it so that current and/or former employees can’t leave reviews. For business owners this is great news because it means that disgruntled and ex-employees with a grudge can no longer post bad reviews. Here is the new section that deals with Conflict of Interest:

How to optimize your Google My Business listing

Additionally, Google made some changes with regard to reputation marketing software. Reputation marketing software can help filter out people who were planning on leaving negative reviews so that they aren’t given the opportunity to leave that bad review online. (This is sometimes referred to as “review gating.”) Google wants to prevent that practice, so on April 12, 2018, Google updated their review policy to include information on this. In general, you don’t want to “Discourage or prohibit negative reviews or selectively solicit positive reviews from customers.”

Also, whatever you do, do not offer a bribe in exchange for a review. Not only does it go against Google’s terms, it goes against the laws of reviews in general: do you really want to bribe someone to leave you a good review — or do you want to earn it?

When customers leave reviews for you — good or bad — make sure you respond to them. Not only does it show that customer that you appreciate their feedback, it also shows potential customers that you care.

So what happens if you get a negative review? First, don’t freak out. Everybody has a bad day and most people recognize that. Also, if you have a troll that gave you a one-star review and left a nasty comment, most people with common sense recognize that review for what it is. It’s generally not worth stressing over.

TIP: Asking someone to leave a review on Google is very cumbersome. To give your customers a direct link to your Google My Business listing so they can leave a review online for you, read and follow the directions in this post on How to Create a Direct Review Link to Your Google My Business Listing.

To learn more about strategically getting more online reviews, check out this article from Moz.

Photos and videos

The Internet used to be all about text and information, but more and more the visual appeal of the Internet is what grabs people’s attention — and that means photos and videos. Videos are so hot that you don’t even need sound. Studies show that as much as 85% of Facebook videos are viewed with the sound off.

However, many business owners are still under the misperception that to get into videos (or even photography) you have to hire a professional video production company or studio. Not true. Some of the best photos and videos are done on the fly — and with a smartphone!

Adding photos of your business is a great way to humanize your brand and let your customers get a “behind-the-scenes” look at what your company is all about… AND your customers can post photos on your Google My Business listing, too! (Surprise!)

AGENCY TIP: If you’re optimizing Google My Business listings for your clients, you know how difficult it is to get pictures from them so you can add them to their GMB listing. (Your clients are busy and often hard to track down.) There’s a new tool called localPics that solves that problem. This tool makes it super simple to send your clients text message reminders that it’s time to upload pictures. The owner (or whoever the designated “photographer” is) simply takes pictures or goes into their phone’s photo gallery, selects the pictures they want to upload, and the pictures are automatically uploaded to their Google My Business listing! What could be easier?

How to optimize your Google My Business listing

The ability to add photos to your Google My Business listing has been around for a while, but adding videos is a relatively new feature that Google introduced. Instead of being afraid, get excited! You can now add a 30-second video about your company that will grab people’s attention on the most popular place people go to search and find information: Google!

To get started, log in to your Google My Business dashboard. You will either see the “Add Videos” image on the Overview tab:

How to optimize your Google My Business listing

Or you can also click on the blue + sign to add a video:

How to optimize your Google My Business listing

When you click on the “Add Video” button, you can either drag the video you want to upload or select the video from your computer.

How to optimize your Google My Business listing

It’s super simple!

Google states that it can take up to 24 hours for the video to display, but most videos show up after just a few minutes. The videos should be 30 seconds long, but we’ve uploaded longer videos just fine. (Keep in mind that people have short attention spans, so don’t overdo it with videos that are too long — 30 seconds is just about right!)

Now, for you marketers out there that are salivating thinking of the great marketing and promotional videos you can upload, hold on for just a moment. Make sure your videos are taken at the place of business and are of people that work at your business or directly pertain to your business. (Google My Business is not the place for stock photos and marketing or promotional videos.) Google can remove the videos if the primary subject of the content is not related to the business location.

Owners who upload videos will be shown in the “By Owner” tab. When customers or clients upload videos, those videos will appear in the “Customer” tab. ALL of the videos will be displayed in the “Video” tab.

Google has given us some general Google My Business Video Guidelines to follow:

  • Duration: Up to 30 seconds long
  • File Size: Up to 100 MB
  • Resolution: 720p or higher

As a bonus, once you have two or more videos on your GMB listing, you’ll get a Videos subtab that shows up on mobile devices!

Business descriptions

Good news! Google now allows business owners to include a business description on your Google My Business listing. (And it’s about time!) Google recently made this announcement via Twitter and business owners were thrilled.

How to optimize your Google My Business listing

As usual, Google has provided us with some guidelines to follow: Google Business Description Guidelines. It’s important you adhere to these rules because Google does review your business Description.

How to optimize your Google My Business listing

You’re allowed 750 characters in your business description, but only 250 characters show up before they get cut off in the Knowledge Panel. So you want to make sure that you carefully create your business description and put the most important information and keywords — including your city — towards the front of the description.

Google really does review your business description to make sure people aren’t being deceptive or are spamming, so be sure to follow these guidelines:

How to optimize your Google My Business listing

You only have 750 characters (and only 250 of those show up in the company’s Knowledge Panel), so you want to make sure that every character counts.

On a desktop computer, the business description appears in the Knowledge Panel towards the bottom, below your reviews. (It’d be great if Google would bump the business description up towards the top of the Knowledge Panel where it should be… Let’s hope they move it there soon!)

How to optimize your Google My Business listing

On a mobile device, you can only see a business’ description if you click on the About tab:

How to optimize your Google My Business listing

Services/Menus

If you sell services, like a spa, nail salon, hair salon, copying company, or even a holistic center, and have a “menu” of services, the new Services list in Google My Business is a great new addition. This feature is only available for food and drink, health, beauty, and other services businesses that don’t have a third-party “menu” link.

The Services list allows you to categorize and list out all your services (or food items) and prices so that potential customers can easily see what you have to offer.

This list itemizes out each service (or food item) you offer. To get started, log in to your Google My Business listing and click on Info:

How to optimize your Google My Business listing

Then scroll down and you will see the “Services” section where you can Add or edit your items:

How to optimize your Google My Business listing

This is where you can create categories, add items, and you can also add a description of each item (if you want to):

How to optimize your Google My Business listing

If you own a service business with set prices, I’d highly recommend you include your list of services and make sure you update these services and prices if things change.


Get more out of your GMB listing

Google is always looking at the engagement searchers and you, as the owner, are having with your Google My Business listing. The more interaction, the better your chances of ranking higher in the local three-pack and organic rankings in general. That means you need to keep optimizing your Google My Business listing.

As new features come out, plan on using them to keep your GMB listing fully optimized.

TECHIE TIP: If you’re managing multiple listings or franchises, you can use Google’s API v4.1 to more easily add Google My Business descriptions and Offer Posts. And if you’re really techie, you can even add “customer media endpoints” that allow users to retrieve photos and videos uploaded by customers at their business (normally GMB users aren’t notified of photo and video uploads).

Google has even introduced a new notification that alerts users who have opted in to receive alerts about newly posted media on their Google My Business Locations. Wow! (If you have someone on your team that can code, you’re at an advantage!)

Hopefully these features have given you a new reason to login to your Google My Business account and get busy! If you have any other questions about optimizing your GMB listing, let me know in the comments.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz’s Link Data Used to Suck… But Not Anymore! The New Link Explorer is Here – Whiteboard Friday

Posted by randfish

Earlier this week we launched our brand-new link building tool, and we’re happy to say that Link Explorer addresses and improves upon a lot of the big problems that have plagued our legacy link tool, Open Site Explorer. In today’s Whiteboard Friday, Rand transparently lists out many of the biggest complaints we’ve heard about OSE over the years and explains the vast improvements Link Explorer provides, from DA scores updated daily to historic link data to a huge index of almost five trillion URLs.

https://fast.wistia.net/embed/iframe/tw8djfqtem?seo=false&videoFoam=true

https://fast.wistia.net/assets/external/E-v1.js

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week I’m very excited to say that Moz’s Open Site Explorer product, which had a lot of challenges with it, is finally being retired, and we have a new product, Link Explorer, that’s taking its place. So let me walk you through why and how Moz’s link data for the last few years has really kind of sucked. There’s no two ways about it.

If you heard me here on Whiteboard Friday, if you watched me at conferences, if you saw me blogging, you’d probably see me saying, “Hey, I personally use Ahrefs, or I use Majestic for my link research.” Moz has a lot of other good tools. The crawler is excellent. Moz Pro is good. But Open Site Explorer was really lagging, and today, that’s not the case. Let me walk you through this.

The big complaints about OSE/Mozscape

1. The index was just too small

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

Mozscape was probably about a fifth to a tenth the size of its competitors. While it got a lot of the quality good links of the web, it just didn’t get enough. As SEOs, we need to know all of the links, the good ones and the bad ones.

2. The data was just too old

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So, in Mozscape, a link that you built on November 1st, you got a link added to a website, you’re very proud of yourself. That’s excellent. You should expect that a link tool should pick that up within maybe a couple weeks, maybe three weeks at the outside. Google is probably picking it up within just a few days, sometimes hours.

Yet, when Mozscape would crawl that, it would often be a month or more later, and by the time Mozscape processed its index, it could be another 40 days after that, meaning that you could see a 60- to 80-day delay, sometimes even longer, between when your link was built and when Mozscape actually found it. That sucks.

3. PA/DA scores took forever to update

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

PA/DA scores, likewise, took forever to update because of this link problem. So the index would say, oh, your DA is over here. You’re at 25, and now maybe you’re at 30. But in reality, you’re probably far ahead of that, because you’ve been building a lot of links that Mozscape just hasn’t picked up yet. So this is this lagging indicator. Sometimes there would be links that it just didn’t even know about. So PA and DA just wouldn’t be as accurate or precise as you’d want them to be.

4. Some scores were really confusing and out of date

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

MozRank and MozTrust relied on essentially the original Google PageRank paper from 1997, which there’s no way that’s what’s being used today. Google certainly uses some view of link equity that’s passed between links that is similar to PageRank, and I think they probably internally call that PageRank, but it looks nothing like what MozRank was called.

Likewise, MozTrust, way out of date, from a paper in I think 2002 or 2003. Much more advancements in search have happened since then.

Spam score was also out of date. It used a system that was correlated with what spam looked like three, four years ago, so much more up to date than these two, but really not nearly as sophisticated as what Google is doing today. So we needed to toss those out and find their replacements as well.

5. There was no way to see links gained and lost over time

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

Mozscape had no way to see gained and lost links over time, and folks thought, “Gosh, these other tools in the SEO space give me this ability to show me links that their index has discovered or links they’ve seen that we’ve lost. I really want that.”

6. DA didn’t correlate as well as it should have

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So over time, DA became a less and less indicative measure of how well you were performing in Google’s rankings. That needed to change as well. The new DA, by the way, much, much better on this front.

7. Bulk metrics checking and link reporting was too hard and manual

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So folks would say, “Hey, I have this giant spreadsheet with all my link data. I want to upload that. I want you guys to crawl it. I want to go fetch all your metrics. I want to get DA scores for these hundreds or thousands of websites that I’ve got. How do I do that?” We didn’t provide a good way for you to do that either unless you were willing to write code and loop in our API.

8. People wanted distribution of their links by DA

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

They wanted distributions of their links by domain authority. Show me where my links come from, yes, but also what sorts of buckets of DA do I have versus my competition? That was also missing.

So, let me show you what the new Link Explorer has.

Moz's new Link Explorer

Click on the whiteboard image above to open a high-resolution version in a new tab!

Wow, look at that magical board change, and it only took a fraction of a second. Amazing.

What Link Explorer has done, as compared to the old Open Site Explorer, is pretty exciting. I’m actually very proud of the team. If you know me, you know I am a picky SOB. I usually don’t even like most of the stuff that we put out here, but oh my god, this is quite an incredible product.

1. Link Explorer has a GIANT index

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So I mentioned index size was a big problem. Link Explorer has got a giant index. Frankly, it’s about 20 times larger than what Open Site Explorer had and, as you can see, very, very competitive with the other services out there. Majestic Fresh says they have about a trillion URLs from their I think it’s the last 60 days. Ahrefs, about 3 trillion. Majestic’s historic, which goes all time, has about 7 trillion, and Moz, just in the last 90 days, which I think is our index — maybe it’s a little shorter than that, 60 days — 4.7 trillion, so almost 5 trillion URLs. Just really, really big. It covers a huge swath of the web, which is great.

2. All data updates every 24 hours

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So, unlike the old index, it is very fresh. Every time it finds a new link, it updates PA scores and DA scores. The whole interface can show you all the links that it found just yesterday every morning.

3. DA and PA are tracked daily for every site

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

You don’t have to track them yourself. You don’t have to put them into your campaigns. Every time you go and visit a domain, you will see this graph showing you domain authority over time, which has been awesome.

For my new company, I’ve been tracking all the links that come in to SparkToro, and I can see my DA rising. It’s really exciting. I put out a good blog post, I get a bunch of links, and my DA goes up the next day. How cool is that?

4. Old scores are gone, and new scores are polished and high quality

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So we got rid of MozRank and MozTrust, which were very old metrics and, frankly, very few people were using them, and most folks who were using them didn’t really know how to use them. PA basically takes care of both of them. It includes the weight of links that come to you and the trustworthiness. So that makes more sense as a metric.

Spam score is now on a 0 to 100% risk model instead of the old 0 to 17 flags and the flags correlate to some percentage. So 0 to 100 risk model. Spam score is basically just a machine learning built model against sites that Google penalized or banned.

So we took a huge amount of domains. We ran their names through Google. If they couldn’t rank for their own name, we said they were penalized. If we did a site: the domain.com and Google had de-indexed them, we said they were banned. Then we built this risk model. So in the 90% that means 90% of sites that had these qualities were penalized or banned. 2% means only 2% did. If you have a 30% spam score, that’s not too bad. If you have a 75% spam score, it’s getting a little sketchy.

5. Discovered and lost links are available for every site, every day

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So again, for this new startup that I’m doing, I’ve been watching as I get new links and I see where they come from, and then sometimes I’ll reach out on Twitter and say thank you to those folks who are linking to my blog posts and stuff. But it’s very, very cool to see links that I gain and links that I lose every single day. This is a feature that Ahrefs and Majestic have had for a long time, and frankly Moz was behind on this. So I’m very glad that we have it now.

6. DA is back as a high-quality leading indicator of ranking ability

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

So, a note that is important: everyone’s DA has changed. Your DA has changed. My DA has changed. Moz’s DA changed. Google’s DA changed. I think it went from a 98 to a 97. My advice is take a look at yourself versus all your competitors that you’re trying to rank against and use that to benchmark yourself. The old DA was an old model on old data on an old, tiny index. The new one is based on this 4.7 trillion size index. It is much bigger. It is much fresher. It is much more accurate. You can see that in the correlations.

7. Building link lists, tracking links that you want to acquire, and bulk metrics checking is now easy

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

Building link lists, tracking links that you want to acquire, and bulk metrics checking, which we never had before and, in fact, not a lot of the other tools have this link tracking ability, is now available through possibly my favorite feature in the tool called Link Tracking Lists. If you’ve used Keyword Explorer and you’ve set up your keywords to watch those over time and to build a keyword research set, very, very similar. If you have links you want to acquire, you add them to this list. If you have links that you want to check on, you add them to this list. It will give you all the metrics, and it will tell you: Does this link to your website that you can associate with a list, or does it not? Or does it link to some page on the domain, but maybe not exactly the page that you want? It will tell that too. Pretty cool.

8. Link distribution by DA

Moz's Link Data Used to Suck... But Not Anymore! The New Link Explorer is Here - Whiteboard Friday

Finally, we do now have link distribution by DA. You can find that right on the Overview page at the bottom.

Look, I’m not saying Link Explorer is the absolute perfect, best product out there, but it’s really, really damn good. I’m incredibly proud of the team. I’m very proud to have this product out there.

If you’d like, I’ll be writing some more about how we went about building this product and a bunch of agency folks that we spent time with to develop this, and I would like to thank all of them of course. A huge thank you to the Moz team.

I hope you’ll do me a favor. Check out Link Explorer. I think, very frankly, this team has earned 30 seconds of your time to go check it out.

Try out Link Explorer!

All right. Thanks, everyone. We’ll see you again for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Efficient Link Reclamation: How to Speed Up & Scale Your Efforts

Posted by DarrenKingman

Link reclamation: Tools, tools everywhere

Every link builder, over time, starts to narrow down their favorite tactics and techniques. Link reclamation is pretty much my numero-uno. In my experience, it’s one of the best ROI activities we can use for gaining links particularly to the homepage, simply because the hard work — the “mention” (in whatever form that is) — is already there. That mention could be of your brand, an influencer who works there, or a tagline from a piece of content you’ve produced, whether it’s an image asset, video, etc. That’s the hard part. But with it done, and after a little hunting and vetting the right mentions, you’re just left with the outreach.

Aside from the effort-to-return ratio, there are various other benefits to link reclamation:

  1. It’s something you can start right away without assets
  2. It’s a low risk/low investment form of link building
  3. Nearly all brands have unlinked mentions, but big brands tend to have the most and therefore see the biggest routine returns
  4. If you’re doing this for clients, they get to see an instant return on their investment

Link reclamation isn’t a new tactic, but it is becoming more complex and tool providers are out there helping us to optimize our efforts. In this post, I’m going to talk a little about those tools and how to apply them to speed up and scale your link reclamation.

Finding mentions

Firstly, we want to find mentions. No point getting too fancy at this stage, so we just head over to trusty Google and search for the range of mentions we’re working on.

As I described earlier, these mentions can come in a variety of shapes and sizes, so I would generally treat each type of mention that I’m looking for as a separate project. For example, if Moz were the site I was working on, I would look for mentions of the brand and create that as one “project,” then look for mentions of Followerwonk and treat that as another, and so on. The reasons why will become clear later on!

So, we head to the almighty Google and start our searches.

To help speed things up it’s best to expand your search result to gather as many URLs as you can in as few clicks as possible. Using Google’s Search Settings, you can quickly max out your SERPs to one hundred results, or you can install a plugin like GInfinity, which allows you to infinitely scroll through the results and grab as many as you can before your hand cramps up.

Now we want to start copying as many of these results as possible into an Excel sheet, or wherever it is you’ll be working from. Clicking each one and copying/pasting is hell, so another tool to quickly install for Chrome is Linkclump. With this one, you’ll be able to right click, drag, and copy as many URLs as you want.

Linkclump Pro Tip: To ensure you don’t copy the page titles and cache data from a SERP, head over to your Linkclump settings by right-clicking the extension icon and selecting “options.” Then, edit your actions to include “URLs only” and “copied to clipboard.” This will make the next part of the process much easier!

Filtering your URL list

Now we’ve got a bunch of URLs, we want to do a little filtering, so we know a) the DA of these domains as a proxy metric to qualify mentions, and b) whether or not they already link to us.

How you do this bit will depend on which platforms you have access to. I would recommend using BuzzStream as it combines a few of the future processes in one place, but URL Profiler can also be used before transferring your list over to some alternative tools.

Using BuzzStream

If you’re going down this road, BuzzStream can pretty much handle the filtering for you once you’ve uploaded your list of URLs. The system will crawl through the URLs and use their API to display Domain Authority, as well as tell you if the page already links to you or not.

The first thing you’ll want to do is create a “project” for each type of mention you’re sourcing. As I mentioned earlier this could be “brand mentions,” “creative content,” “founder mentions,” etc.

When adding your “New Project,” be sure to include the domain URL for the site you’re building links to, as shown below. BuzzStream will then go through and crawl your list of URLs and flag any that are already linking to you, so you can filter them out.

Next, we need to get your list of URLs imported. In the Websites view, use Add Websites and select “Add from List of URLs”:

The next steps are really easy: Upload your list of URLs, then ensure you select “Websites and Links” because we want BuzzStream to retrieve the link data for us.

Once you’ve added them, BuzzStream will work through the list and start displaying all the relevant data for you to filter through in the Link Monitoring tab. You can then sort by: link status (after hitting “Check Backlinks” and having added your URL), DA, and relationship stage to see if you/a colleague have ever been in touch with the writer (especially useful if you/your team uses BuzzStream for outreach like we do at Builtvisible).

Using URL Profiler

If you’re using URL Profiler, firstly, make sure you’ve set up URL Profiler to work with your Moz API. You don’t need a paid Moz account to do this, but having one will give you more than 500 checks per day on the URLs you and the team are pushing through.

Then, take the list of URLs you’ve copied using Linkclump from the SERPs (I’ve just copied the top 10 from the news vertical for “moz.com” as my search), then paste the URLs in the list. You’ll need to select “Moz” in the Domain Level Data section (see screenshot) and also fill out the “Domain to Check” with your preferred URL string (I’ve put “Moz.com” to capture any links to secure, non-secure, alternative subdomains and deeper level URLs).

Once you’ve set URL Profiler running, you’ll get a pretty intimidating spreadsheet, which can simply be cut right down to the columns: URL, Target URL and Domain Mozscape Domain Authority. Filter out any rows that have returned a value in the Target URL column (essentially filtering out any that found an HREF link to your domain), and any remaining rows with a DA lower than your benchmark for links (if you work with one).

And there’s my list of URLs that we now know:

1) don’t have any links to our target domain,

2) have a reference to the domain we’re working on, and

3) boast a DA above 40.

Qualify your list

Now that you’ve got a list of URLs that fit your criteria, we need to do a little manual qualification. But, we’re going to use some trusty tools to make it easy for us!

The key insight we’re looking for during our qualification is if the mention is in a natural linking element of the page. It’s important to avoid contacting sites where the mention is only in the title, as they’ll never place the link. We particularly want placements in the body copy as these are natural link locations and so increase the likelihood of your efforts leading somewhere.

So from my list of URLs, I’ll copy the list and head over to URLopener.com (now bought by 10bestseo.com presumably because it’s such an awesome tool) and paste in my list before asking it to open all the URLs for me:

Now, one by one, I can quickly scan the URLs and look for mentions in the right places (i.e. is the mention in the copy, is it in the headline, or is it used anywhere else where a link might not look natural?).

When we see something like this (below), we’re making sure to add this URL to our final outreach list:

However, when we see this (again, below), we’re probably stripping the URL out of our list as there’s very little chance the author/webmaster will add a link in such a prominent and unusual part of the page:

The idea is to finish up with a list of unlinked mentions in spots where a link would fit naturally for the publisher. We don’t want to get in touch with everyone, with mentions all over the place, as it can harm your future relationships. Link building needs to make sense, and not just for Google. If you’re working in a niche that mentions your client, you likely want not only to get a link but also build a relationship with this writer — it could lead to 5 links further down the line.

Getting email addresses

Now that you’ve got a list of URLs that all feature your brand/client, and you’ve qualified this list to ensure they are all unlinked and have mentions in places that make sense for a link, we need to do the most time-consuming part: finding email addresses.

To continue expanding our spreadsheet, we’re going to need to know the contact details of the writer or webmaster to request our link from. To continue our theme of efficiency, we just want to get the two most important details: email address and first name.

Getting the first name is usually pretty straightforward and there’s not really a need to automate this. However, finding email addresses could be an entirely separate article in itself, so I’ll be brief and get to the point. Read this, and here’s a summary of places to look and the tools I use:

  • Author page
  • Author’s personal website
  • Author’s Twitter profile
  • Rapportive & Email Permutator
  • Allmytweets
  • Journalisted.com
  • Mail Tester

More recently, we’ve been also using Skrapp.io. It’s a LinkedIn extension (like Hunter.io) that installs a “Find Email” button on LinkedIn with a percentage of accuracy. This can often be used with Mail Tester to discover if the suggested email address provided is working or not.

It’s likely to be a combination of these tools that helps you navigate finding a contact’s email address. Once we have it, we need to get in touch — at scale!

Pro Tip: When using Allmytweets, if you’re finding that searches for “email” or “contact” aren’t working, try “dot.” Usually journalists don’t put their full email address on public profiles in a scrapeable format, so they use “me@gmail [dot] com” to get around it.

Making contact

So, because this is all about making the process efficient, I’m not going to repeat or try to build on the other already useful articles that provide templates for outreach (there is one below, but that’s just as an example!). However, I am going to show you how to scale your outreach and follow-ups.

Mail merges

If you and your team aren’t set in your ways with a particular paid tool, your best bet for optimizing scale is going to be a mail merge. There are a number of them out there, and honestly, they are all fairly similar with either varying levels of free emails per day before you have to pay, or they charge from the get-go. However, for the costs we’re talking about and the time it saves, building a business case to either convince yourself (freelancers) or your finance department (everyone else!) will be a walk in the park.

I’ve been a fan of Contact Monkey for some time, mainly for tracking open rates, but their mail merge product is also part of the $10-a-month package. It’s a great deal. However, if you’re after something a bit more specific, YAMM is free to a point (for personal Gmail accounts) and can send up to 50 emails a day.

You’ll likely need to work through the process with the whatever tool you pick but, using your spreadsheet, you’ll be able to specify which fields you want the mail merge to select from, and it’ll insert each element into the email.

For link reclamation, this is really as personable as you need to get — no lengthy paragraphs on how much you loved [insert article related to my infographic] or how long you’ve been following them on Twitter, just a good old to the point email:

Hi [first name],

I recently found a mention of a company I work with in one of your articles.

Here’s the article:[insert URL]

Where you’ve mentioned our company, Moz, would you be able to provide a link back to the domain Moz.com, in case users would like to know more about us?

Many thanks,
Darren.

If using BuzzStream

Although BuzzStream’s mail merge options are pretty similar to the process above, the best “above and beyond” feature that BuzzStream has is that you can schedule in follow up emails as well. So, if you didn’t hear back the first time, after a week or so their software will automatically do a little follow-up, which in my experience, often leads to the best results.

When you’re ready to start sending emails, select the project you’ve set up. In the “Websites” section, select “Outreach.” Here, you can set up a sequence, which will send your initial email as well as customized follow-ups.

Using the same extremely brief template as above, I’ve inserted my dynamic fields to pull in from my data set and set up two follow up emails due to send if I don’t hear back within the next 4 days (BuzzStream hooks up with my email through Outlook and can monitor if I receive an email from this person or not).

Each project can now use templates set up for the type of mention you’re following up. By using pre-set templates, you can create one for brand mention, influencers, or creative projects to further save you time. Good times.

I really hope this has been useful for beginners and seasoned link reclamation pros alike. If you have any other tools you use that people may find useful or have any questions, please do let us know below.

Thanks everyone!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Big, Fast, and Strong: Setting the Standard for Backlink Index Comparisons

Posted by rjonesx.

It’s all wrong

It always was. Most of us knew it. But with limited resources, we just couldn’t really compare the quality, size, and speed of link indexes very well. Frankly, most backlink index comparisons would barely pass for a high school science fair project, much less a rigorous peer review.

My most earnest attempt at determining the quality of a link index was back in 2015, before I joined Moz as Principal Search Scientist. But I knew at the time that I was missing a huge key to any study of this sort that hopes to call itself scientific, authoritative or, frankly, true: a random, uniform sample of the web.

But let me start with a quick request. Please take the time to read this through. If you can’t today, schedule some time later. Your businesses depend on the data you bring in, and this article will allow you to stop taking data quality on faith alone. If you have questions with some technical aspects, I will respond in the comments, or you can reach me on twitter at @rjonesx. I desperately want our industry to finally get this right and to hold ourselves as data providers to rigorous quality standards.


Quick links:

  1. Home
  2. Getting it right
  3. What’s the big deal with random?
  4. Now what? Defining metrics
  5. Caveats
  6. The metrics dashboard
  7. Size matters
  8. Speed
  9. Quality
  10. The Link Index Olympics
  11. What’s next?
  12. About PA and DA
  13. Quick takeaways

Getting it right

One of the greatest things Moz offers is a leadership team that has given me the freedom to do what it takes to “get things right.” I first encountered this when Moz agreed to spend an enormous amount of money on clickstream data so we could make our keyword tool search volume better (a huge, multi-year financial risk with the hope of improving literally one metric in our industry). Two years later, Ahrefs and SEMRush now use the same methodology because it’s just the right way to do it.

About 6 months into this multi-year project to replace our link index with the huge Link Explorer, I was tasked with the open-ended question of “how do we know if our link index is good?” I had been thinking about this question ever since that article published in 2015 and I knew I wasn’t going to go forward with anything other than a system that begins with a truly “random sample of the web.” Once again, Moz asked me to do what it takes to “get this right,” and they let me run with it.

What’s the big deal with random?

It’s really hard to over-state how important a good random sample is. Let me diverge for a second. Let’s say you look at a survey that says 90% of Americans believe that the Earth is flat. That would be a terrifying statistic. But later you find out the survey was taken at a Flat-Earther convention and the 10% who disagreed were employees of the convention center. This would make total sense. The problem is the sample of people surveyed wasn’t of random Americans — instead, it was biased because it was taken at a Flat-Earther convention.

Now, imagine the same thing for the web. Let’s say an agency wants to run a test to determine which link index is better, so they look at a few hundred sites for comparison. Where did they get the sites? Past clients? Then they are probably biased towards SEO-friendly sites and not reflective of the web as a whole. Clickstream data? Then they would be biased towards popular sites and pages — once again, not reflective of the web as a whole!

Starting with a bad sample guarantees bad results.

It gets even worse, though. Indexes like Moz report our total statistics (number of links or number of domains in our index). However, this can be terribly misleading. Imagine a restaurant which claimed to have the largest wine selection in the world with over 1,000,000 bottles. They could make that claim, but it wouldn’t be useful if they actually had 1,000,000 of the same type, or only Cabernet, or half-bottles. It’s easy to mislead when you just throw out big numbers. Instead, it would be much better to have a random selection of wines from the world and measure if that restaurant has it in stock, and how many. Only then would you have a good measure of their inventory. The same is true for measuring link indexes — this is the theory behind my methodology.

Unfortunately, it turns out getting a random sample of the web is really hard. The first intuition most of us at Moz had was to just take a random sample of the URLs in our own index. Of course we couldn’t — that would bias the sample towards our own index, so we scrapped that idea. The next thought was: “We know all these URLs from the SERPs we collect — perhaps we could use those.” But we knew they’d be biased towards higher-quality pages. Most URLs don’t rank for anything — scratch that idea. It was time to take a deeper look.

I fired up Google Scholar to see if any other organizations had attempted this process and found literally one paper, which Google produced back in June of 2000, called “On Near-Uniform URL Sampling.” I hastily whipped out my credit card to buy the paper after reading just the first sentence of the abstract: “We consider the problem of sampling URLs uniformly at random from the Web.” This was exactly what I needed.

Why not Common Crawl?

Many of the more technical SEOs reading this might ask why we didn’t simply select random URLs from a third-party index of the web like the fantastic Common Crawl data set. There were several reasons why we considered, but chose to pass, on this methodology (despite it being far easier to implement).

  1. We can’t be certain of Common Crawl’s long-term availability. Top million lists (which we used as part of the seeding process) are available from multiple sources, which means if Quantcast goes away we can use other providers.
  2. We have contributed crawl sets in the past to Common Crawl and want to be certain there is no implicit or explicit bias in favor of Moz’s index, no matter how marginal.
  3. The Common Crawl data set is quite large and would be harder to work with for many who are attempting to create their own random lists of URLs. We wanted our process to be reproducible.

How to get a random sample of the web

The process of getting to a “random sample of the web” is fairly tedious, but the general gist of it is this. First, we start with a well-understood biased set of URLs. We then attempt to remove or balance this bias out, making the best pseudo-random URL list we can. Finally, we use a random crawl of the web starting with those pseudo-random URLs to produce a final list of URLs that approach truly random. Here are the complete details.

1. The starting point: Getting seed URLs

The first big problem with getting a random sample of the web is that there is no true random starting point. Think about it. Unlike a bag of marbles where you could just reach in and blindly grab one at random, if you don’t already know about a URL, you can’t pick it at random. You could try to just brute-force create random URLs by shoving letters and slashes after each other, but we know language doesn’t work that way, so the URLs would be very different from what we tend to find on the web. Unfortunately, everyone is forced to start with some pseudo-random process.

We had to make a choice. It was a tough one. Do we start with a known strong bias that doesn’t favor Moz, or do we start with a known weaker bias that does? We could use a random selection from our own index for the starting point of this process, which would be pseudo-random but could potentially favor Moz, or we could start with a smaller, public index like the Quantcast Top Million which would be strongly biased towards good sites.

We decided to go with the latter as the starting point because Quantcast data is:

  1. Reproducible. We weren’t going to make “random URL selection” part of the Moz API, so we needed something others in the industry could start with as well. Quantcast Top Million is free to everyone.
  2. Not biased towards Moz: We would prefer to err on the side of caution, even if it meant more work removing bias.
  3. Well-known bias: The bias inherent in the Quantcast Top 1,000,000 was easily understood — these are important sites and we need to remove that bias.
  4. Quantcast bias is natural: Any link graph itself already shares some of the Quantcast bias (powerful sites are more likely to be well-linked)

With that in mind, we randomly selected 10,000 domains from the Quantcast Top Million and began the process of removing bias.

2. Selecting based on size of domain rather than importance

Since we knew the Quantcast Top Million was ranked by traffic and we wanted to mitigate against that bias, we introduced a new bias based on the size of the site. For each of the 10,000 sites, we identified the number of pages on the site according to Google using the “site:” command and also grabbed the top 100 pages from the domain. Now we could balance the “importance bias” against a “size bias,” which is more reflective of the number of URLs on the web. This was the first step in mitigating the known bias of only high-quality sites in the Quantcast Top Million.

3. Selecting pseudo-random starting points on each domain

The next step was randomly selecting domains from that 10,000 with a bias towards larger sites. When the system selects a site, it then randomly selects from the top 100 pages we gathered from that site via Google. This helps mitigate the importance bias a little more. We aren’t always starting with the homepage. While these pages do tend to be important pages on the site, we know they aren’t always the MOST important page, which tends to be the homepage. This was the second step in mitigating the known bias. Lower-quality pages on larger sites were balancing out the bias intrinsic to the Quantcast data.

4. Crawl, crawl, crawl

And here is where we make our biggest change. We actually crawl the web starting with this set of pseudo-random URLs to produce the actual set of random URLs. The idea here is to take all the randomization we have built into the pseudo-random URL set and let the crawlers randomly click on links to produce the truly random URL set. The crawler will select a random link from our pseudo-random crawlset and then start a process of randomly clicking links, each time with a 10% chance of stopping and a 90% chance of continuing. Wherever the crawler ends, the final URL is dropped into our list of random URLs. It is this final set of URLs that we use to run our metrics. We generate around 140,000 unique URLs through this process monthly to produce our test data set.

Phew, now what? Defining metrics

Once we have the random set of URLs, we can start really comparing link indexes and measuring their quality, quantity, and speed. Luckily, in their quest to “get this right,” Moz gave me generous paid access to competitor APIs. We began by testing Moz, Majestic, Ahrefs, and SEMRush, but eventually dropped SEMRush after their partnership with Majestic.

So, what questions can we answer now that we have a random sample of the web? This is the exact wishlist I sent out in an email to leaders on the link project at Moz:

  1. Size:
    • What is the likelihood a randomly selected URL is in our index vs. competitors?
    • What is the likelihood a randomly selected domain is in our index vs. competitors?
    • What is the likelihood an index reports the highest number of backlinks for a URL?
    • What is the likelihood an index reports the highest number of root linking domains for a URL?
    • What is the likelihood an index reports the highest number of backlinks for a domain?
    • What is the likelihood an index reports the highest number of root linking domains for a domain?

  2. Speed:
    • What is the likelihood that the latest article from a randomly selected feed is in our index vs. our competitors?
    • What is the average age of a randomly selected URL in our index vs. competitors?
    • What is the likelihood that the best backlink for a randomly selected URL is still present on the web?
    • What is the likelihood that the best backlink for a randomly selected domain is still present on the web?
  3. Quality:
    • What is the likelihood that a randomly selected page’s index status (included or not included in index) in Google is the same as ours vs. competitors?
    • What is the likelihood that a randomly selected page’s index status in Google SERPs is the same as ours vs. competitors?
    • What is the likelihood that a randomly selected domain’s index status in Google is the same as ours vs. competitors?
    • What is the likelihood that a randomly selected domain’s index status in Google SERPs is the same as ours vs. competitors?
    • How closely does our index compare with Google’s expressed as “a proportional ratio of pages per domain vs our competitors”?
    • How well do our URL metrics correlate with US Google rankings vs. our competitors?

Reality vs. theory

Unfortunately, like all things in life, I had to make some cutbacks. It turns out that the APIs provided by Moz, Majestic, Ahrefs, and SEMRush differ in some important ways — in cost structure, feature sets, and optimizations. For the sake of politeness, I am only going to mention name of the provider when it is Moz that was lacking. Let’s look at each of the proposed metrics and see which ones we could keep and which we had to put aside…

  1. Size: We were able monitor all 6 of the size metrics!

  2. Speed:
    • We were able to include this Fast Crawl metric.
    • What is the average age of a randomly selected URL in our index vs. competitors?
      Getting the age of a URL or domain is not possible in all APIs, so we had to drop this metric.
    • What is the likelihood that the best backlink for a randomly selected URL is still present on the web?
      Unfortunately, doing this at scale was not possible because one API is cost prohibitive for top link sorts and another was extremely slow for large sites. We hope to run a set of live-link metrics independently from our daily metrics collection in the next few months.
    • What is the likelihood that the best backlink for a randomly selected Domain is still present on the web?
      Once again, doing this at scale was not possible because one API is cost prohibitive for top link sorts and another was extremely slow for large sites. We hope to run a set of live-link metrics independently from our daily metrics collection in the next few months.
  3. Quality:
    • We were able to keep this metric.
    • What is the likelihood that a randomly selected page’s index status in Google SERPs is the same as ours vs. competitors?
      Chose not to pursue due to internal API needs, looking to add soon.
    • We were able to keep this metric.
    • What is the likelihood that a randomly selected domain’s index status in Google SERPs is the same as ours vs. competitors?
      Chose not to pursue due to internal API needs at the beginning of project, looking to add soon.
    • How closely does our index compare with Google’s expressed as a proportional ratio of pages per domain vs our competitors?
      Chose not to pursue due to internal API needs. Looking to add soon.
    • How well do our URL metrics correlate with US Google rankings vs. our competitors?
      Chose not to pursue due to known fluctuations in DA/PA as we radically change the link graph. The metric would be meaningless until the index became stable.

Ultimately, I wasn’t able to get everything I wanted, but I was left with 9 solid, well-defined metrics.

On the subject of live links:

In the interest of being TAGFEE, I will openly admit that I think our index has more deleted links than others like the Ahrefs Live Index. As of writing, we have about 30 trillion links in our index, 25 trillion we believe to be live, but we know that some proportion are likely not. While I believe we have the most live links, I don’t believe we have the highest proportion of live links in an index. That honor probably does not go to Moz. I can’t be certain because we can’t test it fully and regularly, but in the interest of transparency and fairness, I felt obligated to mention this. I might, however, devote a later post to just testing this one metric for a month and describe the proper methodology to do this fairly, as it is a deceptively tricky metric to measure. For example, if a link is retrieved from a chain of redirects, it is hard to tell if that link is still live unless you know the original link target. We weren’t going to track any metric if we couldn’t “get it right,” so we had to put live links as a metric on hold for now.

Caveats

Don’t read any more before reading this section. If you ask a question in the comments that shows you didn’t read the Caveats section, I’m just going to say “read the Caveats section.” So here goes…

  • This is a comparison of data that comes back via APIs, not within the tools themselves. Many competitors offer live, fresh, historical, etc. types of indexes which can differ in important ways. This is just a comparison of API data using default settings.
  • Some metrics are hard to estimate, especially like “whether a link is in the index,” because no API — not even Moz — has a call that just tells you whether they have seen the link before. We do our best, but any errors here are on the the API provider. I think we (Moz, Majestic, and Ahrefs) should all consider adding an endpoint like this.
  • Links are counted differently. Whether duplicate links on a page are counted, whether redirects are counted, whether canonicals are counted (which Ahrefs just changed recently), etc. all affect these metrics. Because of this, we can’t be certain that everything is apples-to-apples. We just report the data at face value.
  • Subsequently, the most important takeaway in all of these graphs and metrics is direction. How are the indexes moving relative to one another? Is one catching up, is another falling behind? These are the questions best answered.
  • The metrics are adversarial. For each random URL or domain, a link index (Moz, Majestic, or Ahrefs) gets 1 point for being the biggest, for tying with the biggest, or for being “correct.” They get 0 points if they aren’t the winner. This means that the graphs won’t add up to 100 and it also tends to exaggerate the differences between the indexes.
  • Finally, I’m going to show everything, warts and all, even when it was my fault. I’ll point out why some things look weird on graphs and what we fixed. This was a huge learning experience and I am grateful for the help I received from the support teams at Majestic and Ahrefs who, as a customer, responded to my questions honestly and openly.

The metrics dashboard

The Dashboard for All MetricsWe’ve been tracking these 9 core metrics (albeit with improvements) since November of 2017. With a close eye on quality, size, and speed, we have methodically built an amazing backlink index, not driven by broad counts but instead by intricately defined and measured metrics. Let’s go through each of those metrics now.

Size matters

It does. Let’s admit it. The diminutive size of the Mozscape index has been a limitation for years. Maybe someday we will write a long post about all the efforts Moz has made to grow the index and what problems stood in our way, but that’s a post for a different day. The truth is, as much as quality matters, size is huge for a number of specific use-cases for a link index. Do you want to find all your bad links? Bigger is better. Do you want to find a lot of link opportunities? Bigger is better. So we came up with a number of metrics to help us determine where we were relative to our competitors. Here are each of our Size metrics.

Index Has URL

What is the likelihood a randomly selected URL is in our index vs. competitors?

This is one of my favorite metrics because I think it’s a pure reflection of index size. It answers the simple question of “if we grabbed a random URL on the web, what’s the likelihood an index knows about it?” However, you can see my learning curve in the graph (I was misreporting the Ahrefs API due to an error on my part) but once corrected, we had a nice reflection of the indexes. Let me restate this — these are comparisons in APIs, not in the web tools themselves. If I recall correctly, you can get more data out of running reports in Majestic, for example. However, I do think this demonstrates that Moz’s new Link Explorer is a strong contender, if not the largest, as we have led in this category every day except one. As of writing this post, Moz is winning.

Index Has Domain

What is the likelihood a randomly selected domain is in our index vs competitors?

When I said I would show “warts and all,” I meant it. Determining whether a domain is in an index isn’t as simple as you would think. For example, perhaps a domain has pages in the index, but not the homepage. Well, it took me a while to figure this one out, but by February of this year I had it down.

The scale of this graph is important to note as well. The variation is between 99.4 and 100% between Moz, Majestic, and Ahrefs over the last few months. This indicates just how close the link indexes are in terms of knowing about root domains. Majestic has historically tended to win this metric with near 100% coverage, but you would have to select 100 random domains to find one that Moz or Ahrefs doesn’t have information on. However, Moz’s continued growth has allowed us to catch up. While the indexes are super close, as of writing this post, Moz is winning.

Backlinks Per URL

Which index has the highest backlink count for a randomly selected URL?

This is a difficult metric to really pin down. Unfortunately, it isn’t easy to determine what backlinks should count and what shouldn’t. For example, imagine a URL has one page linking to it, but that page includes that link 100 times. Is that 100 backlinks or one? Well, it turns out that the different link indexes probably measure these types of scenarios differently and getting an exact definition out of each is like pulling teeth because the definition is so complicated and there are so many edge cases. At any rate, I think this is a great example of where we can show the importance of direction. Whatever the metrics actually are, Moz and Majestic are catching up to Ahrefs, which has been the leader for some time. As of writing this post, Ahrefs is winning.

Root Linking Domains Per URL

Which index reports the highest RLD count for a randomly selected URL?

Simple, right? No, even this metric has its nuances. What is a root linking domain? Do subdomains count if they are on subdomain sites like Blogspot or WordPress.com? If so, how many sites are there on the web which should be treated this way? We used a machine learned methodology based on surveys, SERP data, and unique link data to determine our list, but each competitor does it differently. Thus, for this metric, direction really matters. As you can see, Moz has been steadily catching up and as of writing today, Moz is finally winning.

Backlinks Per Domain

Which index reports the highest backlink count for a randomly selected domain?

This metric was not kind to me, as I found a terrible mistake early on. (For the other techies reading this, I was storing backlink counts as INT(11) rather than BIGINT, which caused lots of ties for big domains when they were larger than the maximum number size because the database defaults to same highest number.) Nevertheless, Majestic has been stealing the show on this metric for a little while, although the story is deeper than that. Their dominance is such an outlier that it needs to be explained.

One of the hardest decisions a company has to make regarding its backlink index is how to handle spam. On one hand, spam is expensive to the index and probably ignored by Google. On the other hand, it is important for users to know if they have received tons of spammy links. I don’t think there is a correct answer to this question; each index just has to choose. A close examination of the reason why Majestic is winning (and continuing to increase their advantage) is because of a particularly nefarious Wikipedia-clone spam network. Any site with any backlinks from Wikipedia are getting tons of links from this network, which is causing their backlink counts to increase rapidly. If you are worried about these types of links, you need to go take a look at Majestic and look for links ending in primarily .space or .pro, including sites like tennis-fdfdbc09.pro, troll-warlord-64fa73ba.pro, and badminton-026a50d5.space. As of my last tests, there are over 16,000 such domains in this spam network within Majestic’s index. Majestic is winning this metric, but for purposes other than finding spam networks, it might not be the right choice.

Linking Root Domains Per Domain

Which index reports the highest LRD count for a randomly selected domain?

OK, this one took me a while to get just right. In the middle of this graph, I corrected an important error where I was looking at domains only for the root domain on Ahrefs rather than the root domain and all subdomains. This was unfair to Ahrefs until I finally got everything corrected in February. Since then, Moz has been aggressively growing its index, Majestic has picked up LRD counts through the previously discussed network but steadied out, and Ahrefs has remained relatively steady in size. Because of the “adversarial” nature of these metrics, it gives the false appearance that Ahrefs is dropping dramatically. They aren’t. They are still huge, and so is Majestic. The real takeaway is directional: Moz is growing dramatically relative to their networks. As of writing this post, Moz is winning.

Speed

Being the “first to know” is an important part in almost any industry and with link indexes it is no different. You want to know as soon as possible when a link goes up or goes down and how good that link is so you can respond if necessary. Here is our current speed metric.

FastCrawl

What is the likelihood the latest post from a randomly selected set of RSS feeds is indexed?

Unlike the other metrics discussed, the sampling here is a little bit different. Instead of using the randomization above, we make a random selection from a million+ known RSS feeds to find their latest post and check to see if they have been included in the various indexes of Moz and competitors. While there are a few errors in this graph, I think there is only one clear takeaway. Ahrefs is right about their crawlers. They are fast and they are everywhere. While Moz has increased our coverage dramatically and quickly, it has barely put a dent in this FastCrawl metric.

Now you may ask, if Ahrefs is so much faster at crawling, how can Moz catch up? Well, there are a couple of answers, but probably the biggest is that new URLs only represent a fraction of the web. Most URLs aren’t new. Let’s say two indexes (one new, one old) have a bunch of URLs they’re considering crawling. Both might prioritize URLs on important domains that they’ve never seen before. For the larger, older index, that will be a smaller percentage of that group because they have been crawling fast a long time. So, during the course of the day, a higher percentage of the old index’s crawl will be dedicated to re-crawl pages it already knows about. The new index can dedicate more of its crawl potential to new URLs.

It does, however, put the pressure on Moz now to improve crawl infrastructure as we catch up to and overcome Ahrefs in some size metrics. As of this post, Ahrefs is winning the FastCrawl metric.

Quality

OK, now we’re talking my language. This is the most important stuff, in my opinion. What’s the point of making a link graph to help people with SEO if it isn’t similar to Google? While we had to cut some of the metrics temporarily, we did get a few in that are really important and worth taking a look.

Domain Index Matches

What is the likelihood a random domain shares the same index status in Google and a link index?

Domain Index Matches seeks to determine when a domain shares the same index status with Google as it does in one of the competing link indexes. If Google ignores a domain, we want to ignore a domain. If Google indexes a domain, we want to index a domain. If we have a domain Google doesn’t, or vice versa, that is bad.

This graph is a little harder to read because of the scale (the first few days of tracking were failures), but what we actually see is a statistically insignificant difference between Moz and our competitors. We can make it look more competitive than it really is if we just calculate wins and losses, but we have to take into account an error in the way we determined Ahrefs index status up until around February. To do this, I show wins/losses for all time vs. wins/losses over the last few months.

As you can see, Moz wins the “all time,” but Majestic has been winning more over the last few months. Nevertheless, these are quite insignificant, often being the difference between one or two domain index statuses out of 100. Just like the Index Has Domain metric we discussed above, nearly every link index has nearly every domain, and looking at the long-term day-by-day graph shows just how incredibly close they are. However, if we are keeping score, as of today (and the majority of the last week), Moz is winning this metric.

Domain URL Matches

What is the likelihood a random URL shares the same index status in Google as in a link index?

This one is the most important quality metric, in my honest opinion. Let me explain this one a little more. It’s one thing to say that your index is really big and has lots of URLs, but does it look like Google’s? Do you crawl the web like Google? Do you ignore URLs Google ignores while crawling URLs that Google crawls? This is a really important question and sets the foundation for a backlink index that is capable of producing good relational metrics like PA and DA.

This is one of the metrics where Moz just really shines. Once we corrected for an error in the way we were checking Ahrefs, we could accurately determine whether our index was more or less like Google’s than our competitors. Since the beginning of tracking, Moz Link Explorer has never been anything but #1. In fact, we only had 3 ties with Ahrefs and never lost to Majestic. We have custom-tailored our crawl to be as much like Google as possible, and it has paid off. We ignore the types of URLs Google hates, and seek out the URLs Google loves. We believe this will pay huge dividends in the long run for our customers as we expand our feature set based on an already high-quality, huge index.

The Link Index Olympics

Alright, so we’ve just spent a lot of time delving into these individual metrics, so I think it’s probably worth it to put these things into an easy-to-understand context. Let’s pretend for a moment that this is the Link Index Olympics, and no matter how much you win or lose by, it determines whether you receive a gold, bronze or silver medal. I’m writing this on Wednesday, April 25th. Let’s see how things play out if the Olympics happened today:

As you can see, Moz takes the gold in six of the nine metrics we measure, two silvers, and one bronze. Moreover, we’re continuing to grow and improve our index daily. As most of the above graphs indicate, we tend to be improving relative to our competitors, so I hope that by the time of publication in a week or so our scores will even be better. But the reality is that based on the metrics above, our link index quality, quantity, and speed are excellent. I’m not going to say our index is the best. I don’t think that’s something anyone can really even know and is highly dependent upon the specific use case. But I can say this — it is damn good. In fact, Moz has won or tied for the “gold” 27 out of the last 30 days.

What’s next?

We are going for gold. All gold. All the time. There’s a ton of great stuff on the horizon. Look forward to regular additions of features to Link Explorer based on the data we already have, faster crawling, and improved metrics all around (PA, DA, Spam Score, and potentially some new ones in the works!) There’s way too much to list here. We’ve come a long way but we know we have a ton more to do. These are exciting times!

A bit about DA and PA

Domain Authority and Page Authority are powered by our link index. Since we’re moving from an old, much smaller index to a larger, much faster index, you may see small or large changes to DA and PA depending on what we’ve crawled in this new index that the old Mozscape index missed. Your best bet is just to compare yourselves to your competitors. Moreover, as our index grows, we have to constantly adjust the model to address the size and shape of our index, so both DA and PA will remain in beta a little while. They are absolutely ready for primetime, but that doesn’t mean we don’t intend to continue to improve them over the next few months as our index growth stabilizes. Thanks!

Quick takeaways

Congratulations for getting through this post, but let me give you some key takeaways:

  1. The new Moz Link Explorer is powered by an industry-leading link graph and we have the data to prove it.
  2. Tell your data providers to put their math where their mouth is. You deserve honest, well-defined metrics, and it is completely right of you to demand it from your data providers.
  3. Doing things right requires that we sweat the details. I cannot begin to praise our leadership, SMEs, designers, and engineers who have asked tough questions, dug in, and solved tough problems, refusing to build anything but the best. This link index proves that Moz can solve the hardest problem in SEO: indexing the web. If we can do that, you can only expect great things ahead.

Thanks for taking the time to read! I look forward to answering questions in the comments or you can reach me on Twitter at @rjonesx.

Also, I would like to thank the non-Mozzers who offered peer reviews and critiques of this post in advance — they do not necessarily endorse any of the conclusions, but provided valuable feedback. In particular, I would like to thank Patrick Stox of IBM, JR Oakes of Adapt Partners, Alexander Darwin of HomeAgency, Paul Shapiro of Catalyst SEM, the person I most trust in SEO, Tony Spencer, and a handful of others who wished to remain anonymous.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!