SEO: Crawling and Robots
Get the best search news, tips and resources, delivered each day.
Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.
Managing Sitemap XML with Google Search Console
What to look for in Search Console indexing reports, plus learn why and how to create a dynamic Sitemap...
Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section
Pay close attention to the directives you give to Bingbot and other search engines in your robots.txt...
Technical SEO in the wild: Real-world issues and fixes
A website has a lot of moving parts that affect one another and sometimes a tiny little change can cause a huge...
Why the SaaS model is right for programmatic
By charging a simple subscription fee for the programmatic buying platform, cost-efficient buying and scale spending efficiencies...
Bing crawling, indexing and rendering: A step-by-step on how it works
Crawlers are technical but understandable thanks to Microsoft's Senior Program Manager Frédéric Dubut's presentation at SMX...
How to boost search rankings using only your internal linking strategy
Links, even within a website, show relationships between content. They transfer value and importance between pages. Even more importantly,...
Link building tool roundup: Site crawlers
Link building is hard work! Here's how to utilize site crawling tools to make sure your effort pays off....
Bing says it is improving web crawler efficiency
Bing is working on making sure its crawler doesn't miss new content and at the same time overload your web...
Why Google Cache lies to you and what to do about it (if anything)
You shouldn't panic if you see weirdness in the Google Cache. Still, it's worth investigating in case an odd display is a sign of a bigger...
Mobile-first indexing: Will it change your rankings on desktop?
Getting conflicting information about whether the mobile-first indexing change will impact your rankings? Brush up on the different processes...
Ask the SMXperts — Page speed, site migrations and crawling
Our SMXperts share their tactical insights on optimizing crawl budgets, fine-tuning page speed and how to execute on large-scale error-free site...
5 ways to improve ROI on seasonal pages by optimizing your SEO crawl budget
What is a crawl budget? Google’s goal is to make useful information available to people searching the web. To accomplish that, Google...
Old content still showing up in Google search results? Google might not find that content important
Google explains how crawl rate can be used as confirmation that a page is OK to remove from your website, and thus their...
Ask an SMXpert — Advanced Technical SEO for Bing
SMXpert and Microsoft Senior Program Manager Frederic Dubut answers questions on how Bing handles redirects, crawl budgets and...
The ultimate guide to HTTP status codes and headers for SEO
Do you understand how HTTP protocol works and the impact it has on crawling and indexing web pages? No? Here's a guide written by Contributor...
How to be proactive and prevent a negative SEO campaign — Part 3
There is no such thing as being negative SEO-proof, says contributor Joe Sinkwitz. All you can do is take steps to lessen the probability of...
Ask the SMXperts — Schema & Structured Data, JavaScript
Three #SMXperts answer questions from the Advanced Technical SEO session at SMX Advanced and share their insights on JavaScript and structured...
Predicting the value of a search engine ranking signal
Contributor Dave Davies deconstructs a new Google patent that covers how machine learning can predict a ranking signal value when the value is...
SMX Advanced session recap: Mobile-first and AMP
Contributor Patrick Stox recaps an advanced SEO session where speakers shared how to perform audits in a mobile-first world and tactics to...
The ultimate guide to bot herding and spider wrangling — Part 3
In this third and final installment, contributor Stephan Spencer outlines common coding, mobile and localization issues and offers workarounds...