SEO: Crawling and Robots
Screaming Frog releases SEO Spider version 12.0
The update adds PageSpeed Insights API integration and...
Bingbot goes evergreen
Bing will adopt Chromium-based Edge browser to run JavaScript and render...
Google to update GoogleBot’s user agent
Make sure you are prepared for the new GoogleBot user agent, which launches this...
In response to Google change, Yoast SEO will opt users into all snippet features by default
The change comes with versions 12.2, coming October...
Server migrations are “uneventful for Google systems”
Expect Googlebot to readjust your crawl frequency,...
Google can crawl AJAX just fine
No special action necessary for...
Google selects canonical URLs based on your site and user preference
If a different URL is chosen, it doesn’t negatively affect your...
You can’t opt into or out of mobile-first indexing
Google's John Mueller reminds us that one day, there will only be mobile-first...
Google’s indexing issues are resolved
Google has just confirmed the indexing issues from yesterday are fully...
Yes, Googlebot can detect JavaScript redirects
JavaScript redirects are followed similarly to server-side redirects, Google...
Google testing tools now support evergreen Googlebot
After three months, the Google testing tools are now using the latest version of...
Video: Catalyst’s Paul Shapiro on going beyond technical SEO
Natural language processing, SEO testing, technical SEO audits and more were all discussed in this...
Google notifying webmasters to remove noindex from robots.txt files
Google is now sending reminders that you need to stop depending on your noindex in your robots.txt...
Google adds JavaScript SEO basics to its Search developer’s guide
The new section aims to provide beginners with an overview of how Googlebot processes...
SMX replay: SEO that Google tries to correct for you
Patrick Stox explains the processes to pay attention to and the ones to leave to...
Google to stop supporting noindex directive in robots.txt
Changes are coming to how Google understands some of the unsupported directives in your robots.txt...
Google posts draft to formalize Robots Exclusion Protocol Specification
Google open sources its robots.txt parser as part of this...
Yoast, Google devs propose XML Sitemaps for WordPress Core
The proposal also includes an API for further...
I/O announcements have some applauding and others shaking their fists
Marketers continue to accuse Google of stealing their...
Google’s web crawler Googlebot now runs on the latest version of Chromium: Why that’s a good thing
Googlebot has been upgraded and should now be able to crawl more modern web sites with less of an...