By Ever since Google began to focus on the loading speed and experience of a website they have requested access for their crawler (known as Googlebot) to CSS and JS files on a website. This dates back to 2012 when then Google community liasion Matt Cutts requested this access in a video uploaded to an official YouTube channel.
Access to the information in a website’s CSS files and JS files is necessary for a search engine crawler to accurately render and understand how a page might appear to users across various devices and screen types.
The Problem
Recently we had a new client that hired us for our Technical SEO and Publisher SEO services complaining about Google telling them that their pages were not mobile-friendly when viewed in the SERPs. Google does this to try and notify someone they know is connected to a website (using Google Search Console) in hopes of them seeing the notification and fixing the problem, the notification does not appear to the general public or other users.
Our client first requested his tech team and webhosting provider to investigate the issue. As you might predict they all came up with either wild theories or nothing at all. When the client notified us of the issue we made the fix a priority and I dove in. This article might make it sound like the problem was discovered easily, but in reality it took a lot of time and energy to research and investigate more common causes of this known problem first.
What I found was that the Mobile-Friendly Testing tool was showing that Googlebot Smartphone saw the pages on our client’s site as broken. The CSS was not rendering at all and the text and clickable / tapable elements on mobile were all smashed together. Loading the site on various mobile devices and simulators showed zero problems with rendering (aside from it being much slower than we would prefer due to their advertising clutter).
Clicking into one of the errors the tool showed us that Googlebot had been blocked from seeing CSS files in the WordPress theme folder. This issue has been known for sometime but after the recent Page Experience Updates to Google’s algorithm for both Mobile and Desktop, having all pages on your website be marked as “not mobile-friendly” could cause major ranking issues.
Here is what I saw when we looked under the hood at their robots.txt
(note: you can also see this data for your website in Google Search Console under “Mobile Usability”.)
Officially Google supports the Robots Exclusion Protocol for Robots.txt files. And, Google maintains a great resource for developers to help them build good functioning Robots.txt files.
In this specific instance one of the disallow directives was being read by Googlebot to not crawl the theme files (see why below), which resulted in the bad rendering by Googlebot, which led to the labeling of all pages on the website not being mobile-friendly, which was likely causing some of the ranking and traffic grief our client was feeling. Here is that directive:
- Disallow: /*?*
The directive isn’t clear but it appears who ever wrote it was trying to block anything that matched a url pattern possibly to keep Google from crawling content automatically produced by a plugin or some custon function, though the client’s tech team had no explanation for why this would have been done. According to Google’s documentation here is how they apply rules using wildcards (*):
“When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.”
We have since tested this Robots.txt Disallow directive on numerous WordPress websites and always gotten the same response, theme .CSS and .JS files are blocked. However, this only happens when the .CSS or .JS file is followed by a “?” and some parameter (most commonly ‘ver’ for version). It also isn’t limited to just theme files but also to any page on the website that has a “?” in the URL.
I speculate that whoever wrote this was possibly trying to block the search pages from the index. Whatever the directive was intended to be, it is clear that Google’s “lest restrictive rule” appears to have determined that it would need to block crawling to the theme files and other urls with parameters in them as it encounters them across the website.
The Fix
This was a relatively quick fix, we had to tell whoever on the clients team made this to erase it and change it to something more simple and not as easily confused by Googlebot or other engines/crawlers. It was not a super simple research project however. It took several hours of testing various other more common causes before I even considered looking at the robots.txt as the problem. When hiring an agency or someone to do SEO for your WordPress website you should consider that things like this may arise from time to time and your SEO team will need time to research, investigate, and understand the issue before prescribing a solution or fixing it for you.
Once the Robots.txt was set to something much simpler we used Google Search Console to recrawl a few pages then used it to validate the fix to the mobile issues it thought the website was having. It will take Google awhile to update the entire site, but so far we are seeing the label fall off from pages fairly quickly and new content is not being labeled as not mobile-friendly now.
Here’s the same page viewed a few days after the fix was implemented:
If you see weird characters or wildcards in your Robots.txt file but not the exact line shown above I would recommend you test your Robots.txt with a few URLs of your theme’s CSS and JS files. To do this specifically for Google you can test to see how Googlebot might understand your directives by using Google’s own Robots.txt Tester (requires a Search Console for your website).
If you are uncertain about what you are doing with your Robots.txt for WordPress and there’s a lot in there it is probably better to go with something really simple and work from there. You can either go with a Robots.txt file that just allows everything to be crawled or you can use one that allows everything except the WP Admin folder to be crawled. Either way your site won’t get dinged in the rankings for the bot not being able to render the pages of your site correctly.
Allow All Robots.txt:
User-agent: *
Allow: /Disallow WP Admin Robots.txt:
User-agent: *
Disallow: /wp-admin/To help you compare, here is our full Robots.txt as of time of publishing:
User-agent: *
Disallow: /wp-admin/
User-agent: yandex
Disallow: /
________.__
/ _____| | _____ ___ ______
\_____ \| | \__ \\ \/ \__ \
/ | |__/ __ \\ / / __ \_
/_______ |____(____ /\_/ (____ /
\/ \/ \/
____ ___ __ .__
| | | | _____________ |__| ____ ____
| | | |/ \_ __ \__ \ | |/ \_/ __ \
| | /| < | | \// __ \| | | \ ___/
|______/ |__|_ \|__| (____ |__|___| /\___ >
#We terminate bad robots
______
<((((((\\\
/ . }\
;--..--._|}
(\ '--/\--' )
\\ | '-' :'|
\\ . -==- .-|
\\ \.__.' \--._
[\\ __.--| // _/'--.
\ \\ .'-._ ('-----'/ __/ \
\ \\ / __>| | '--. |
\ \\ | \ | / / /
\ '\ / \ | | _/ /
\ \ \ | | / /
\ \ \ /
User-agent: t-800
User-agent: hal9000
User-agent: roberto
User-agent: ultron
User-agent: teslabot
Disallow: /Having SEO issues and need help with your WordPress website? We offer elite level SEO knowledge and expertise combined with exclusive plugins, a massive library of premium plugin licenses, and a long track record of success helping businesses get more qualified traffic from search engines that increases their sales / conversions using the latest SEO tactics.