With the upcoming Google page experience update coming to desktop, today Google launched a new page experience report for desktop in Google Search Console. “To support the upcoming rollout of page experience ranking to desktop, Search Console now has a dedicated desktop section in its Page Experience report to help site owners understand Google’s ‘good page experience’ criteria,” Google wrote.
How to access. You can access the report by clicking here or by going to Google Search Console, and clicking on the Page Experience link under the experience tab.
What it looks like. Here is a screenshot of this report for one of my sites:
More details. Google first launched the page experience report in April 2021 before the launch of the page experience update. The new Google Page Experience report offers metrics, such as the percentage of URLs with good page experience and search impressions over time, enabling you to quickly evaluate performance. You can also drill into specific pages to see what improvements need to be made.
Why we care. You can use this report to make the necessary adjustments to the desktop versions of your pages before Google rolls out the desktop version of the page experience update. As a reminder, we do not expect there to be a huge ranking change due to this update, but it may impact sites more if their stories show in the top stories section, since a solid page experience score is required to show in the top stories carousel.
A little less than two months ago Search Engine Land made the decision to stop publishing versions of our content using Google’s Accelerated Mobile Pages. For us, it boiled down to wanting to simplify our reporting and our desire to end the process of hosting Search Engine Land content on 3rd party servers.
Since then, a lot has happened, but the bottom line is we have seen very little disruption to our traffic and have reaped the benefit of having a clearer picture of our audience analytics.
What happened to traffic? For us, it is difficult to draw any major conclusions about traffic changes since we turned off AMP. Search Engine Land is a media website that primarily produces journalism, so we are very much tied to the news that emerges. As you would expect, when big news like core updates or major Google Ads changes happens our traffic jumps. But as the news dies down during the holiday season we usually see month-to-month declines. That why year-over-year benchmarking is generally favored by news organizations.
We did not see any year-over-year declines in traffic that we could tie to AMP aside from the loss of pageviews to a handful of pieces that routinely spike for organic traffic. For example, an older article about Google SERP Easter Eggs ranks highly for us and usually spikes a few times during the year (including Easter time!). Mobile traffic to that post was previously going to the AMP version. However, we turned off AMP at a time that piece was spiking on mobile and did not see that traffic shift back to our native page. The page itself has never really driven quality traffic so the lost traffic isn’t really a problem.
Safeguarding. Around the time we shut off AMP we also took a few steps that could safeguard us in case the experiment caused a big traffic decline. We increased our publishing volume for starters. We also adjusted the strategy in our newsletters to better optimize for click-through rate. That move was also in response to Apple’s privacy change in iOS 15 that now makes open rates a less reliable metric.
The big win. One of the main reasons for turning off AMP was to better understand our metrics. Despite several failed attempts at AMP stitching in Google Analytics, we never could tell how our audience moves from our AMP pages to our native ones. Users were undoubtedly being double-counted as unique in both the AMP and our native website dashboards. The clearest indicator that this was true is in the change we’ve seen in return visitors since we turned off AMP. The number of sessions by return visitors has jumped by 30% since we made the change, and now we have a far better picture of our most valuable audience set.
Why we care. We went into this experiment knowing there was some risk, but haven’t seen anything to make us reconsider the move. The biggest question mark had always been around the Page Experience Update. AMP pages were as fast as they come, so the worry was that our native pages that don’t benchmark as fast as AMP would lose out. It is true that we saw the percentage of pages with “good” Core Web Vitals scores in Goole Search Console plummet when we turned off AMP, but we do not believe it hurt traffic or rankings. It makes sense because many SEOs are still struggling to tie their own wins or losses directly to the Page Experience Update.
So we’re not looking back. And if you have your story about turning off AMP we’d love to hear it.
Over the past couple of weeks there have been complaints from some Shopify site owners that Google was showing a (1) in the title name for their pages in the Google search results page. The issue turned out to be related to a chat feature activated on those Shopify sites, the chat feature fixed the issue and the Google search results should soon no longer show (1) in the title name.
What it looked like. I found a screenshot of this happening for a site in the Shopify forums dating back a couple of weeks ago, here is that screenshot showing the (1) at the beginning of the title name in Google Search.
What it looks like now. The issue was resolved and Google recrawled and processed this specific URL, so the (1) is no longer there:
It will take time. If you still see a (1) before your title name in the Google Search results, give it more time. Google has to recrawl and reprocess all of the URLs that were impacted and that can take time. If you want to expedite it, you can use the URL inspection tool in Google Search Console and submit that URL to the index manually. But again, the issue will resolve itself over time.
Google’s statement. Google published a statement on this issue in the Google forums, basically saying it was an issue with the chat feature dynamically embedding (1) in the title attributes of these pages and thus Googlebot picked up on it and indexed it. Google’s Caio Barros wrote:
Hello, all!
We have been receiving some reports of a “(1)” showing up in some titles in search results. Upon some investigation, our Product Experts noticed that this behavior happened to websites built in Shopify and were using a chat app. It looks like these sites used a chat-bot script which added a “(1)” to the page’s title element. Titles changed with JavaScript can still be picked up, and used as title links in Search.
However, it looks like that script has been fixed to no longer change the page’s title element, so as Googlebot reprocess pages, it will no longer see the “(1)” as a part of the pages’ title, and we can take that into account when generating title links in Search. Keep in mind that title links in Search aren’t always exactly the same as the title element of a page, so it’s not guaranteed that Google will drop that element immediately after reprocessing.
There’s no need to do anything special to have pages reprocessed. This should happen automatically over time. We crawl and reprocess pages at different rates, usually you’ll see important pages like a site’s homepage reprocessed fairly quickly, within a few days at most. Other pages may take longer to be reprocessed.
Thank you all for the reports!
Why we care. If you see (1) in your titles in the Google or Bing search results, it was likely due to this chat feature in Shopify. Again, the chat feature fixed the issue and the search engines will eventually recrawl and reprocess those titles and show them correctly in the search results. It is a widespread issue, not a Google bug, but it was related to a feature in Shopify that had this unintended consequence in search.
“There are roughly three and a half billion Google searches made every day,” said Craig Dunham, CEO of enterprise SEO platform Deepcrawl, at our recent MarTech conference. “According to research from Moz, 84% of people use Google at least three times a day, and about half of all product searches start with Google. The way that consumers are engaging with brands is changing, and it’s doing so rapidly.”
He added, “Consumers begin their journey with the tool that many of us use hundreds of times a day. Thus, the connection to revenue becomes clear — it starts with search.”
Source: Craig Dunham and Scott Brinker
The concept of digital transformation has been around for years, but it’s taken a whole new form in the wake of recent societal shifts. New technologies and the 2020 pandemic have led to a “greater focus on the need to drive optimal digital experiences for our customers,” says Dunham.
A brand’s website is often the first, and most lasting, impression customers will have of your organization. Here are some strategic actions he recommends marketers take to ensure their online properties are optimized for the search-first age.
Educate your team and higher-ups about the necessity of organic search
“The website is a shared responsibility and it requires proper strategic leadership,” Dunham said. “The first step is to take some time and educate yourself, your leadership, your board and your organization so they more broadly promote organic KPIs as business-wide objectives.”
“There’s great data out there on the impact of the efficiency of SEO as a low-cost acquisition channel,” he added.
Source: Craig Dunham
Aside from sharing communication from Google on the importance of search from a business perspective, marketers can look for case studies from reputable organizations to encourage search prioritization. This can help higher-ups start seeing organic traffic as a key business metric.
“I was in a meeting recently and I had a digital leader say to me that you know website performance should not be an SEO metric — it has to be a business metric,” he said.
Create a cross-functional search ops task force
“Much of the data and insight generated by CEOs and their tools today are rarely utilized to their full potential,” Dunham said. “This is in part due to SEO not being seen as a business priority. As a result, it’s been siloed — pulling in teams from across the organization breaks down those silos.”
The more team members are involved with search processes, the more they’ll see its impact. People from each department will have more opportunities to contribute to growing online visibility using their unique skillsets.
“We know that businesses that are able to implement these organizational-wide search operations systems and practices — connecting a range of perspectives and search activities that are happening — are going to be the ones that will have a competitive advantage,” said Dunham.
Apply SEO testing automation
More and more brands are turning to automation tools to streamline tasks. According to Dunham, these solutions can be used for search-related activities as well.
“Automation can be well-deployed within web development processes,” Dunham said. “Until recently, this technology didn’t exist.”
Brands now have access to a wide variety of automation tools to streamline SEO-related tasks. The key is to pick solutions that align with your organization’s goals and give you full control over their deployment: “There are additional risk mechanisms that can be put in place to ensure you don’t release bad code that will result in large traffic losses, ultimately driving down revenue across your critical web pages,” said Dunham.
If brands can optimize their internal process, teams and tools around organic search, they’ll increase their chances of achieving long-term success in the search-first digital landscape.
That’s 100B times: • Your search history wasn’t a data point • What you searched (like cat memes) was your business & only yours • You recognized your right to privacy & chose it!pic.twitter.com/HyvMOBK26E
A year ago, the search company announced it was hitting over a 100 million searches per day. Now if you look at its public traffic statistics page, it shows all time searches passed 100,024,437,307 and the highest daily number of queries it saw to date was 110,439,133, that record was this past Monday. DuckDuckGo continues to grow on a daily basis.
Why we care. Pressure over consumer privacy has prompted Apple and Google to block third-party cookies from tracking users across the web. That same focus on privacy has also helped propel DuckDuckGo past 100 billion searches all-time. Its success, while modest, may provide new or existing search engines with a roadmap to chipping away at Google’s dominance or avoiding it altogether by concentrating on an underserved base of users.
It is still miles behind Google, but DuckDuckGo is gradually closing in on Yahoo! and Bing, so a future in which it is as much a part of the conversation as Bing may not be that far off. Nevertheless, optimizing specifically for any non-Google search engine remains highly unlikely.
The Microsoft Bing team said that the IndexNow protocol is now at a place where those participating are co-sharing URLs submitted, meaning if you use IndexNow to submit URLs to Microsoft Bing, Microsoft will immediately share those URLs with Yandex, the company announced.
Co-sharing URLs. The promise of IndexNow was to submit a URL to one search engine via this protocol and not only will that search engine immediately discover that URL, but it will also be discovered on all the other participating search engines. Right now, that is just Microsoft Bing and Yandex, but Google is exploring using this protocol.
Microsoft said “the IndexNow protocol ensures that all URLs submitted by webmasters to any IndexNow-enabled search engine immediately get submitted to all other similar search engines. As a result of co-sharing URLs submitted to IndexNow-enabled search engines, webmasters just need to notify one API endpoint. Not only does this save effort and time for webmasters, but it also helps search engines in discovery, thus making the internet more efficient.”
Microsoft said that Bing “has already started sharing URLs from IndexNow with Yandex and vice-versa, with other search engines closely following suit in setting up the required infrastructure.”
When this first launched, the participating search engines have not yet begun co-sharing URLs – but now they are.
IndexNow API. Also, you no longer need to submit the URLs to https://www.bing.com/IndexNow?url=url-changed&key=your-key or https://yandex.com/indexnow?url=url-changed&key=your-key. IndexNow.org is also directly accepting these submissions at https://api.indexnow.org/indexnow?url=url-changed&key=your-key
Microsoft Bing updated this help document to make it easier to understand how to set this up at any of those URLs mentioned above.
80,000 sites. Microsoft said that 80,000 websites are now using IndexNow for URL submission. “80k websites have already started publishing and reaping the benefits of faster submission to indexation,” the company said. Last November, the company said 60,000 of those websites were using IndexNow directly through Cloudflare, which added a toggle button to turn on this feature for websites using Cloudflare.
What is IndexNow. IndexNow provides a method for websites owners to instantly inform search engines about latest content changes on their website. IndexNow is a simple ping protocol so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.
How it works. The protocol is very simple — all you need to do is create a key on your server, and then post a URL to the search engine to notify IndexNow-participating search engines of the change. The steps include:
Host the key in text file named with the value of the key at the root of your web site.
Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call.
Why we care. Like we said before, instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site. The protocol is very simple and it requires very little developer effort to add this to your site, so it makes sense to implement this if you care about speedy indexing. Plus if you use Cloudflare, it can be turned on with the flip of a switch.
Now that co-sharing URLs is enabled, you should see your content flow faster between Microsoft Bing and Yandex, hopefully other search engines will adopt this protocol going forward.
Yoast SEO added keyword phrase performance tracking to Yoast SEO 17.9, in partnership Wincher, the company announced. This lets those who use Yoast SEO to better understanding of how your posts are ranking over time in Google Search.
Wincher tool. The Wincher SEO tool is a tool that helps you track your keywords and positions over time. Here is what it looks like when you activate it in Yoast SEO:
Free and paid. This integration works with both Yoast SEO free and Yoast SEO Premium versions. The free version lets you track up to a total of 5 keywords for your entire site while the premium Wincher account lets you track up to 500 to 10,000 keywords depending on the plan you are on.
More details. To learn more about how to use the Wincher SEO tool in Yoast SEO, see this help document.
Why we care. Yoast SEO is a very popular add on for WordPress sites and this integration may be something SEOs want to turn on to track keyword performance over time. It is probably worth exploring adding for some of your client sites as well.
Yoast said “this integration gives you a better understanding of how your most important posts are ranking over time and can give you the right tools to climb to that #1 position.” I don’t know if it will help you climb to that number one spot, but it can show you if you do or how much further you have to go to hit that position.
Understanding your current marketing processes, knowing how to measure success, and being able to identify where you are looking for improvements, are all critical pieces of the SEO platform decision-making process. But deciding whether your company needs an SEO platform calls for the same evaluative steps involved in any software adoption, starting with a comprehensive self-assessment.
Use the following questions as a guideline to determine the answers.
Do we have the right human resources in place?
Employing people to implement and use SEO platforms is a prerequisite to success. If you have marketing staff, utilizing SEO toolsets can make them more efficient and effective. The vast majority of organic search marketers struggle to justify their SEO budgets. SEO platforms and tools are a key component of helping to keep overall costs down while getting the required work done. Their analytical capabilities can also help SEOs prove the impact of their work on the bottom line.
Do we have C-level buy-in?
Enterprise SEO software can be a five- or six-figure investment annually. It is critical to demonstrate the value of SEO to C-level executives by running pilot test projects and agreeing to a definition of “success” in advance.
Do we have the right technical resources?
Successful SEO needs dedicated technical resources deployed to it to act on the recommendations and opportunities surfaced by the analytics and reports.
Who will own enterprise SEO?
Enterprise SEO is commonly placed into marketing, editorial or IT, depending on the nature of the business. Unfortunately, in large companies, it usually ends up with either whoever has the budget, or whoever can best articulate the business case. In a best-case scenario, it should be both.
Can we invest in staff training?
It is vital to provide training to technical, design, content, and marketing teams, and reinforce it on a regular basis. A successful enterprise SEO implementation will find ways to inject SEO knowledge into existing training programs and identify internal evangelists to broadly distribute the messages. Training needs to be comprehensive, consistent, and continuous. Some tool companies include or offer training for an additional fee, so be sure to ask about this.
To what extent do we need to share reports with non-SEO staff?
Some tool providers focus significant development resources on simple interfaces that can be utilized by people in other organizational roles — such as writers or C-suite executives. If this is important to you, make sure you specifically look for this when evaluating possible platforms.
Have we established KPIs and put a system in place for tracking, measuring, and reporting results?
It’s important to know upfront what you want your SEO to achieve. Do you want to improve SERP rankings or the time visitors spend on your site? Is conversion – whether a product purchase or whitepaper download – your key objective? Having goals will help you decide if you’re ready to put an enterprise platform to good use, as well as help you decide which tool will best meet your organizational needs.
How will we measure success?
Depending on your site’s monetization strategy, make sure you know how you’ll determine if the rollout of the platform and the successful execution of the established KPIs actually increased sales, conversions or pageviews.
Do we have realistic expectations?
It is not uncommon for enterprise SEO efforts to take at least six months to generate tangible results. If SEO is a new initiative within the organization, there are cultural shifts and workflow processes that will need to be implemented and refined. Setting realistic timelines and goals will help build support at all levels of the enterprise.
Do we have an SEO culture?
Many organizations begin to invest in SEO but find that a lack of understanding of SEO across the organization cripples its progress. Broad educational programs are often required to provide consistent performance and results.
Malte Ubl, a Software Engineer at Google, reminded us that coming next month, Google will be releasing the desktop version of the page experience update. He posted this on Twitter as a reminder:
Page experience ranking in Google Search based on Core Web Vitals on Desktop (Laptop/Tablets/the 523.94 remaining actual Desktop computers) is going out next month. Metric threshold are the same as on mobile. https://t.co/oVFF0W2Zihhttps://t.co/KKBIEi6v0Z
Google told us this was coming this past November and it should go live next month. The rollout will take a couple months, it will start in February 2022 and finish rolling out by the end of March 2022. This update will include all the current signals of the mobile version of the page experience update, outside of the page needing to be mobile friendly.
Same thresholds. As a reminder, the same metric thresholds will work for desktop. So what the thresholds were for mobile, will be the same for desktop. The original blog post said “the same three Core Web Vitals metrics: LCP, FID, and CLS, and their associated thresholds will apply for desktop ranking.” Malte Ubl reconfirmed this in his tweet saying “metric threshold are the same as on mobile.”
Don’t expect drastic changes. Google said with this rollout and this new Google update, do not expect drastic changes. “While this update is designed to highlight pages that offer great user experiences, page experience remains one of many factors our systems take into account… Given this, sites generally should not expect drastic changes,” said Google. We expect the same to be true for the desktop rollout.
Search Console tools any day now. Google will be updating the Google Search Console tools and reports to help site owners prepare for this update. “We are also planning to help site owners understand how their desktop pages are performing with regards to page experience using a Search Console report which will launch before desktop becomes a ranking signal,” Google said. So expect these new tools and reports to be released any day now.
Mobile vs desktop. Which factors is will be included in this desktop version? Google said all of them with the exception of the mobile friendliness requirement, which is kind of obvious. Here is a chart Google designed showing the specific factors:
Why we care. As I said last time, while I do not believe this page experience update will be a significant update where you will see tons of sites see their rankings drastically change, those working towards improving their page experience have been primarily focused on their mobile pages. Now, that you have your mobile pages ready for this update, you can shift focus towards your desktop pages.
The new reports are not out yet, but I hope they will be out shortly.
The majority of website operators are unaware of the importance of web server logs. They do not record, much less analyze their website’s server logs. Large brands, in particular, fail to capitalize on server log analysis and irretrievably lose unrecorded server log data.
Organizations that choose to embrace server log analysis as part of their ongoing SEO efforts often excel in Google Search. If your website consists of 100,000 pages or more and you wish to find out how and why server logs pose a tremendous growth opportunity, keep reading.
Why server logs matter
Each time a bot requests a URL hosted on a web server a log record entry is automatically created reflecting information exchanged in the process. When covering an extended period of time, server logs become representative of the history of requests received and of the responses returned.
The information retained in server log files typically include client IP address, request date and time, the page URL requested, the HTTP response code, the volume of bytes served as well as the user agent and the referrer.
While server logs are created at every instance a web page is requested, including user browser requests, search engine optimization focuses exclusively on the use of bot server log data. This is relevant with regard to legal considerations touching on data protection frameworks such as GDPR/CCPA/DSGVO. Because no user data is ever included for SEO purposes, raw, anonymized web server log analysis remains unencumbered by otherwise potentially applicable legal regulations.
It’s worth mentioning that, to some extent, similar insights are possible based on Google Search Console Crawl stats. However, these samples are limited in volume and time span covered. Unlike Google Search Console with its data reflecting only the last few months, it is exclusively server log files that provide a clear, big picture outlining long-term SEO trends.
The valuable data within server logs
Each time a bot requests a page hosted on the server, a log instance is created recording a number of data points, including:
The IP address of the requesting client.
The exact time of the request, often based on the server’s internal clock.
150.174.193.196 is the IP of the requesting entity.
[15/Dec/2021:11:25:14 +0100] is the time zone as well as the time of the request.
"GET /index.html HTTP/1.0" is the HTTP method used (GET), the file requested (index.html) and the HTTP protocol version used.
200 is the server HTTP status code response returned.
1050 is the byte size of the server response.
"Googlebot/2.1 (+http://www.google.com/bot.html)" is the user agent of the requesting entity.
"www.example.ai" is the referring URL.
How to use server logs
From an SEO perspective, there are three primary reasons why web server logs provide unparalleled insights:
Assisting to filter out undesirable bot traffic with no SEO significance from desirable search engine bot traffic originating from legitimate bots such as Googlebot, Bingbot or YandexBot.
Providing SEO insights into crawl prioritization and thereby enabling the SEO team with an opportunity to proactively tweak and finetune their crawl budget management.
Allowing for monitoring and providing a track record of the server responses sent to search engines.
Fake search engine bots can be a nuisance, but they only rarely affect websites. There are a number of specialized service providers like Cloudflare and AWS Shield that can help in managing undesirable bot traffic.In the process of analyzing web server logs, fake search engine bots tend to play a subordinate role.
In order to accurately gauge which parts of a website are being prioritized other than major search engines, bot traffic has to be filtered when performing a log analysis. Depending on the markets targeted, the focus can be on search engine bots like Google, Apple, Bing, Yandex or others.
Especially for websites where content freshness is key, how frequently those sites are being re-crawled can critically impact their usefulness for users. In other words, if content changes are not picked up swiftly enough, user experience signals and organic search rankings are unlikely to reach their full potential.
Only through server log filtering is it possible to accurately gauge relevant search engine bot traffic.
While Google is inclined to crawl all information available and re-crawl already known URL patterns regularly, its crawl resources are not limitless. That’s why, for large websites consisting of hundreds of thousands of landing pages, re-crawl cycles depend on Google‘s crawl prioritization allocation algorithms.
That allocation can be positively stimulated with reliable up-time, highly responsive web services, optimized specifically for a fast experience. These steps alone are conducive to SEO. However, only by analyzing complete server logs that cover an extended period of time is it possible to identify the degree of overlap between the total volume of all crawlable landing pages, the typically smaller number of relevant, optimized and indexable SEO landing pages represented in the sitemap and what Google regularly prioritizes for crawling, indexing and ranking.
Such a log analysis as an integral part of a technical SEO audit and the only method to uncover the degree of crawl budget waste. And whether crawlable filtering, placeholder or lean content pages, an open staging server or other obsolete parts of the website continue to impair crawling and ultimately rankings. Under certain circumstances, such as a planned migration, it is specifically the insights gained through an SEO audit, including server log analysis, that often make the difference between success and failure for the migration.
Additionally, the log analysis offers for large websites critical SEO insights. It can provide an answer to how long Google needs to recrawl the entire website. If that answer happens to be decisively long — months or longer — action may be warranted to make sure the indexable SEO landing pages are crawled. Otherwise, there’s a great risk that any SEO improvements to the website go unnoticed by search engines for potentially months after release, which in turn is a recipe for poor rankings.
A high degree of overlap between indexable SEO landing pages and what Google crawls regularly is a positive SEO KPI.
Server responses are critical for great Google Search visibility. While Google Search Console does offer an important glimpse into recent server responses, any data Google Search Console offers to website operators must be considered a representative, yet limited sample. Although this can be useful to identify egregious issues, with a server log analysis it is possible to analyze and identify all HTTP responses, including any quantitatively relevant non-200 OK responses that can jeopardize rankings. Possible alternative responses can be indicative of performance issues (e.g., 503 Service Unavailable scheduled downtime) if they are excessive.
Excessive non-200 OK server responses have a negative impact on organic search visibility.
Where to get started
Despite the potential that server log analysis has to offer, most website operators do not take advantage of the opportunities presented. Server logs are either not recorded at all or regularly overwritten or incomplete. The overwhelming majority of websites do not retain server log data for any meaningful period of time. This is good news for any operators willing to, unlike their competitors, collect and utilize server log files for search engine optimization.
When planning server log data collection, it is worth noting which data fields at a minimum must be retained in the server log files in order for the data to be usable. The following list can be considered a guideline:
remote IP address of the requesting entity.
user agent string of the requesting entity.
request scheme (e.g., was the HTTP request for http or https or wss or something else).
request hostname (e.g., which subdomain or domain was the HTTP request for).
request path, often this is the file path on the server as a relative URL.
request parameters, which can be a part of the request path.
request time, including date, time and timezone.
request method.
response http status code.
response timings.
If the request path is a relative URL, the fields which are often neglected in server log files are the recording of the hostname and scheme of the request. This is why it is important to check with your IT department if the request path is a relative URL so that the hostname and scheme are also recorded in the server log files. An easy workaround is to record the entire request URL as one field, which includes the scheme, hostname, path and parameters in one string.
When collecting server log files, it is also important to include logs originating from CDNs and other third-party services the website may be using. Check with these third-party services about how to extract and save the log files on a regular basis.
Overcoming obstacles to server log analysis
Often, two main obstacles are put forward to counter the urgent need to retain server log data: cost and legal concerns. While both factors are ultimately determined by individual circumstances, such as budgeting and legal jurisdiction, neither have to pose a serious roadblock.
Cloud storage can be a long-term option and physical hardware storage is also likely to cap the cost. With retail pricing for approximately 20 TB hard drives below $600 USD, the hardware cost is negligible. Given that the price of storage hardware has been in decline for years, ultimately the cost of storage is unlikely to pose a serious challenge to server log recording.
Additionally, there will be a cost associated with the log analysis software or with the SEO audit provider rendering the service. While these costs must be factored into the budget, once more it is easy to justify in the light of the advantages server log analysis offers.
While this article is intended to outline the inherent benefits of server log analysis for SEO, it should not be considered as a legal recommendation. Such legal advice can only be given by a qualified attorney in the context of the legal framework and relevant jurisdiction. A number of laws and regulations such as GDPR/CCPA/DSGVO can apply in this context. Especially when operating from the EU, privacy is a major concern. However, for the purpose of a server log analysis for SEO, any user-related data is of no relevance. Any records that can not be conclusively verified based on IP address are to be ignored.
With regard to privacy concerns, any log data which does not validate and is not a confirmed search engine bot must not be used and instead can be deleted or anonymized after a defined period of time-based on relevant legal recommendations. This tried and tested approach is being applied by some of the largest website operators on a regular basis.
When to get started
The major question remaining is when to start collecting server log data. The answer is now!
Server log data can only be applied in a meaningful way and lead to actionable advice if it is available in sufficient volume. The critical mass of server logs’ usefulness for SEO audits typically ranges between six and thirty-six months, depending on how large a website is and its crawl prioritization signals.
It is important to note that unrecorded server logs can not be acquired at a later stage. Chances are that any efforts to retain and preserve server logs initiated today will bear fruits as early as the following year. Hence, collecting server log data must commence at the earliest possible time and continue uninterrupted for as long as the website is in operation and aims to perform well in organic search.
