Server logs for SEO: practical use cases

If you want to land an SEO job, make sure to tell them you like server logs. A lot of hiring managers and SEOs are impressed by that. Why? I have no idea. Sure, I've used server logs in the past, but with a specific problem in mind. If you have no idea what to use them for, or what problem you want to solve with them, they're pretty useless huge text files in my opinion.

Therefore, in this short guide, I'll show some ideas on how to use server logs for SEO problems you might find during your work.

Wouter van der Meij
Wouter van der Meij
Follow Wouter:

Swipe the summary

Here are some ideas on how to use server logs for SEO:

Create a sitemap of all URLs that haven't received a direct hit from Googlebot in the past X days.

Map out what types of pages are crawled most often by GoogleBot.

Find errors: 404s, 410s. Note: you don't see 500 errors in your logs, a 500 error is a problem on your server, so the error will never be written to the log.

Find and block unwanted bots

Create a list of often loaded files and crawl their size. Maybe you can optimize the size of files that get a lot of hits.

How many hits go to products that are sold out or are out of stock?

Analyze crawl waste: find out what percentage of GoogleBot hits go to files and URLs that don't need to be crawled.

Analyze the crawl depth of the website. See how deep Google goes and how often she crawls that deep.

Server log analysis: practical use cases
  1. Create a sitemap of all URLs that haven't received a direct hit from Googlebot in the past X days. This can be a great way to get new pages indexed fast. Especially handy for big eCommerce sites, news sites and job listing sites.
  2. Map out what types of pages are crawled most often by GoogleBot. If a type of page is crawled a lot that doesn't need to, is there a reason for it? (for example, it shows a livestream of data somewhere in the sidebar, and is therefore always updated when the bot comes by).
  3. Find errors: 404s, 410s and 302 redirects. Note: you don't see 500 errors in your logs, a 500 error is a problem on your server, so the error will never be written to the log.
  4. Find and block unwanted bots.
  5. Create a list of often loaded files and crawl their size. Maybe you can optimize the size of files that get a lot of hits.
  6. How many hits go to products that are sold out or are out of stock?
  7. Analyze crawl waste: find out what percentage of GoogleBot hits go to files and URLs that don't need to be crawled.
  8. Analyze the crawl depth of the website. See how deep Google goes and how often she crawls that deep. If you have a straightforward URL structure you can even chunk it down to each level to see how often, and how far, Google goes.

Server log analysis might sound as advanced SEO and you might impress some hiring managers by simply mentioning it in your interview. But, make sure you're using them for something useful. I hope this short list will help in some ways and gives you a little inspiration to think outside the regular SEO box.