SEOLivly
Menu
Home Tools Blog Services Contact Run the free scan

How to design an SEO experiment you can trust

By Derek Murphy  · 

Most SEO experiments prove nothing because they change five things at once and then argue about the result. An experiment needs one variable, one page group, one measurement and a date you decided in advance. Without those four, you have a vibe with a spreadsheet.

I learned this the expensive way, which is the only way most people learn it. The test that finally worked was the boring one: change the title on ten pages, leave everything else alone, and compare the impressions after six weeks.

Start with a question that has a possible no

"Will this help?" is not a question. "Does adding a year to the title increase clicks on pages that already rank between positions 5 and 15?" has a possible no, a defined group and a measurable outcome.

The narrower the question, the more likely the answer is useful. "Does local SEO work?" is a project. "Does adding the city to the title of the service page change local pack clicks?" is an experiment.

One variable

Change the title or the meta description, not both. Change the internal links or the body copy, not both. If you change two things, you cannot say which one moved the number, and the next experiment starts from a guess again.

This feels slow. It is faster than repeating the same ambiguous result for a year.

Choose the pages before you look at the results

Write down the URLs, the current metrics and the change. Do it before you edit anything. If you pick the pages after seeing which ones improved, you are not running an experiment, you are writing a story about one.

Match a control group if you can: ten pages you changed and ten similar pages you did not. On a small site the control group is imperfect because traffic varies, but it is better than comparing the changed pages to the whole site.

Measure the thing the change should affect

A title change should affect impressions and click-through rate before it affects conversions. A content update should affect impressions and average position over weeks. A speed fix should affect the metric you were actually trying to move, which may not be rankings at all.

Use Search Console for impressions, clicks and position. Use your analytics for sessions and conversions. Do not use rank trackers as the only source, because they sample a location and a device and can disagree with your own data.

Decide the length of the test in advance

Four to six weeks for a title or content test is a reasonable default on a small site. Anything shorter catches the noise of a weekend. Decide before you start, and do not extend the test because the first two weeks looked good.

Also decide what counts as a result. A 10% change in clicks on a small site can be noise. A change in average position from 12 to 9 across ten pages is more informative. Write the threshold down.

What to do with the result

If it worked, apply the change to the matched pages and write down the rule. If it did not, write down the rule too. Negative results are the most valuable thing a small team can accumulate, because they stop the next person from repeating the test.

One more thing: check the pages before and after with the website auditor. A change that broke a canonical or duplicated a title is not a failed experiment, it is a broken page. If the test moved a lot of pages, run a broken links check afterwards to make sure nothing fell over.

The honest version

A test log template

Keep one page per test. It takes two minutes to start and it is the difference between a library of knowledge and a folder of screenshots.

  • Question: the one sentence with a possible no.
  • Hypothesis: what you expect and why.
  • Pages: the URLs changed, and the control URLs if you have them.
  • Change: exactly what was edited, with a copy of the old text.
  • Metric: the number that should move, and the threshold for a result.
  • Start date and end date: decided before the first edit.
  • Result: the numbers, the confounders and the decision.

The confounders line matters more than it looks. A site redesign, a Google update, a holiday or a paid campaign can move the number while your test is running. Write down what else changed in the window, even if you cannot control it. The next person reading the log needs to know whether the result is clean.

After ten logs, patterns appear. You learn which titles work on your site, which pages respond to content updates, and which experiments are a waste of time. That is a more valuable asset than any single result.

If the test needs a developer, write the ticket before the experiment starts. Half of the tests that fail on small sites fail because the change was rolled out gradually over two weeks, which turns a clean comparison into a guess about which pages got the change when.

Most of what people call SEO experiments are content updates with a hypothesis attached. That is fine. The value is in the discipline: one change, a defined group, a measurement and a written result. Do that ten times and you know more about your site than any audit can tell you. The tools hub has the checkers for the pre-flight and the after-check, and the Fix Sprint is the version where I run the experiment and the fix together.

Check your own page

The free website auditor runs forty checks on one URL and lists the worst problems first. No signup. If you would rather hand the list to somebody, the fix service has the prices on the page.

Run the free scan

Keep reading