Metadata optimisation - PART II

Why there is a second part

Part two of rank analysis: the readings that appear once a few months of history have accumulated. What a single measurement cannot say, a time series can.

Part one of rank analysis is about reading a single measurement: separating noise from signal, subtracting the control group, computing the effect of a move. Those are daily jobs.

But after three months you hold something else: a time series. And a time series answers four questions single measurements cannot.

The first: does this phrase actually belong to us? If you have sat in the same band for three months, that is your natural place. If you keep oscillating, you are not holding on there.

The second: are our moves accumulating? Individual effects may be small, but stacked they either produce a slope or they do not.

The third: what have we lost? Gains are noticed, losses are not. A time series shows the ranks quietly given back.

The fourth: is it the season or is it us? You can only see that a phrase moves at the same time every year with a year of history.

Do the moves accumulate

A single metadata move's effect is usually small: a few places. The question worth asking is whether those small effects stack.

If they do, the average rank of the phrases on your list should be visibly better at the end of three months. That is a slope no individual move shows.

If they do not, there are two explanations. The first: every move wins in one place and loses in another. A word removed from the subtitle weakens the phrases it was holding up, and the net effect is zero.

The second: the market is moving at your speed. If you gain three places while your rivals gain three, your relative position has not changed. The control group's average over the same period shows that.

The simplest way to measure accumulation: record the average rank of the phrases on your list weekly and look at the three-month slope. When averaging, leave out the phrases outside 200 - their rank is unknown and an invented number distorts the average.

The second way is band counting: how did your number of top-10 phrases change week by week? That figure is sturdier than an average, because it is not affected by unknown values.

Seasonality

The clearest pattern to appear once a year of history exists is seasonality, and missed, it produces badly wrong decisions.

A phrase's popularity is not flat across the year. Tax apps behave differently in one month, fitness apps in January, travel apps around school holidays.

When popularity rises two things happen. Traffic from the phrase rises - good news. And the number of apps entering the phrase rises too, which means holding the same rank gets harder.

Those two can cancel each other out. In season your rank can fall while your traffic rises; out of season your rank can rise while your traffic falls. An analysis looking only at rank reads both wrongly.

The right comparison is the same period last year. Comparing against last month turns a seasonal move into an apparent effect of your own.

Without a year of history, the thing to do is look at the control group. If every rival moved in the same direction too, the cause may well be seasonal.

Joining the series to a record

A measurement on its own is a chart. Joined to a record it becomes an instrument for learning.

Three records have to be joined. Your own moves: date, country, which field, its old form, its new form. External events: releases, rating changes, a rival being pulled. Measurement gaps: which days a measurement could not be taken.

Marked on top of the chart, three months of curve becomes readable. Every jump has a candidate cause standing beside it.

The greatest benefit of that joining is keeping the negative results. "We tried adding a phrase to the subtitle five times and it did nothing twice" makes you choose the same move more carefully next time.

The second benefit is handover. So that when somebody leaves the team the measurement knowledge does not leave with them. An annotated time series lets a new person build context in a week.

The third is making the argument concrete. "Our rankings are bad" cannot be argued with; "we have fallen four places on average since mid-February and there was a release that week" can.

Frequently asked

How much history do I need before these readings work?

Three months is enough for band persistence and accumulation. Seasonality needs a year; with less you cannot separate a seasonal move from the effect of your own.

What do I do with the phrases outside 200 when averaging ranks?

Leave them out. Their rank is unknown and an invented number distorts the average. A sturdier measure is the weekly change in your count of top-10 phrases, which is not affected by unknowns.

How do I find the silent losses?

Scan the history backwards every quarter: which phrases were higher three months ago than today? The list is usually surprising, because nobody looks at something already won.

My moves do not seem to be accumulating. What should I check?

Two possibilities. Each move may be winning in one place and losing in another - check the words you removed from the subtitle. Or the market may be moving at your speed; look at the control group's average over the same period.

What if no decision comes out of the analysis?

There is usually a problem in the measurement setup: too many gaps, several moves made at once, or a missing control group. What needs fixing is the measurement, not the analysis.