What 500 Competitor URLs Taught Me About Keyword Selection

I spent a stretch of last year doing something most SEOs talk about but rarely actually sit down and finish: I read five hundred competitor URLs, properly. Not skimmed a dashboard, not exported a list and filed it — read them. Title, slug, headings, intro, structure, internal links, the lot. Five companies in the same broad market, roughly a hundred ranking URLs each, examined one by one over several weeks.

I want to be careful with how I frame what came out of it, because the SEO industry has a bad habit of dressing up observation as measurement. I did not run a controlled experiment. I did not have a clean sample or a statistically defensible methodology. What I had was five hundred real pages and the patience to look at all of them. So treat what follows as qualitative pattern-finding — what consistently showed up when I looked closely — not as data with decimal points. The patterns were strong and repeated enough that I changed how I select keywords because of them. That is the honest framing, and it is also the useful one.

Why read URLs at all

The premise is simple. A competitor's ranking URL is the visible result of a keyword selection decision they made. Somebody chose to target that term, build that page, and try to rank for it — and the page either succeeded or it is sitting in a position that tells you how well the choice worked. Five hundred such URLs is five hundred keyword decisions, with outcomes attached. If you read enough of them, patterns surface: the kinds of choices that tend to produce strong pages, and the kinds that produce pages stuck in obscurity.

You cannot get this from a keyword tool. A tool gives you volume and a difficulty score for a term in isolation. It does not show you what happens when real companies in your market actually commit to terms and live with the results. The URLs do. They are a record of decisions and consequences, and that is a more honest teacher than any metric.

Pattern one: the specific pages were doing the heavy lifting

The clearest pattern, and the one that showed up in every single competitor's library, was about specificity. When I sorted each competitor's URLs by how well they appeared to be performing, the same shape appeared again and again. The pages holding strong, stable positions were overwhelmingly the specific ones — narrow topics, precise questions, clearly-defined sub-subjects. The pages targeting broad, head terms were, with very few exceptions, either ranking poorly or not really ranking at all.

I want to be precise about what I am and am not claiming. I am not claiming "X% of specific pages outrank broad pages." I did not measure that and I will not invent it. What I am saying is qualitative and it was unmistakable: across five hundred URLs and five companies, the correlation between a page being specific and a page performing was strong enough that you could see it without any analysis at all. The broad pages were a graveyard. The specific pages were where the traffic lived.

This is not a new idea — the case for long-tail keywords has been made many times. But there is a difference between knowing something as advice and seeing it as a pattern repeated five hundred times across companies you actively compete with. The second one changes behaviour. It changed mine.

Pattern two: the winning pages matched intent, not just words

The second pattern was about a subtler kind of mismatch. Plenty of competitor pages clearly targeted a reasonable keyword and were still underperforming — and when I read those pages closely, a recurring cause appeared: the page format did not match what the searcher actually wanted.

A page targeting an obviously comparison-style query — the kind where the searcher wants to weigh options — but built as a long narrative essay with no comparison structure. A page targeting a quick factual question stretched into a three-thousand-word guide the searcher would never finish. A page targeting a buying-intent term written as a gentle educational explainer that never acknowledged the reader was close to a decision. The keyword was fine. The page answered a different question than the one being asked.

The pages that performed were the ones where format and intent lined up: comparison queries answered with comparison pages, quick questions answered concisely, buying-intent terms met with pages that respected the reader's readiness to decide. This pushed me to a conclusion I now treat as a rule — keyword selection is not finished when you have picked the term. It is finished when you have also decided, correctly, what kind of page that term demands. Choosing the word is half the decision. (Our overview of turning keywords into a content plan covers the intent-labelling step in more detail.)

Pattern three: the slug told the story before the page loaded

A smaller observation, but a practically useful one. After a few dozen URLs I noticed I could predict a page's keyword strategy from its slug alone, before the page even loaded — and I was usually right.

A clean, descriptive slug like /reduce-email-bounce-rate told me the team had a specific keyword in mind and built the page around it deliberately. A vague slug — a date string, a post number, a slug crammed with five loosely-related words — told me the opposite: either no clear keyword decision, or a page targeting something too broad to name cleanly. And the predictive part: the clean, specific slugs disproportionately belonged to the better-performing pages. The messy slugs disproportionately belonged to the graveyard.

The slug is not the cause — nobody ranks because of a tidy URL. But the slug is an honest tell. A team that produces specific, descriptive slugs is a team doing specific, deliberate keyword selection, and that discipline shows up in results. When you are scanning a competitor's library, the slugs let you triage fast: read the clean-slug pages to learn what good selection looks like, skim the messy-slug pages to confirm what weak selection produces.

Pattern four: clustered pages held up; orphans drifted

The fourth pattern only became visible because I was reading whole libraries rather than isolated pages. I started noticing the internal linking — which pages were woven into a network of related articles, and which sat alone with no relevant pages linking to them.

The pages that were part of a cluster — surrounded by related articles, linked up to a broader hub, linking down to narrower ones — were consistently the steadier performers. The orphan pages, even some on perfectly good keywords, tended to underperform or hold weaker positions. Again, qualitative, not measured. But it was consistent enough that it reframed keyword selection for me as a structural decision rather than a list-building one. The question is not only "is this a good keyword?" It is "does this keyword belong to a group of related keywords I am also going to cover, so the resulting page has neighbours?" A great keyword chosen in isolation, with no related pages around it, performed worse in these libraries than a merely-good keyword chosen as part of a cluster. The structural context of a keyword choice mattered as much as the choice. Our guide to topic clusters is the right companion read here.

Pattern five: a lot of competitor rankings were quietly stale

The last pattern was the most encouraging one for anyone trying to compete. As I read, I kept checking how current pages were — and a meaningful share of competitor pages holding decent positions had clearly not been touched in years. Outdated references, old screenshots, advice that no longer reflected how things work, ranking anyway on accumulated authority.

For keyword selection, this reframes what a "taken" keyword means. When a tool shows a competitor ranking for a term, the instinct is to treat that term as occupied — someone strong already holds it, move on. But reading the actual pages told a different story. Plenty of those occupied positions were held by stale, mediocre pages coasting on age. That is not a closed door; it is a door held shut by inertia. A term where the ranking page is three years old and visibly tired is, in practice, one of the most winnable terms on the board — far more winnable than the difficulty score implies. I now treat "who ranks for this and how good is their page really" as a core input to keyword selection, and the only way to answer the second half of that question is to open the page and look.

How this changed my keyword selection

Five hundred URLs later, my selection process is different in four concrete ways, and each one traces directly to a pattern above.

First, I bias hard toward specificity — not because a guide told me to, but because I watched specific pages win and broad pages lose, five hundred times over. Second, I do not consider a keyword selected until I have also decided the correct page format for it, because intent mismatch was quietly killing competitor pages built on perfectly good terms. Third, I never select a keyword in isolation; I select it as a member of a cluster, because clustered pages held up and orphans drifted. Fourth, before I write off a keyword as "taken," I open the page that currently ranks and judge it honestly — because a surprising amount of competitor real estate is defended by nothing more than age.

None of these are revolutionary individually. The value was not the ideas; it was the conviction. Reading five hundred real pages turned four pieces of advice I sort-of believed into four rules I now actually follow. That is what close observation does that a dashboard cannot — it makes the abstract concrete enough to act on.

The honest limitation

I will end where I began, on the limits. This was five companies and roughly five hundred URLs in one broad market. The patterns were strong and consistent, but I cannot promise they generalise to every industry, and I did not measure anything precisely enough to attach a number to. Another market might behave differently. Treat this as a method as much as a finding: the real recommendation is not "believe my five patterns," it is "go read your own competitors' URLs, in volume, and see what patterns your market produces." The patterns are yours to discover. The exercise is what pays.

What I expected to find and didn't

It is worth being honest about the hypotheses that did not survive the reading, because the failed expectations were as instructive as the patterns that held.

I expected to find that page length correlated strongly with performance — that the long, comprehensive pages would dominate. They did not, at least not on their own. There were plenty of long competitor pages performing poorly and plenty of moderate-length pages performing well. Length, read across five hundred URLs, looked like a consequence of doing the topic justice, not a cause of ranking. The pages that won were the ones that fully answered the question; sometimes that took three thousand words and sometimes it took eight hundred. Padding a page to hit a word count was, if anything, visible as a weakness — the bloated pages tended to be the ones with the intent mismatch from pattern two.

I also expected publication recency alone to predict performance, and it did not cleanly. Some old pages were stale and weak, exactly as pattern five describes — but other old pages were old and excellent, kept current through updates, and performing as well as anything newer. "Old" was not the signal. "Old and untouched" was. That distinction matters for keyword selection: the question is never just how long ago a competitor published, it is whether they have maintained what they published. A competitor's age on a term tells you nothing until you also know whether they are still tending it.

The discipline the exercise really teaches

Stepping back from the five patterns, the deepest thing five hundred URLs taught me was not any single observation. It was a discipline: that keyword selection should be grounded in evidence about what actually happens in your market, not in metrics about keywords in the abstract.

A keyword tool encourages a particular mental model — keywords as entries in a database, each with a volume and a difficulty score, to be picked like items off a menu. Reading competitor URLs replaces that model with a better one: keywords as decisions other companies have already made, with visible consequences you can study. Once you have seen five hundred of those decisions and their outcomes, you stop selecting keywords by their abstract numbers and start selecting them by analogy to outcomes you have actually observed. "This term is specific, it fits a cluster I am building, the page currently ranking is stale, and the intent clearly wants a comparison" is a far richer basis for a decision than "volume 880, difficulty 34." The first is grounded in evidence. The second is grounded in a model that has never seen your market. The reading is what gets you from one to the other.

Where an AI agent fits

The honest problem with reading five hundred URLs is that it took weeks, and weeks is exactly why almost nobody does it. The method is sound; the manual labour is the barrier. Classifying each URL by specificity, judging format-versus-intent fit, reading slugs, mapping internal-link clusters, checking how stale each page is — that is enormous, structured, repetitive work.

It is also precisely what an SEO AI agent can do at a scale a person cannot sustain. Orova can read competitor URLs in volume, classify them by specificity and intent fit, flag stale pages whose positions are quietly contestable, and surface the structural patterns across a whole market — turning a multi-week reading project into an analysis you review rather than perform. The judgement that follows stays yours: which patterns matter for your business, which terms to chase, what to build. But the agent removes the reason this powerful exercise almost never gets finished — and an analysis you finish always beats the one you abandon at competitor two.