Publishing a page no longer guarantees that a search engine will keep it. Webmasters have noticed a steady shift over the past two years: crawlers visit as often as ever, but indexes have become selective, quietly dropping pages judged thin, duplicated or unhelpful. The change analysed across our search industry coverage is reshaping SEO priorities — from getting indexed at all, to deserving to be.
The drivers are scale and cost. The web now produces more pages daily than any index can economically store, and AI-generated content has multiplied the flood. In response, search engines have raised the bar: pages need a clear purpose, original information and signals of real use — links, engagement, distinctive expertise — to earn and keep a place in the index.
What gets dropped first
Site owners auditing their coverage reports find a familiar pattern among the casualties: tag and filter pages with near-identical text, location pages cloned by the thousand with only a town name changed, scraped or spun articles, and orphaned pages nothing links to. Large sites feel it most; small, focused sites often report their indexing is cleaner than ever, because nearly everything they publish carries a distinct reason to exist.
Google’s own guidance through Search Central has grown explicit that indexing is not guaranteed and that quality thresholds apply site-wide: a domain drowning in thin pages can drag down the crawl priority of its good ones.
The new playbook
Practitioners are adapting with pruning shears. Consolidate overlapping pages into one strong resource. Delete or noindex utility pages that were never meant to rank. Spend the crawl budget saved on fewer, better URLs, and watch the “crawled — currently not indexed” report as an editorial verdict rather than a technical bug.
News and niche sites feel it differently
The selectivity lands unevenly across the web. News publishers, whose value is speed, fight a different battle — convincing crawlers to visit constantly while their archives are pruned behind them. Specialist sites report the opposite fortune: a small catalogue site or a deep hobby resource, every page genuinely distinct, often sails through unaffected and even gains visibility as bloated competitors thin out. The common thread is intent. Pages built to answer something specific for someone real are being kept; pages built to occupy a query are being shown the door. Indexation, once a plumbing metric reported by technicians, has become an editorial conversation in publishing houses — which is, the search engines would say, precisely the point of the exercise.
The shift has a philosophical edge. For two decades, getting into the index was step one and ranking was the contest. Now admission itself is the contest. Sites that treat every page as a small act of publishing — original, necessary, maintained — are discovering that the pickier index rewards exactly the behaviour their readers wanted all along.
