In SEO for large sites, a significant share of the problems often comes from technical mistakes replicated across thousands of pages: duplicate URLs, wasted crawling, migration without a redirect map, and slow templates. On a small site such a mistake hurts a few pages, while on a large site it can hurt the organic traffic of broad sections. Many of these problems aren't visible in normal browsing, but they often leave traces in Google Search Console.
In brief
- סריקה היא משאב. On very large sites, unnecessary URLs can take crawl time away from pages it's important for Google to know.
- כתובת אחת לכל תוכן. Duplicates and conflicting signals cause Google to choose for itself which page to show.
- הגירה מתחילה במיפוי. Every old URL needs a direct destination via a 301 redirect, and redirects stay in place for at least a year.
- מתקנים בתבנית, לא בעמוד. On a large site, one fix in a template affects thousands of pages at once.
What is the difference between SEO for large sites and SEO for a regular site?
The main difference is scale: on a large site, the question is not only how a page ranks, but whether Google reaches it at all, understands it and chooses it.
Organic search optimization is the work intended to increase traffic from search engines without paying for each click. On a site of dozens of pages, most of the work is content and links. On a site of tens of thousands of pages, such as an online store, a content site or a listings site, a large part of the work is technical SEO. Technical SEO is handling the infrastructure that allows search engines to crawl, render and index the site: URL structure, redirects, speed, code and structured data.
According to Google Search Central, the guide to managing crawl budget is intended mainly for sites with more than a million unique pages whose content changes about once a week, or for sites with more than 10,000 pages that change daily. Google notes that these are rough estimates, not precise thresholds. A site where many URLs are classified in the indexing report as Discovered – currently not indexed also belongs to this group, regardless of its size. However, this status alone does not prove a crawl budget problem, and other causes are worth checking too, such as weak internal linking or pages in which Google didn't find enough value.
What is crawl budget, and when is it really a problem?
Crawl budget is the number of URLs Google can and wants to crawl on a site, and it becomes a problem when low-value URLs take the place of important pages.
According to Google, the budget is determined by two factors. The first is the crawl capacity limit, meaning how much load Google can put on the server without overloading it. A server that responds quickly and stably gets more crawling. When response times lengthen or 5xx or 429 errors appear, Google slows down. The second factor is crawl demand, which is influenced by the popularity of the pages, the need to refresh Google's copy of them and the number of duplicate or unnecessary URLs Google identifies.
A common source of waste is faceted navigation. Filters, sorting options and parameters can generate an almost unlimited number of URLs for the same content. Google recommends consolidating duplicate content, blocking in robots.txt URLs that there's no point crawling, and not using noindex for this purpose, because Google still requests those pages. Pages removed permanently should return 404 or 410, and soft 404 pages, which display "not found" but return a success code, keep consuming crawl budget.
How do you prevent duplicate pages and pages that compete with each other?
Define one canonical URL for each piece of content, and make sure all the signals on the site point to it consistently.
Canonicalization is choosing one preferred URL out of several URLs that show identical or very similar content. If you don't specify it, Google chooses for itself. Even when you do specify it, it's a signal Google considers rather than a binding directive. According to Google, redirects and the rel="canonical" tag are strong signals, and inclusion in the sitemap is a weak signal. They can be combined, as long as they all point to the same URL.
Google lists several common mistakes. Don't use robots.txt or noindex to choose a canonical URL, don't give different canonical URLs to the same page through different methods, and use absolute rather than relative URLs. An easy mistake to miss is internally linking to the duplicate version instead of the canonical version. When the menu links to a URL with a parameter and the tag points to a clean URL, Google receives two conflicting messages.
Pages that compete for the same intent
Pages that aren't technically duplicates can also compete with each other, when several of them answer the same question. In such a situation the search engine has difficulty deciding which one to show, and sometimes none of them ranks high. A common solution is an intent map: one page responsible for each search intent, with overlapping pages merged via redirect or given a different angle.
How do you migrate a site with 301 redirects and minimize the impact on traffic?
Map every old URL to an equivalent new URL, redirect with a direct 301 redirect, and keep the redirects for at least a year.
A site migration is moving a site to a new URL structure, a new system or a new domain. A 301 redirect is a permanent server-side redirect that tells the browser and the search engine the URL has moved permanently. According to Google, permanent redirects don't cause a loss of PageRank, so they are the main tool in any migration. However, even correct redirects don't guarantee a migration without fluctuations in ranking and traffic.
According to Google's guide to moving a site with URL changes, here is what's important to check:
- מיפוי מלא. A list of old URLs from sitemaps, server logs, analytics and Search Console, including images, video, JavaScript and CSS files.
- הפניה ישירה. From the old URL straight to the final destination. Googlebot can follow up to 10 hops, but Google recommends short chains, ideally no more than 3.
- בלי הפניה גורפת לדף הבית. Redirecting many URLs to an irrelevant page hurts the transfer of signals.
- עדכון האתר החדש. Canonical tags, hreflang and internal links should point to the new URLs.
- מפת אתר חדשה. Submit it in Search Console, and track the indexing and query reports.
Google notes that on a medium-sized site it may take a few weeks or more for the new URLs to appear gradually, and on large sites it may take longer, and ranking changes during this period are expected. Large sites can move in stages, by section. The Change of Address tool in Search Console is suitable only for a domain or subdomain move, not for moving to HTTPS or changing paths.
Do Core Web Vitals affect organic search?
Yes, they are part of the page experience Google recommends improving, but Google doesn't publish how much weight they have in ranking.
Core Web Vitals are three metrics of user experience on a page. According to Google Search Central, as of the article's last update they are LCP, which measures loading and is considered good up to 2.5 seconds; INP, which measures responsiveness and is considered good up to 200 milliseconds; and CLS, which measures visual stability and is considered good up to 0.1. Google writes that a good page experience aligns with what its ranking systems seek to reward.
On a large site, it's worth measuring by template rather than by individual pages. According to Google, the Core Web Vitals report in Search Console is based on real usage data and groups URLs with similar user experience, so it's easy to identify a product page or category page template that slows down the whole group. Uncompressed images, third-party scripts and late-loading components that shift the page are common causes.
Why is structured data especially important on large sites?
Because on a large site it's implemented once in the template, and every new page gets it automatically, but one fault in the template can also break it on every page based on that template.
Structured data is markup in an agreed format that describes to the search engine what's on the page: a product, an article, questions and answers or company details. According to Google, the recommended format is JSON-LD, and the markup must describe content visible to the user on the page. Adding structured data doesn't guarantee a rich result, and to be eligible for one the markup must include all the required properties.
Google recommends testing the markup with the Rich Results Test during development, and tracking it after launch in the rich results reports in Search Console. On large sites this is especially important, because a change in the template or in the way the page is served can quietly break the markup. In our view, a clear structure also matters for visibility in AI engines, which we wrote about in the article מה זה GEO/AEO.
Which technical SEO mistakes are common, and how do you identify them in Search Console?
Many of the mistakes appear as a familiar status in the page indexing report. The table shows possibilities to check rather than a diagnosis for every large site, because the same status can stem from several causes.
| Mistake | What you see in Search Console | The fix |
|---|---|---|
| Filters and parameters that generate endless URLs | Many URLs with the status Discovered – currently not indexed | Blocking valueless parameters in robots.txt and linking internally only to the canonical version |
| Duplicates without a canonical URL | Duplicate without user-selected canonical | A complete and consistent rel="canonical" tag on all versions |
| Conflicting signals for the canonical URL | Duplicate, Google chose different canonical than user | Aligning the tag, redirects, sitemap and internal links to the same URL |
| Removed pages that return a success code | Soft 404 | Returning 404 or 410 for pages removed permanently |
| Redirect chains or loops | Redirect error | A direct redirect to the final destination and updating internal links |
| A server that can't handle the crawl load | Server error (5xx), and Hostload exceeded in the URL Inspection tool | Increasing server resources and improving response times |
| Migration without a redirect map | A spike in URLs with the status Not found (404) after launch | Mapping every old URL to an equivalent destination with a 301 redirect |
| Structured data markup broken in the template | Errors in the rich results reports | Fixing the template and retesting in the Rich Results Test |
| Slow templates | Groups of URLs with Poor or Need improvement status in the Core Web Vitals report | Improving the template itself: images, scripts and layout stability |
The status Alternate page with proper canonical tag, on the other hand, is usually a sign that canonicalization is working. According to Google, no action is required in this case.
What do you check in a technical SEO audit for a large site?
Check the whole chain, from crawling to display in the results, and prioritize by impact on the pages that bring traffic and revenue.
- דוח אינדוקס הדפים. Which statuses grew, in which sections of the site, and since when.
- מלאי כתובות. How many URLs exist compared with how many pages there really are, and what share of them was generated by parameters.
- robots.txt ומפות אתר. What is blocked, what is included, and whether the sitemaps contain only canonical URLs that return 200, with a correct update date.
- כתובות קנוניות. Alignment between the tag, the redirects, the sitemap and the internal links.
- הפניות. Chains, loops, temporary redirects instead of permanent ones, and blanket redirects to the homepage.
- קודי תגובה וביצועי שרת. 5xx errors, soft 404 pages and response times, especially at peak hours.
- Core Web Vitals לפי תבנית. Product page, category, article and homepage, each separately.
- נתונים מובנים. Whether the markup exists in every template, is valid, and describes what is visible on the page.
- קישוריות פנימית. Whether important pages are reachable in a small number of clicks, and whether there are orphan pages.
- מפת כוונות. Whether each main search intent has one page responsible for it.
How we approach this at CONNECT
At CONNECT, part of the Bruckner Yaar Levi advertising group, SEO work for a large site starts with a full technical audit, and ends with a list of fixes prioritized by impact on the pages that bring traffic and revenue. We work at the template level rather than the individual page level, and coordinate with the development team so the fixes go into the regular development cycles.
Before a migration we build a full redirect map and test it before and after going live. After launch we track the indexing, traffic and rankings of the important pages every month, and examine the same pages from the perspective of AI engines too, when the work also includes GEO/AEO.
Frequently asked questions
From what site size do you need to worry about crawl budget?
According to Google, the crawl budget guide is intended mainly for sites with more than a million unique pages that change about once a week, or for sites with more than 10,000 pages that change daily. Google emphasizes that these are rough estimates. A smaller site where many URLs are classified as Discovered – currently not indexed should also check the topic. If new pages are crawled on the day they are published, there is probably no problem.
How long does it take Google to process a site migration?
According to Google, on a medium-sized site it may take a few weeks or more for the new URLs to appear gradually in the results, and on large sites it may take longer. During this period, ranking fluctuations are expected. To shorten the process, it's important that the redirects are direct, that the new sitemap is submitted in Search Console, and that the new server can handle the increased crawl load that comes after a move.
How long should 301 redirects be kept after a migration?
Google recommends keeping the redirects for at least a year, and preferably longer. During this time Google transfers the signals to the new URLs and recrawls links pointing to the old URLs. Other sites continue to link to the old URLs even years after the move, so removing the redirects early may lose links built over time.
Does noindex save crawl budget?
No. According to Google, the noindex tag is not suitable for managing crawl budget, because Google still needs to request the page to see the tag. To prevent crawling of valueless URLs, use robots.txt. On the other hand, robots.txt is not suitable for choosing a canonical URL, because a blocked URL can still be indexed. Each tool has a different role, and it's important not to mix them up.
Does structured data guarantee rich results?
No. According to Google, structured data makes a page eligible for rich results, as long as it includes the required properties and complies with the guidelines, but Google does not commit to showing such a result. The markup must describe content visible on the page. The recommended format is JSON-LD, and validity is checked with the Rich Results Test and the rich results reports in Search Console.
ראו גם: Organic Search (SEO) · AI Search Optimization (GEO/AEO) · מה זה GEO/AEO · E-commerce Performance · Data & Measurement
Sources: Google Search Central – Managing crawl budget for large sites · Google Search Central – Moving a site with URL changes · Google Search Central – Consolidating duplicate URLs · Search Console – Page indexing report · Google Search Central – Introduction to structured data · Google Search Central – Core Web Vitals · Search Console – Core Web Vitals report
רוצים ביקורת טכנית לאתר שלכם? השאירו פרטים.