How to Keep RAG Data Fresh When You Don't Own the Source

Keeping RAG data fresh means making sure that the text your retriever finds still matches what the source says today. For documents you own, that is a solved problem: a webhook or a change feed tells your pipeline when something moved. This post is about how to keep RAG data fresh when the source is a public page on someone else's website, where nothing tells you anything. We compared archive copies of a fixed list of public pages to measure how fast such a copy goes stale. A third of the pages we could compare had lost at least 5% of their text within 30 days, and two thirds within 180 days.

Why does RAG data go stale?

Because the index is a copy, and a copy stops changing the moment you make it. A RAG pipeline loads a page, splits it into chunks, embeds them and stores them in a vector database. From then on the retriever answers from the stored version, and it has no way of knowing that the original moved. Similarity search ranks text by how close it is to the question. A chunk with last quarter's price scores exactly as well as it did last quarter.

The model then does what it was built to do with retrieved context, which is to trust it. The result is a confident answer that was correct at some point in the past, with a citation that looks fine. Nothing in the pipeline raises an error, so the first person to notice is usually the customer.

Why does the usual advice stop at sources you own?

Because the usual advice depends on the source telling you that it changed. We asked ChatGPT, Perplexity and Google AI Mode how to keep RAG data fresh before writing this, and all three recommend the same design: webhooks or change-data capture to detect edits, then re-embed only what changed, on a pipeline connected to SharePoint, Confluence, Notion or Postgres. That design is right. LangChain's indexing API is built for the second half of it, with a record manager that stores a hash of each document so unchanged content is not embedded again.

A page on another company's website gives you none of the first half. It has no webhook for you, and the answer engines give it one table row at most: "Websites: scheduled crawling", or "public web: daily crawl", with no measurement behind the number. The signals the web does have are aimed at someone else. IndexNow lets a site tell search engines that a URL was "added, updated, or deleted", but the notification goes to participating search engines, and only the site's owner can send it. A sitemap's `lastmod` field is available to anyone, but it is written by the site, and Google's own guidance is that it uses the value only "if it's consistently and verifiably" accurate, and ignores `changefreq` altogether.

So for a public source, you have to go and look. The practical questions are how often to look, and how to do it without paying to re-embed pages that did not change.

How fast does a public web page change?

Faster than a monthly refresh can follow. On 9 November 2026 we took a fixed list of 60 public pages, chosen before the run: 30 pricing pages for widely used software and AI products, 20 documentation and changelog pages, and 10 official pages that publish rates and statistics. For each one we asked the Internet Archive for its dated captures, took the newest, and compared it with the capture nearest to 7, 30, 90 and 180 days earlier. Both sides of each comparison came from the same archive crawler, so a difference is a change in the page and not a change in who asked. We removed scripts, styles, navigation and footers, and split what was left into blocks of text of 20 characters or more.

49 pages gave a usable comparison in at least one window. The other 11 had no recent captures or delivered almost no text without running scripts, so they say nothing either way.

The two right-hand columns measure different things. Losing 5% of a page's text can be a rewritten paragraph. Losing a number is the failure a RAG system pays for, because a number is what the user came for. The number column is a count of values that were in the old copy and are no longer anywhere on the new one, and we checked examples by hand. Mistral's pricing page listed 10, 15 and 30 dollars a month in API credits on 7 October. By 12 October it listed 15 and 25.5, and it still did on 8 November. The US Bureau of Labor Statistics' CPI page gave +0.4% for August on 9 October and +0.6% for September on 8 November. A copy indexed at the start of that month gives a wrong answer by the end of it, with nothing to show that it has.

Change was not even across the list.

Treat these as a floor on a small list. We chose the 60 by hand, so they are not a random sample of the web. Capture dates vary, so "30 days" means between 22 and 38. Three pages are index pages whose entries rotate by design. A rewording counts as a change even when the facts held. The list skews towards pages that are popular enough to be archived often, and we can only compare what the server sent, so pages built entirely in the browser fall out.

How do you detect changes on a website you do not own?

Ask the server first, then compare the content yourself. HTTP has had a mechanism for this for decades. A server can send an `ETag` or a `Last-Modified` header with a page. You store it, send it back on the next request, and if nothing changed the server answers with a 304 Not Modified and no page at all. RFC 9110 says an origin server "SHOULD send an ETag" where it can detect changes reliably. In practice about half of our list could be checked that way. Of the 54 pages that answered a live request with a 200, 31 sent a validator, and 27 of those answered our repeat request with a 304. The other half have to be downloaded and compared.

Hash the extracted text rather than the raw response, because raw HTML carries session tokens and build identifiers that change on every request. Hash before you chunk, so an unchanged page costs one comparison and no embedding. When a page has changed, replace all of its chunks in one step, so retrieval never mixes old and new versions of the same page. Delete the chunks of anything that has disappeared.

Then store two dates rather than one: when the page last changed, and when you last confirmed it. Most schemas keep only an ingestion time, which cannot tell a page checked this morning from one last seen in June.

How often should you re-fetch a public page?

Set the interval from what a wrong answer costs and how fast that kind of page moves. The survey gives a starting point, and it is only that: measure your own list before you trust ours.

For a page holding a figure that changes daily, such as a live price, stop indexing it. Fetch the page when the question arrives and use the index only to find it. We covered that step in how AI search fetches web pages.

Does it matter where the re-fetch comes from?

Yes, because a change of origin looks exactly like a change of content. In our two-origin test of the 300 most-visited domains, eleven sites served the French version of their site to a server in Paris and the Irish version to a home connection in Dublin, and three refused the cloud server while answering the household. If your refresh job moves between regions, or retries through a different network after a failure, your content hash changes when the page did not. You pay to re-embed the page, and worse, you may replace an Irish price with a French one.

Pin each source to one market, and record it. Treat a refusal as a failed check rather than an empty page, or your pipeline will delete good chunks because a bot check answered instead of the site. Before adding a source, check what its owner allows: our survey of sites blocking AI crawlers sets out the rules checks that apply, and a re-fetch does not change them.

How does Teneo fit a RAG refresh pipeline?

Teneo supplies the re-fetch step for public sources: the page requested from the market you name, through a connection whose owner agreed to share it, paid for per request. It is the part of the ladder a webhook cannot do for you.

The connections come from Teneo Beacon, the app people install to share unused bandwidth from their own devices. Beacon only shares bandwidth after the operator accepts a consent screen, and they can pause or quit at any time. It never accesses the operator's files or browsing, and the traffic it carries comes from vetted business customers. In October 2026 the Beacon page listed more than 30,000 verified operators in over 150 countries, which is what lets a refresh job stay pinned to one market.

The requests are made by agents, and hundreds of them are deployed on the Teneo Protocol. The data platforms in scope are X, Reddit, TikTok, LinkedIn, Instagram, YouTube, Google Maps, Google Search and Yelp, plus agents such as Amazon's, where the market is an argument of the request: the `product` command takes the ASIN and the storefront domain. For arbitrary pages, Teneo's Live Web Data API is in early access. In our own load test on 15 October it loaded 97.7% of 150,000 pages, including online shops and job boards. That is our test on our set of pages, and you should run yours.

Every request is priced per command and paid in USDC through x402, the open standard that uses the HTTP 402 status code to ask for payment on the request itself. The Agent SDK documents that users see the price before executing a task. That pricing is what makes a refresh schedule easy to budget, because you pay for the pages you actually re-check. At the Amazon agent's published 0.0025 dollars a query, re-checking 1,000 products a day for 30 days costs 75 dollars, and re-checking them weekly for four weeks costs 10 dollars.

Teneo does not publish change detection, a webhook for page changes, or a guaranteed set of validator fields in an agent's response. Hashing, the two dates and the decision about when to re-embed all stay in your pipeline.

Why is demand for fresh public web data rising?

Because more products answer from the live web, and more sites decide who gets to read it. AI search is the clearest case: Google said in May 2026 that AI Mode had passed one billion monthly users, with queries "more than doubling every quarter since launch". RAG products built on public sources face the same arithmetic on a smaller scale, and every new source adds another page that has to be re-checked on a schedule, indefinitely.

Meanwhile a re-check from a data centre gets harder to rely on. Since 15 September 2026, new domains on Cloudflare block bots classified as Training or Agent by default on pages that display ads. AI search is one of four buyer groups in our guide to what bandwidth sharing is used for, and freshness is what turns each of them from a one-off collection into a recurring one. A page you fetched once is a cost. A page you have to keep confirming is a standing demand for connections in the right market, and that demand grows with every source added.

Frequently asked questions

How do you keep RAG data fresh?

For sources you own, use webhooks or change-data capture and re-embed only what changed. For public pages, which offer no webhook, re-fetch on a schedule set by how fast each kind of page changes, ask the server with ETag or Last-Modified first, hash the extracted text before chunking, and store when each page was last checked as well as when it last changed.

Why does my RAG return outdated information?

Usually because the index still holds an old copy of a source that has changed, or holds both old and new versions. Embeddings rank by similarity, not by date, so an old chunk scores as well as it ever did. For public pages the source cannot tell you it changed, so the copy goes stale until something re-fetches it.

How often should you re-index a RAG knowledge base?

Update changed documents incrementally rather than rebuilding on a calendar, and save full rebuilds for changes to your embedding model or chunking. For public web sources, a starting point is daily to weekly for prices and version numbers, weekly for documentation and monthly for official pages. Measure your own sources before relying on any fixed interval.

How do you detect changes on a website you do not own?

Store the ETag or Last-Modified header and send it back on the next request; a 304 response means the page has not changed. In our survey, 27 of 54 live pages answered that way. For the rest, download the page, extract the text, and compare a hash of it with the hash you stored, always from the same market.

How fast do public web pages change?

In our survey of pages compared across Internet Archive captures on 9 November 2026, 33.3% of 48 pages had lost at least 5% of their text after 30 days and 66.0% of 47 after 180 days. 18.8% had lost at least one price, percentage or version number after 30 days. It is a small hand-picked list, so treat it as a floor.

Can Teneo tell me when a page changes?

No. Teneo does not publish change detection or a webhook for page changes. It supplies the re-fetch: a public page requested from the market you name through an opted-in connection, paid per request in USDC through x402. Hashing, change decisions and re-embedding stay in your own pipeline.

What to do next

Before the pilot, take the 20 sources your users ask about most, find the oldest copy of each in your index, and compare it with the live page by hand. The script, the page list and the results behind this post are published at teneo-protocol.ai/data/2026-11-09-page-drift/, so you can run the same comparison on your own list. For the questions to ask any provider, read how to get live web data for AI agents. If you want to supply the connections a refresh job needs, install Teneo Beacon and follow the setup guide for your platform. Builders can browse the agents in the Agent Console.