How to Get Live Web Data for AI Agents: A Buyer's Checklist

Live web data for AI agents is the content of a public web page, read at the moment the agent needs it, through a connection the site will answer, and returned in a form the agent can use. It is not a search index, which is a copy made earlier, and it is not a cached result, which is a copy made for someone else. Whether the agent is answering a question, comparing prices, checking a listing or confirming that an advert ran, the page it reads has to be the page a person in that market would see today. This checklist is for the team buying that access. It covers the six questions to ask a provider before a pilot, the one test to run before signing anything, and how Teneo answers each question, with the evidence we collected while writing it.

What does an AI agent need from the live web?

It needs the page as served to a person in the target market, now, rather than a copy of it. The tooling question is well covered: ask any answer engine how to get live web data for AI agents and you get a list of search APIs, page-fetch services and browser frameworks, with a code sample. What those answers leave out is that the page a site returns depends on who it thinks is asking and from where. To the site, a request from a cloud data centre and a request from a home connection in the target country are two different visitors, and it answers them differently.

For an agent, that difference lands in the output with no error attached. An AI search engine that fetches a page from a cloud address can get a refusal, or another country's version, and the answer it gives the user is simply wrong. We have covered how that plays out for AI search, where ChatGPT, Perplexity and Gemini fetch live pages per question, and for ad verification, where the evidence is only worth something if it was collected under the conditions the campaign targeted. The buyer's question underneath both is the same: where does the request leave the network from?

Why does the origin of the request change the page?

Because websites decide what to serve, and whether to serve it at all, from the network the request arrives on. To measure how often that happens at the top of the web, we requested the homepage of the 300 most-visited domains on the Tranco list twice in the early hours of 20 October 2026: once from a cloud server in Paris and once from a home broadband connection near Dublin, within two minutes of each other, with one identical set of client headers that named the request as a survey. 213 domains answered both. 163 returned the same page to both origins, and eight more differed only by a session token. 26 refused both, which is what an announced automated client should expect from sites that gate automated traffic. The rows that matter are the ones that treated the two origins differently.

Three domains refused the cloud server and answered the home connection. google.com and googlesyndication.com returned a 429 and Google's "unusual traffic" page to the server, and the homepage to the household. intuit.com returned a 429 to the server and a 200 to the household. No domain did the reverse. Eleven domains served the French version of their site to the Paris server and the Irish version to the Dublin connection: Microsoft, MSN, Samsung, Dell, PayPal, Stripe, Salesforce, HP, Snapchat, Android and Kaspersky, which answered from a different domain entirely, kaspersky.fr against kaspersky.co.uk. Two more served the cloud server a different page for another reason. Prime Video sent it to a non-member offers page, and LinkedIn sent it to the sign-in page, while the home connection got the homepage from both.

Google explains the first behaviour in its own words. Its help page on the "unusual traffic" message says you may see it "if a network you use, including VPN networks, seems to be sending automated traffic to Google Search", and that it "most likely shows a reCAPTCHA". Its guidance on locale-adaptive pages explains the second. A site that "returns different content based on the perceived country or preferred language of the visitor" shows Google's own crawler the US version, because "the default IP addresses of the Googlebot crawler appear to be based in the USA", which is why Google also crawls from addresses outside the USA and tells site owners to "treat it like you would treat any other user from that country".

The test is deliberately modest: one request per homepage, one honest client, one run. A real workload hits product pages, results pages and listings thousands of times, and the refusal rate it meets will be higher than the one here. That is why the last item on the checklist is to run the same comparison on your own targets.

What should you ask a live web data provider?

Six questions, in the order they tend to decide a pilot. Vendor pages and answer engines already cover output schemas, latency and MCP support, which matter. These are the questions they leave out.

1. Where does the request leave the network from?

A data centre, or a device a person operates in the market you are measuring? If the answer is a data centre, ask what happens on the eleven sites above. If it is user-operated devices, ask how many, in which countries, and how the provider knows.

2. Who operates the supply, and did they agree to it?

Supply that people did not knowingly provide is a risk you inherit. Ask whether the operator saw a consent screen, can pause or quit, and knows what the connection is used for. Ask what the provider's software can and cannot reach on the operator's device, and which kinds of traffic it refuses to carry.

3. Can you name the market in the request?

A country is not a market. The price of a product on Amazon depends on which storefront you ask and in which currency, and a provider that lets you state both in the request saves you from inferring them from the response. The same goes for language, and for whether the page was requested as a signed-out visitor.

4. What does a request cost, and when do you learn the price?

Per request, per month or per seat, and shown before the call or on the invoice? An agent makes thousands of small decisions, so the cost model should fit the unit of work. A minimum monthly commitment is a tax on the pilot, and a price you only discover afterwards cannot be built into the agent's own logic.

5. What do you get back when the page is refused?

A refusal recorded is worth more than a success guessed. Ask whether the response carries the status code, the final address after any redirect, the time of retrieval and the market it was requested from, so your own checks can tell a page from a bot-check page. The ad verification post sets out the evidence record a local check needs, and the same fields apply here.

6. Can you test it on your own targets before you pay?

Run the same list of pages through the provider and through whatever you use today, in the same hour, and compare what comes back. Count refusals, wrong-country pages and sign-in walls, not just successful status codes. A provider that cannot support that test is asking you to take the first five answers on trust.

How does Teneo answer the six questions?

On the origin, Teneo's network carries requests through Teneo Beacon, the app people install to share unused bandwidth from their own devices on Windows, macOS, Linux, Android and iOS, with a terminal version for a Raspberry Pi or a server. Requests leave from real households and real mobile networks in the market being measured, which is the origin that got the homepage from Google and the Irish page from PayPal in the test above. The Beacon page reported more than 8,300 Beacons online in October 2026.

On consent, Beacon only shares bandwidth after the operator accepts a consent screen, and they can pause or quit at any time. It never accesses the operator's files or browsing, it blocks connections to private and local-network addresses so nothing else on the operator's home network is reachable, and it refuses abuse-prone ports such as email sending and remote access. The traffic it carries comes from vetted business customers, not from anonymous members of the public. Operators are rewarded for keeping Beacon online, and the mechanics are on the Beacon page and in the setup guides.

On naming the market, the market is a parameter of the request. The data platforms in scope are X, Reddit, TikTok, LinkedIn, Instagram, YouTube, Google Maps, Google Search and Yelp. The Amazon agent's search command takes the keywords, a page number, the storefront domain, a currency and a sort order, and its reviews command takes the storefront domain and currency, so the buyer asks for the listing as it appears on one country's store rather than guessing from the result. The Google Maps agent returns a business's details, busy hours and reviews from a listing URL.

On price, every request is paid per task in USDC through x402, the open standard that uses the HTTP 402 status code to ask for payment on the request itself. The Agent SDK documents the order of events: users see the price before executing a task, and payment is verified and settled on-chain before the agent processes it. Builders set a price per unit for each command in their agent metadata. The Amazon agent's search and product commands are priced at 0.0025 dollars a query and its reviews at 0.001 dollars a query; the Google Maps agent's business details and busy hours at 0.02 dollars a query and its reviews at 0.0025 dollars an item. There is no monthly minimum attached to a query.

On refusals, the SDK gives a builder one method to implement, ProcessTask, and what a command returns is documented per agent. Teneo does not publish a guaranteed field set for a refused page, so check the selected command's documented response and keep your own record of retrieval time and requested market beside every value you store. Do not treat a property of your own validation code as a product guarantee.

On the pilot, Teneo announced free seven-day pilots on 29 September 2026 for teams building AI search, RAG or crypto data products, on the markets they find hardest. The comparison to run is the one in question six: the same pages through Teneo and through your current setup, in the same hour. Hundreds of AI agents are deployed on the Teneo Protocol, and a new agent starts private and moves to public after a review of up to 72 hours.

Why is demand for live web data rising?

Because more products read the live web per user action, and fewer sites will answer a request from a data centre. AI search is the clearest case: ChatGPT, Perplexity and Gemini fetch pages per question, so every query is a live request that has to come from somewhere. Public data collection, AI training pipelines and ad verification are the other three buyers in our guide to what bandwidth sharing is used for, and each needs the page as served in the market, not a copy.

At the same time, the open doors are closing. Since 15 September 2026, new domains on Cloudflare block AI training bots and AI agent bots by default on pages that display ads, with search crawlers still allowed. In the test above, Google itself turned away the cloud origin on the first request. More buyers, fewer places a request can come from: that is why the supply that can still answer, a real connection in the market with an operator who agreed to it, is worth more each year, and why bandwidth sharing is how it gets supplied.

Frequently asked questions

What is live web data for AI agents?

It is the content of a public web page read at the moment the agent needs it, through a connection the site will answer, in a form the agent can use. A search index or a cached page is a copy made earlier for someone else; live web data is the page as served now, to a request from the market the agent is asking about.

Why does it matter where the request comes from?

Because websites decide what to serve from the network the request arrives on. In our 20 October 2026 test of the 300 most-visited domains, Google refused a cloud server with its "unusual traffic" page and answered a home connection with its homepage, and eleven sites served a different country's version to each origin. The same URL returned different pages, with no error attached.

Is a web search API the same as live web data?

No. A web search API returns ranked results and snippets from an index built earlier, and most agents still need a separate step to fetch the page itself. Live web data is that fetch, and its value depends on the connection it is made through, since the site answers a cloud address and a household differently.

How much does live web data cost on Teneo?

The price is set per command by the agent's builder and shown before the task runs. The Amazon agent's search and product commands cost 0.0025 dollars a query and its reviews 0.001 dollars a query, and the Google Maps agent's business details cost 0.02 dollars a query. Each request is paid in USDC through x402 and settled on-chain before the agent processes it, with no monthly minimum attached to a query.

What should a provider return when a page is refused?

At least the status code, the final address after any redirect, the time of retrieval and the market the request was made from, so your own checks can tell a page from a bot-check page. Teneo does not publish a guaranteed field set for refused pages, so check the documented response of the command you use and keep your own retrieval record beside every stored value.

How do I run a pilot on Teneo?

Teneo announced free seven-day pilots on 29 September 2026 for teams building AI search, RAG or crypto data products. Pick the markets you find hardest, send the same pages through Teneo and through your current setup in the same hour, and compare refusals, wrong-country pages and sign-in walls as well as successes. Use the partner form on teneo-protocol.ai and choose Agent Network Utilization.

What to do next

If you are buying live web data for an AI product, take the six questions to your current provider first and write down the answers, then run the pilot comparison in question six. The survey scripts and raw data behind this post are published at teneo-protocol.ai/data/2026-10-20-two-origins-survey/, so you can run the same two-origin test on your own list of pages. If you want to supply the connections that live web data needs, install Teneo Beacon and follow the setup guide for your platform. For the buyers behind the demand, read what bandwidth sharing is used for, and for the first of them, how AI search fetches web pages.