AI has already consumed much of the open internet. What is the next logical step for training AI and improving model performance? Several emerging areas are creating sources of net-new data that extend beyond the text, images, and videos already available online:
- Synthetic data generated through models and simulations. Companies such as Physical Intelligence are developing foundation models for robots, while platforms such as NVIDIA Isaac Sim allow developers to generate synthetic training data and test robotic systems across realistic virtual environments.
- Real-world training data captured from human activity in physical environments. Shift is offering free home cleanings while workers record household tasks that can be used to train AI. Elsewhere, workers are being paid to wear head-mounted cameras while folding, sorting, cooking, assembling, and cleaning, creating first-person demonstrations of how humans interact with the physical world.
Companies are already building on top of legacy systems rather than replacing them outright. PillPilot, - a company I came across recently - illustrates this by sitting on top of existing pharmacy management systems and turning previously inaccessible workflow data into an operating layer for autonomous agents. One moat that doesn’t seem as obvious is the network layer beneath modern software, where every digital action produces valuable interaction data that can be used to train more capable agents.
Most websites expose functionality through structured APIs that are behind the network layer of the services. These APIs define specific endpoints that external software can retrieve data or trigger actions using protocols such as HTTP or HTTPS with formats like JSON. Both integrations with third party applications interact with these systems by sending them requests.
Underneath this layer, modern web applications rely on a complex set of network interactions between browser servers and microservers. When a user performs an action in a web interface, the browser sends a series of network requests to the backend servers to process the request and send back a structured response. APIs formalize this. To create your own API for a specific website, you would need to observe and map the website’s network interactions and identify the backend requests tied to key user actions, and then replicate those request flows in a structured way so they can be exposed though a programmable interface.
Data is the hottest commodity in the world and the internet was not built for AI (X402 is one of the first real attempts to fix that, reviving a dormant HTTP status code from 1991 that was always meant for machines but never implemented). It’s communicating with the backend of the internet that I think is the next logical step for augmented data and actionable intelligence. As models (1) fundamentally change (2) workflows become more optimized, (3) the data pipeline transforms into an auditable system, and (4) GPUs are running faster - these are all harbingins for unobtrusive data architecture at the network layer.
Reverse engineering of internal APIs, i.e. analyzing network requests to identify undocumented back endpoints do have hurdles such as anti-crawl provision making edge compute and network layer proxy infrastructure (distributed systems that inspect or modify traffic closer to the user) and open synthetic data generation from web interactions knock on effects of this idea.
Cloudflare has their crawl endpoint where agents can now crawl an entire website with a single API call using Browser Rendering’s new /crawl. Maango recently released a report that they crawled 1M domains and 90% do not have an AI policy in place. They used Tranco top 1M domains (ranked websites by popularity using real internet traffic). The scraper analyzed a myriad of different AI policies and even after scanning 80K to policies, the vast majority had no anti-bot rules. However, stakes are rising for corporations - Anthropic had agreed to a proposed $1.5B settlement with publishers over claims that its AI models were trained on copyrighted material without permission, highlighting the growing legal risk around training data. In December, OpenAI updated its crawler documentation, removing the statement that the ChatGPT-User agent follows robots.txt, suggesting that AI browsing triggered by users may access pages even if a site blocks bots.
The gap between AI agents and what they’re doing and what policies companies have in place is enormous - making it a digital oil field with no formal laws.