Editorial illustration for New Framework Shifts LLM Output to Typed JSON for Safer Web Data Collection
New Framework Shifts LLM Output to Typed JSON for Safer...
LLMs can't scrape websites. They can write code that attempts to scrape websites, and that code usually fails. It breaks silently on a layout change or a missing library, hours after you think the job is done. The gap between a simple request and a reliably repeatable data pipeline remains huge.
A new research paper suggests a fix. Don't ask the model for freewheeling Python scripts. Force it to output a typed JSON configuration that slots into a deterministic execution engine.
It's a boring, bureaucratic solution to a chaotic problem. That's the point.
We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection.
The trade-off is clear. You sacrifice some initial, flashy one-shot capability for a system you can actually schedule and forget. Zero execution-stage tokens means once the JSON plan is set, it runs without further model calls, sidestepping both cost and unpredictability.
Lower wall-clock time is a nice perk, but the real win is verifiability. You can inspect the JSON plan before it runs. You can apply rule-based checks.
You have a structured way to feed errors back for correction. This turns a black-box art project into something resembling infrastructure.
It won't handle every weird edge case on the modern web. But it provides a scaffold, a way to make failure explicit and correctable instead of silent and catastrophic. For anyone who has ever spent a Sunday debugging a phantom scraper failure, that scaffold looks like solid ground.
Further Reading
- AI data extraction: how to extract structured data from websites at scale - Parallel AI
- Structured data extraction from unstructured content using LLM schemas - Simon Willison
- SLOT: Structuring the Output of Large Language Models - arXiv
- Data extraction: The many ways to get LLMs to spit JSON content - Glaforge.dev
- Using the LLM Mesh to parse and output JSON objects - Dataiku Developer
Related Reading
Policy & Regulation After ditching AI fitness apps and a Fitbit, I return to Peloton classes
Policy & Regulation Grammarly faces class-action suit over AI ‘Expert Review’ feature
Policy & Regulation Akamai Proposes SAFE Cybersecurity Guidelines for AI Threats
From Archives NYT says Microsoft built supercomputer that trained ChatGPT on its articles
From Archives OpenAI limits GPT‑5.6 rollout after government request, calls it short‑term step
Common Questions Answered
Why do traditional LLM-generated Python scripts fail for web data collection?
Traditional LLM-generated Python scripts for web scraping fail because they break silently when website layouts change or required libraries are missing, often hours after execution begins. The gap between a simple request and a reliably repeatable data pipeline remains enormous, making these scripts unreliable for production use.
How does the typed JSON configuration framework improve upon freewheeling Python scripts?
The new framework forces LLMs to output typed JSON configurations that slot into a deterministic execution engine instead of generating arbitrary Python code. This approach eliminates execution-stage tokens, meaning the JSON plan runs without further model calls once it's set, reducing both cost and unpredictability.
What are the key benefits of using structured JSON plans for web data collection?
Structured JSON plans offer verifiability since you can inspect the plan before execution and apply rule-based checks to validate it. Additionally, you have a structured way to feed errors back for correction, and the deterministic execution engine provides lower wall-clock time and reliable, schedulable data pipelines that can run without manual intervention.
What trade-off does the JSON configuration framework introduce compared to one-shot LLM capabilities?
The framework sacrifices some initial, flashy one-shot capability in exchange for a system that is actually schedulable and reliable for repeated execution. While you lose the flexibility of immediate, freeform model outputs, you gain a verifiable, deterministic pipeline that can be inspected, validated, and scheduled to run without further model intervention.
Further Reading
- AI data extraction: how to extract structured data from websites at scale — Parallel AI
- Structured data extraction from unstructured content using LLM schemas — Simon Willison
- SLOT: Structuring the Output of Large Language Models — arXiv
- Data extraction: The many ways to get LLMs to spit JSON content — Glaforge.dev
- Using the LLM Mesh to parse and output JSON objects — Dataiku Developer
