The Challenge of Large Product Catalogs
As agentic commerce scales, merchants with thousands or even millions of products face a critical challenge:
How do AI agents efficiently process massive product catalogs without overwhelming memory or network resources?
Traditional JSON feeds load the entire catalog into memory at once — which works fine for small stores, but becomes problematic as catalogs grow.
A 50,000-product feed can easily exceed 100MB, causing slow loading times, high memory usage, and potential timeouts for AI agents processing these feeds.
Today, we're solving this problem with NDJSON streaming.
What Is NDJSON Streaming?
NDJSON (Newline-Delimited JSON) is a format where each line of a file is a valid, self-contained JSON object.
Instead of a single large JSON array, an NDJSON feed is a stream of individual product records, one per line.
This approach offers several key advantages:
1. Incremental Processing
AI agents can read and process products line-by-line, rather than loading the entire feed into memory.
This means an agent can start processing products immediately, even while the feed is still being downloaded.
2. Memory Efficiency
With traditional JSON, a 100MB catalog must be fully loaded into memory before processing can begin.
With NDJSON streaming, only one product (a few KB) needs to be in memory at any given time.
3. Parallel Processing
Different parts of an NDJSON feed can be processed in parallel, enabling distributed systems to handle massive catalogs more efficiently.
4. Resumable Downloads
If a download is interrupted, processing can resume from the last successfully parsed line, rather than starting over.
How NDJSON Works in GXO Feeds
When you generate a product feed through GXO, you now receive two format options:
JSON Format (Traditional)
The complete catalog as a single JSON array — ideal for smaller catalogs (< 5,000 products) or when full catalog access is needed immediately.
[
{ "id": "product-1", "title": "Product 1", ... },
{ "id": "product-2", "title": "Product 2", ... },
...
]
NDJSON Format (Streaming)
The same catalog as a stream of JSON objects, one per line — ideal for large catalogs (5,000+ products) or when incremental processing is preferred.
{"id": "product-1", "title": "Product 1", ...}
{"id": "product-2", "title": "Product 2", ...}
...
Both formats are ACP-compliant and contain identical product data — the only difference is how they're structured for consumption.
Benefits for AI Agents
For AI agents processing product feeds, NDJSON streaming delivers tangible improvements:
Faster Time to First Product
Agents can start working with products immediately, rather than waiting for the entire feed to download and parse.
Lower Memory Footprint
Memory usage remains constant regardless of catalog size, making it feasible to process even million-product catalogs on standard hardware.
Better Error Recovery
If a single product line is malformed, the agent can skip it and continue processing the rest of the feed, rather than failing on the entire catalog.
Improved Scalability
Large retailers can now confidently expose their entire catalog to AI agents without worrying about performance degradation.
When to Use Each Format
Choose JSON Format When:
- Your catalog has fewer than 5,000 products
- You need the complete catalog loaded at once for analysis
- Your processing system requires random access to products
- Memory usage is not a constraint
Choose NDJSON Format When:
- Your catalog has 5,000+ products
- You want to minimize memory usage
- You're processing feeds in a streaming or distributed system
- You need fast time-to-first-product processing
Getting Started
NDJSON feeds are automatically generated for all GXO feeds when catalogs exceed 1,000 products.
You can access your NDJSON feed at:
https://feeds.gxo.dev/v1/brands/{brand-slug}/feeds/primary.ndjson
For partitioned feeds (catalogs over 10,000 products), each partition is available as a separate NDJSON file:
https://feeds.gxo.dev/v1/brands/{brand-slug}/feeds/primary-part-001.ndjson
https://feeds.gxo.dev/v1/brands/{brand-slug}/feeds/primary-part-002.ndjson
...
The feed manifest (primary.manifest.json) provides metadata about all available partitions, including file sizes, item counts, and checksums.
Looking Ahead
NDJSON streaming is just the beginning of our commitment to making large-scale agentic commerce accessible and efficient.
Upcoming enhancements include:
- Compressed NDJSON (
.ndjson.gz) for reduced bandwidth - Incremental updates (delta feeds) for changed products only
- Query endpoints for filtered product subsets
- Real-time streaming for live catalog updates
As AI agents become the primary way customers discover products, ensuring these systems can handle catalogs of any size is essential.
NDJSON streaming makes that possible — today.
Ready to Get Started?
If you're running a large catalog and want to enable NDJSON streaming for AI agents, connect your store to GXO today.
Our automated feed generation will automatically create optimized NDJSON feeds for your products.
