What started as a way to solve our own frustration escalated a little. We wanted a clean daily view of where a product was being sold and at what price. That sounds like a solved problem — prices are public, marketplaces have APIs, comparison sites list everything. How hard could it be to pull it all together?

Harder than I assumed, it turns out. The thing that has surprised me most while building fieldmonitor is how fragmented multichannel market data actually is once you start monitoring it at scale. The naive mental model going in is that a price is a price: one product, one number, easy to compare. In reality the same product shows up wearing a different outfit on every channel, and almost nothing lines up cleanly on the first pass.

A price is never just a price

Fetching a number from a page is the easy part. The trouble is that the number rarely means what you'd assume, and the thing it's attached to is described differently everywhere you look. Pull the same product from a handful of channels and you get a small pile of inconsistencies that all have to be reconciled before any of the data is worth trusting:

  • Inconsistent product titles — one channel lists "Model X Pro 2024", another "X-Pro (2nd gen)", another just the bare SKU buried in the description
  • Bundles versus single units — a listing that looks cheaper is actually a two-pack, or ships with an accessory the standalone product doesn't
  • Currency and VAT differences — some prices include tax, some don't, and cross-border listings quote in whatever currency the seller's storefront defaults to
  • Intermittent stock and availability — a price only counts if you can actually buy at it, and "in stock", "backorder" and "temporarily unavailable" are reported inconsistently, if at all
  • Seller aliases — the same operator trades under three different shop names across three marketplaces, so a single reseller looks like three
  • Region-specific listings — the product exists on a marketplace's German, Dutch and French storefronts as three separate entries with three separate prices

None of these are exotic edge cases. They're the everyday texture of the data. Any one of them is manageable in isolation; the difficulty is that they stack, and they stack differently on every channel.

Identity is the hard problem

Once you sit with the data for a while, it becomes clear that the genuinely difficult engineering isn't fetching prices at all. It's matching — deciding, confidently and repeatedly, that this listing over here and that listing over there are the same physical product, and that this one is a different variant, and that one is a bundle you should treat separately.

Identity is messy. There's no universal, reliable key you can join on across a marketplace, a comparison site and an independent webshop. EAN codes are missing or wrong more often than you'd hope. Titles are marketing copy, not identifiers. So the work becomes a chain of normalisation: cleaning titles, resolving units and bundles, mapping currencies and tax treatment onto a common basis, and collapsing seller aliases back to the real operator behind them. Get the matching wrong and everything downstream is quietly poisoned — a price alert that fires on the wrong variant is worse than no alert at all.

The hard problem was never fetching a price. It's knowing, with confidence, that you're comparing the same product.

Built from real edge cases, not a spec

A lot of the platform didn't come from a design document. It grew out of situations we kept running into ourselves while using it day to day. We were our own first users, and every time the data told us something that turned out to be misleading — a "new competitor" that was really an existing seller under a fresh alias, a "price drop" that was actually a currency artefact — we'd trace it back and harden that part of the pipeline. Version by version, the platform learned to handle the mess because the mess kept showing up in our own screens first.

That dogfooding shaped the product far more than any upfront plan could have. You can't anticipate this kind of fragmentation from a whiteboard; you only see it once real listings are flowing through the system and something looks off.

The reward is that once the data is normalised, the day-to-day insight it produces is genuinely worth the effort. A clean, matched view means the signals you actually care about — a new seller appearing, a real price movement, a promotion kicking off across a channel — surface clearly instead of drowning in noise. There's still plenty we want to add and improve, but seeing that visibility land every day has been the most rewarding part so far.

If there's one thing I'd tell anyone tempted to build this themselves: budget for the matching, not the scraping. The scraping is a weekend. The matching is the product.