A Real-Time Real Estate Data Platform, From Many MLS Feeds to One Search

Real Estate Data Was the Product, and the Problem

A multiple listing service is where real estate agents publish what is for sale, and there are hundreds of them in the United States, each with its own feed, its own field names and its own idea of what a listing looks like. BoxMLS set out to build a search product on top of that mess: one place where an agent and their client could search live listings together, save searches, share favorites and get alerts, under the agent's own brand. The pitch was simple. The engineering underneath it was anything but, because the product was only as good as its data, and the data arrived in a dozen dialects, all day, forever.

We built and ran that platform from 2015 to 2022, and grew the team as the company's ambitions did. The seven years divide into three chapters: the platform, the scale-up, and the launch that tested both.

The Platform

The center of the system was a real-time synchronization engine that consumed listing data from multiple MLS feeds through the RETS protocol, normalized it into a single schema, and served it through an Elasticsearch search index and an API used by every front end. Around it grew everything a multi-tenant product needs.

The Launch That Tested It

In late 2019 BoxMLS won the opportunity every company in its position wants: the California Regional MLS, which describes itself as the largest subscriber-based multiple listing service in the nation, would offer the product to its members as CRMLS Connect. It was the kind of deal that changes a company, and the kind that breaks a platform. The system had to handle an order of magnitude more users and data without disturbing the brands already running on it, on a launch date measured in weeks.

We asked for something unusual: to be turned loose on the technical debt, without task-by-task estimates, against a shared list of scaling risks. The client agreed. In the weeks that followed, Kubernetes and Elasticsearch scaling, infrastructure monitoring and alerting, search index backups, prerendering for search engines and billing safeguards were worked through in order of risk, with stress tests to find the points of failure before members did. The brand and MLS setup steps were made routine, the branded apps were built, and the product launched to the largest MLS audience in the country in spring 2020.

What We Took From It

Two lessons from this work shaped how we run platforms now. The first is that data products live or die on ingestion. Search, apps and branding were the visible work, but the years of unglamorous synchronization engineering were what made a national MLS willing to put its name on the product. The second is about the launch itself: the honest conversation about technical debt, held early and backed by a written list of risks, is what separates a launch that scales from one that makes the news. We have had that conversation with every client since.

The work items from this engagement, from Kubernetes auto-scaling to the Lambda@Edge request handling, are in Our Work.

When You Need This, and When You Do Not

Most companies do not need to build their own data synchronization platform. If a vendor's API gives you the data you need in a shape you can use, take it and spend your engineering on the product. Building ingestion is a commitment measured in years, and it never ends, because the sources keep changing.

It becomes the right choice when the data is the product, when it comes from many sources that will never agree, and when freshness is a feature your customers will pay for. Real estate is the textbook case. So are logistics, healthcare scheduling, event ticketing and any marketplace that aggregates inventory from parties it does not control. In those businesses the ingestion layer is not plumbing. It is the moat, and it deserves to be engineered like one.