How a Platform Team Ended Surprise Reports: A DataOps Story

How a Platform Team Ended Surprise Reports: A DataOps Story

The story below is an invented composite, offered to show how DataOps ideas play out in a team’s working life. No real company, product or person is described, and the people are identified by role only. The ideas, however, come straight from the reference entry in the Atlas, and each turning point maps to something that entry recommends.

The quarter of surprises

A mid-sized software company had a small platform team of five. They looked after the nightly pipeline that fed the executive dashboard: product usage, support volumes and release frequency. For a year it had run quietly. Then, in a single quarter, three things happened. The usage figure dropped sharply for a week and the product director asked why adoption had collapsed. It had not; a feed had lost a region. A support chart showed an impossible spike that turned out to be duplicated rows after a rerun. And a release metric changed definition when an upstream team renamed a status.

Each time, the platform lead was told by a manager, not by the system. In the language of the Atlas, the consumer-detected share of incidents was one hundred percent, and the team knew it was a poor score.

The retrospective

The retrospective was blunt. The scheduler had reported success every night. The lead, who had come from a deployment background, said the team had excellent controls over code changes and almost none over the data itself. Code was reviewed and tested. Data arrived each night with no one asking whether it was complete, fresh or sensible. The Atlas puts this distinction as a change loop versus a run loop: software moves only at deployment, whereas data shifts with each execution.

The team agreed on an experiment, limited to the executive dashboard, which they labelled their only tier-one asset. Everything else would stay as it was.

Month one: see before fixing

The first month was unglamorous. They moved the pipeline code into version control, wrote down the owners of every source, and added three checks: freshness, row count and uniqueness of the main key. They also started an incident log with four columns, recording when something broke, what caused it, who spotted it and how many hours passed before anyone did. Following the Atlas’s suggestion, they scoped the expensive uniqueness check to the newest partition and ran the full-history version weekly.

The row-count check fired false alarms every Monday. A junior engineer noticed the pattern: weekends were quiet, weekdays busy, and the single band did not fit both. They built separate expected ranges, and the noise stopped.

Month two: a gate and a handshake

The duplicate-rows incident convinced the team to add a hold point. New data was now written to a staging area, checked, and swapped into the dashboard’s table only when the blocking checks passed. A failed night left yesterday’s numbers in place with a banner saying so. They accepted the cost the Atlas mentions: some extra latency and duplicated storage during the swap.

Then came the renamed status. The team approached the producing team and drafted a short written agreement stating what each status meant, which fields could never be empty, when the feed would be ready and who to call. Structural checks would never have caught the earlier change, because the column was still text. The agreement was kept in Git beside the code, so any future change had to appear in a pull request that both sides could see.

Month three: promises and budgets

By the third month, the team proposed a timeliness target: dashboard data ready at seven in the morning on nineteen of every twenty working days. Across a 63-day quarter, that left room for roughly three late mornings. When the second of those was spent, the director agreed that risky changes would wait until reliability recovered. The conversation changed. Instead of arguing about blame, people spoke about the remaining budget.

They also reviewed the log. The numbers were sobering and useful: most of the lost time on past incidents had been spent before anyone noticed, not repairing. That pointed at monitoring, which is where the Atlas says to invest.

What changed, and what did not

After two quarters the consumer-detected share had dropped well below half, and the executive dashboard had its first month with no surprises. Yet the team was careful not to overclaim. Exploratory datasets remained untouched. Some checks were turned off after proving noisy, since a pile of ignored warnings hides the few that matter. And the lead insisted that every threshold learned by a tool be reviewed by a person, because automatic baselines can absorb a slow decline without comment.

Lessons worth borrowing

Several lessons transfer. Pick one critical asset. Make failure visible instead of silent. Track who found each problem. Write promises down and attach owners. And treat the data as a product that needs operating, not a by-product of code. The same critical-thinking habits appear in a guide on building the problem-solving mindset DevOps work demands, which complements this story, and in the broader skills roadmap for IT transformation for readers planning their own path.

For the reference behind each turning point, see the Atlas page behind each of the practices in this story. Remember that the Atlas treats its own case studies as self-reported, and this story is a teaching device rather than evidence. Use it to prompt your own retrospective, then measure what happens in your own pipelines.