The ticket arrives on a Tuesday: the sales dashboard shows last Friday’s numbers and the business wants to know why. The pipeline is green. The on-call engineer sees nothing wrong, the managed-service dashboard shows no alerts, and yet an angry stakeholder is right. Support teams who look after data platforms meet this pattern often enough that it deserves a diagnosis guide. Below are five symptoms, the likely cause behind each, and the remedy that DataOps practice recommends, with the Atlas as our guide.
Symptom 1: Everything is green but the report is stale
Likely cause. The scheduler checks that tasks ran, not that fresh rows landed. A source that stops sending data still lets every job exit cleanly, because processing an empty file is a success.
Response. Add a freshness check to the table that readers use, comparing the newest record with the time the business expects it. Make it blocking for the tables that matter. The Atlas cites the SRE Workbook on this very trap: a chain in which every stage declares itself healthy while records disappear between hops. The check belongs on the output, not on the job.
Symptom 2: Numbers are right most days and oddly low on Mondays
Likely cause. Weekly seasonality interacting with a statistical alert. If your monitor computes a mean and standard deviation over all days and fires beyond three sigma, the limits blend busy weekdays with quiet weekends into one wide band. A partial load on an ordinary weekday can then sit comfortably inside it. In the Atlas lab, the all-days limits ran from 695 to 1,219 and missed a load roughly 19% under a normal weekday, while a baseline from weekdays only, a narrower band of 913 to 1,106, flagged the same load.
Response. Split the baseline by weekday type, or compare with the same weekday of earlier weeks. Then review what your tooling learns automatically: the entry cautions that thresholds a tool infers by itself can silently follow a gradual fall downwards, so give your most critical tables a hard minimum that never moves.
Symptom 3: The test failed, yet people still saw the bad data
Likely cause. A build that stops downstream steps after a failed test does not undo the table it just rebuilt. In the transformation tool the Atlas discusses, the tested model already sits in its target schema, so any dashboard reading it directly shows the faulty rows.
Response. Build into a separate audit schema, run the checks there and publish in a second step only on success. This is the write-audit-publish idea. Mention to the client that it adds latency and may double writes, which is why the entry keeps it for critical assets. For the philosophy behind such release discipline, a primer on DevOps principles for dependable software delivery covers the software side of the same thinking.
Symptom 4: Totals double after a retry
Likely cause. A load that appends rows. When the job is retried at night after a transient error, the same day arrives twice.
Response. Replace partitions and use an upsert where a table is updated in place; the Atlas notes that the Airflow guidance favours upsert over insert for this reason. Also decide how late data is handled. Reconcile each partition against the source count, and re-read a short lookback window so that records arriving a day or two late are picked up. One incremental-model example in the entry re-reads three days.
Symptom 5: Nobody knows who should fix it
Likely cause. A pipeline without an owner. The consuming team notices, the platform team routes the ticket, and the producing team never hears about it. The Atlas ties this to the idea of a contract: the producer states what the dataset will contain and who answers for it, at the boundary where silent changes hurt.
Response. Start with a one-page register of critical datasets, each with an owner, a freshness expectation and an escalation path. A managed-service desk can keep that register on behalf of a client and review it monthly. It is far less glamorous than a new tool and far more effective.
A short triage routine for the service desk
When a data ticket lands, resist the urge to open the pipeline code first. Work through the same four questions each time. When did the data last change, and is that the expected time? Did the row count match the source? Did anything about the structure of the input change, such as a column name or a status value? And who owns the upstream feed? Write the answers on the ticket, because they become the raw material for the incident log later. Teams that follow the routine usually close the easy cases in minutes and find the hard ones sooner, since the facts are already gathered when the specialist joins.
Measuring whether the diagnosis worked
Track one figure after you fix anything: how many incidents were reported first by users rather than found by monitors. The Atlas example describes four incidents with a total downtime of 43.75 hours, of which 32.75 hours was detection time, and a consumer-detected share of half. The lesson for a support team is that detection is the part a service contract can improve most. Keep an incident log with start, detection and fix times, and review it with the client each quarter. Our earlier piece on the case for ops engineers to embrace DevOps, from IaC through to SRE offers further perspective on that mindset.
If you would like the full vocabulary behind these fixes, including tiers, contracts and a staged rollout, the reference is the Ops Atlas entry on DataOps. The page admits that the field is young; weigh every suggestion in it against what your own incident history shows before you adopt it.