Your Program Data Looks Bad. Now What?
You spent two years collecting outcome data. You invested in a case management platform, trained your staff, aligned your indicators with your theory of change. You did everything the sector tells you to do. And now the numbers are in.
They're not good.
The employment readiness program you've championed for years? Participants aren't finding stable work at rates meaningfully different from people who never enrolled. The youth mentorship initiative your board loves to talk about at fundraising events? Retention is dropping, and the few participants who complete the program aren't showing the well-being improvements you expected.
This is the moment that separates organizations that use data from organizations that collect it. And most organizations, if they're honest, handle this moment badly.
The Reflex Nobody Talks About
Here's what typically happens when outcome data delivers uncomfortable news: the first instinct isn't to examine the program. It's to examine the measurement.
"Are we tracking the right indicators?" "Was the sample size large enough?" "Maybe we need to adjust the methodology." These are all legitimate questions in the right context. But when they only surface after results disappoint, they're not quality assurance. They're avoidance.
This pattern is so common it barely registers as a problem. A program manager sees a dip in outcomes and immediately flags data collection issues. A director reviews a quarterly report, notices troubling numbers, and asks the evaluation lead to "look into what might be off with the data." The conversation shifts from "what is this telling us?" to "can we explain this away?"
It's not malicious. People who work in social services care deeply about the communities they serve. The programs they've built often represent years of relationship-building, grant applications, staffing decisions, and personal conviction. When data challenges the value of that work, the emotional response is understandable. But understandable isn't the same as useful.
Why Questioning the Measurement Feels Safer Than Questioning the Model
The instinct to scrutinize the data before the program makes a certain kind of sense. Outcome measurement in social services is genuinely difficult. Client populations are complex. External factors (housing markets, employment conditions, policy changes) can influence results in ways a single organization can't control. Measurement tools are often imperfect, especially when organizations are early in their data journey.
All of that is true. And all of it can become a convenient shield.
The distinction matters: questioning your measurement approach because you have specific, evidence-based concerns about its validity is good practice. Questioning your measurement approach because you don't like what it found is something else entirely. One is methodological rigor. The other is motivated reasoning dressed up in evaluation language.
Organizations that can tell the difference between these two responses are far better positioned to learn from their data. The ones that can't tend to cycle through the same pattern: collect data, encounter difficult findings, dispute the method, commission a new approach, and start over without ever acting on what the original data showed.
The Political Pressure to Protect Established Programs
Measurement validity isn't the only force that keeps organizations from confronting underperformance. There's also the political dimension, and it's rarely discussed openly.
Programs don't exist in a vacuum. They exist inside a web of relationships, reputations, and resource commitments. A program that's been running for several years has likely been cited in grant applications, featured in annual reports, and highlighted in presentations to funders and community partners. Board members may have personal connections to the program's origin story. Staff members may have built their careers around delivering it.
When outcome data suggests that program isn't producing the results everyone assumed, the finding doesn't just challenge the intervention. It challenges the judgment of the people who funded it, designed it, and advocated for it. That's a much harder conversation to have.
In practice, this political pressure often shows up in subtle ways. A leadership team might acknowledge the data privately but frame the public narrative differently, emphasizing the program's "process outcomes" or "qualitative value" while quietly downplaying the quantitative findings. A funder might receive a report that buries the underperforming metrics in an appendix while the executive summary focuses on the indicators that look strong. These aren't dramatic acts of dishonesty. They're small, incremental adjustments to the story that, over time, widen the gap between what the data shows and what the organization says.
The cost of this gap isn't abstract. Every quarter an underperforming program continues unchanged, resources that could support something more effective are locked in place. Clients who could benefit from a redesigned approach continue receiving one that the organization's own data suggests isn't working. And the credibility that comes from honest, evidence-informed decision-making erodes a little more.
What "Data-Driven" Actually Requires
Most organizations in the social services sector would describe themselves as committed to data-driven decision-making. It's become standard language in strategic plans, grant proposals, and board presentations. But there's a meaningful difference between an organization that uses data to confirm decisions it's already made and one that allows data to change those decisions.
The first version of "data-driven" is comfortable. You set targets, collect the numbers, and report on progress. When things look good, the data validates your approach. When things don't look good, there's usually enough ambiguity in the measurement to find an alternative explanation.
The second version is harder. It means accepting that your theory of change might be wrong. It means entertaining the possibility that a program your team loves and your funders celebrate isn't producing the outcomes it was designed to achieve. It means making decisions that are politically inconvenient because the evidence points in a direction nobody wanted to go.
This second version is what funders are increasingly looking for. The shift toward outcome-based funding in Canada, including the growing influence of social finance models, is built on the assumption that organizations will use data not just to demonstrate success but to identify and respond to failure. Funders who are investing in outcomes, not activities, need partners who can be honest about what the data reveals, even when the findings are uncomfortable.
Organizations that can demonstrate this kind of intellectual honesty build a different kind of credibility. Instead of presenting every report as a success story, they show funders that they take measurement seriously enough to act on it, including when acting means changing course.
Five Patterns That Signal an Organization Is Avoiding Its Own Data
If you're in a leadership role, it can be difficult to see these dynamics clearly from inside the organization. Here are five patterns worth watching for.
The goalposts move after the results come in. You defined success metrics at the outset, but once the data arrives, the conversation shifts to whether those were really the right metrics. If this only happens when results are poor, that's a signal.
Qualitative evidence is invoked selectively. Client stories and staff observations are valuable complements to quantitative data. But if qualitative evidence only gets cited to offset disappointing numbers (and never to interrogate positive ones), it's being used as insulation rather than insight. As we've written about in outcome-focused reporting, mixing quantitative and qualitative evidence is a best practice, but only when both are applied with equal honesty.
Reports emphasize effort over effect. When an internal or external report spends most of its space on activities completed, partnerships formed, and resources mobilized, but treats outcome data as a brief section near the end, the organization is telling you what it's comfortable being evaluated on. This is the outputs-versus-outcomes trap that the sector has been trying to move beyond for years.
The program is redesigned without reference to the data. Sometimes organizations respond to underperformance by making changes, but the changes aren't informed by what the data actually showed. A program might get a new name, a revised curriculum, or a different delivery model, all without a clear analysis of why the original approach wasn't producing outcomes. This creates the appearance of responsiveness while avoiding the harder diagnostic work.
Difficult findings never reach the board. If the information that makes it to governance tables is consistently more positive than what program staff see in the raw data, there's a filtering problem. Board members can't provide meaningful strategic oversight if they're only seeing the highlight reel.
Building a Culture Where Hard Findings Are Useful, Not Threatening
Acknowledging these patterns is a start. But the more important question is what to do about them. How does an organization build the internal conditions where disappointing data is treated as valuable information rather than a reputational risk?
Separate the data conversation from the funding conversation. One of the biggest obstacles to honest internal dialogue about outcomes is the fear that acknowledging underperformance will jeopardize funding. This fear isn't irrational. But it creates a perverse incentive where organizations that are most honest about their results are penalized, while those that present a rosier picture are rewarded. Leaders can mitigate this by creating internal spaces where data is reviewed for learning purposes, distinct from the reporting process. A monthly program review meeting where staff discuss what the data is showing, including the difficult parts, without worrying about how it will read in a funder report, changes the function of data from a compliance tool to a management tool.
Normalize the expectation that not everything will work. Social programs operate in complex environments with populations facing intersecting challenges. The idea that every intervention will produce the intended outcomes every time isn't realistic, and pretending otherwise sets organizations up for exactly the kind of defensive response described above. Leaders who talk openly about the role of failure in program improvement, who share examples of past programs that were modified or discontinued based on evidence, give their teams permission to engage honestly with data. This doesn't mean lowering standards. It means creating an environment where learning from a setback is valued as much as reporting a success.
Invest in the analytical capacity to understand why, not just what. A number that says a program is underperforming is only the starting point. The more important question is why. Is the program model sound but poorly implemented? Is it well-implemented but targeting the wrong population? Are external conditions undermining outcomes in ways the program can't address alone? Answering these questions requires more than a spreadsheet. It requires staff with the skills and time to dig into the data, cross-reference with qualitative evidence, and develop hypotheses that can guide program adjustments. Organizations that invest in this capacity, including through modern data infrastructure, turn disappointing results into actionable intelligence. Organizations that don't are left guessing, which usually means defaulting to the status quo.
Be transparent with funders, earlier. Many organizations wait until a final report to surface findings, by which point the narrative is already set and the room for honest conversation is limited. A different approach is to bring funders into the learning process earlier, sharing interim data, flagging emerging concerns, and framing adjustments as evidence of good management rather than program failure. This requires funders who are willing to engage that way, and many are more open to it than organizations assume. The Tamarack Institute's open letter to Canadian funders, signed by numerous sector organizations including the Common Approach to Impact Measurement, called explicitly for reporting practices that establish reciprocal data relationships between funders and grantees, rather than top-down imposition of metrics.
Make program adjustment a regular, expected activity. If the only time a program gets seriously re-examined is during a crisis, then modifying a program carries the weight of an admission of failure. But if program review and adjustment are built into the normal operating rhythm (quarterly data reviews, annual program design sessions, regular check-ins on theory of change alignment), then changing course based on evidence becomes routine rather than exceptional. Organizations that have built this kind of shared measurement capacity describe a qualitative shift in their relationship with funders and community partners, moving from defending activity volumes to bringing evidence of movement.
The Credibility That Comes from Honesty
There's an irony in how organizations handle difficult data. The impulse to protect a program's reputation by minimizing unfavorable findings often achieves the opposite of what's intended. Funders, board members, and community partners can usually sense when they're getting the curated version of a story. Over time, that erodes trust far more than a candid conversation about mixed results ever would.
The organizations that build the deepest credibility in this sector are the ones willing to say: "Here's what we expected to happen. Here's what actually happened. Here's what we think explains the gap. And here's what we're doing about it." That narrative, grounded in evidence and oriented toward improvement, is more compelling than a report full of green indicators that nobody quite believes.
This isn't about self-flagellation or treating every program as suspect. It's about taking your own data seriously enough to let it inform your decisions, even when those decisions are hard. It's about recognizing that the purpose of outcome measurement isn't to produce good news. It's to produce useful information.
The sector is moving toward a model where data is expected to drive decisions, not just document activities. The organizations that are best positioned for that shift aren't the ones with the most impressive dashboards or the most polished reports. They're the ones that have built the internal culture, the analytical capacity, and the leadership courage to act on what their data actually says.
Even when it says something uncomfortable.
