Bryn Sadownik's blog

Happy New Year! Stay tuned for a new website in 2025

Bryn Sadownik — Thu, 19 Dec 2024 03:34:37 +0000

You may notice that the Demonstrating Value website has a vintage look on the internet! Since we first launched the site in 2009 half a million people have found their way to our site. We have worked with many amazing community-based organizations over the years, and have shared the framework at 109 events with nearly 3,000 attendees.

You may also notice that the current website is a 'static'. Stay tuned for a new website next year that will simplify our resources, while holding true to the vision that the data and impact measurement at its core must be driven by the needs and capacity of community-based organizations.

blog type:

News

Tools

Measurement in the time of COVID-19

Bryn Sadownik — Fri, 05 Jun 2020 23:02:30 +0000

In the space of weeks, countless community-based organizations have had to shift what they are doing while being faced with reduced staff hours, fewer volunteers, disruptions to revenue, social distancing restrictions and other constraints. Yet the demand and need for many services that non-profits and social enterprises deliver are high. New needs are emerging, and many programs are being introduced. Across the community – in health care delivery, social services, education, to name a few – we are witnessing a great deal of creativity and experimentation in (re)designing programs.

Impact and performance measurement practices that rely on clearly articulated goals, a plausible theory of change, and attention to performance metrics attached to goals, are not particularly useful in this complex and rapidly changing environment. What is needed are processes and information gathering that can support organizations to try out different actions that guide the direction of program development and redesign. Fortunately, there are management and evaluation practices that can help guide community-based organization in these unprecedented times. Here are a few suggestions:

1. Focus on creating an environment from which good things can emerge, rather than trying to impose a solution to bring about predetermined results. When it was clear that schools needed to close, the Ministry of Education where I live left considerable latitude to individual schools and educators across the province to use a range tools and approaches to delivering on-line instruction.

2. Create a space to bring people together (even if this is on-line!) for structured reflection about what is being tried, and what people are finding so far about what is working and what isn’t. Engage not only those who deliver programs and services but the beneficiaries as well as other stakeholders. It is especially important to capture a diversity of experience. We may be facing the same storm, but we are all in very different boats!

3. Be alert to the evolving context by staying on top of what is going on and reaching out to others. A shift in the operating environment can open up new opportunities and partnerships.

4. Move ahead by ‘proving, sensing and responding.’ This can be like walking into a forest and moving ahead so that a path forward reveals itself (and changing directions if it doesn’t). In this context, you have to be ‘awake’ to your surrounding. Monitoring and measurement is still important, but it has to be flexible. You can’t miss seeing the forest for trees.

5. Accept that we are moving ahead in conditions of uncertainty. The recent re-opening of school this month in British Columbia frustrated many parents and educators. Teachers and parents alike had to accept considerable uncertainty about what returning to school would look like. But without actually moving ahead we will not learn about the direction we need to move in, given that it is increasingly clear that we have to find ways to live with the novel corona virus while minimizing its risk for months (or years) until a vaccine or effective treatment is found.

6. Consider using Developmental Evaluation to support program development and redesign. This form of evaluation helps deliver feedback to support the development of new programming so that it can more accurately address the roots of a community’s complex needs.

At the same time that organizations on the ground have had to adapt and evolve new ways of working, it is critical that governments and funders are able to change performance measurement expectations and support learning by providing resources for evaluation and creating a culture that supports experimentation and risk-taking.

Great resources for learning more about some of the ideas brought out in today’s blog include:

David J. Snowden and Mary E. Boone, A Leader’s Framework for Decision Making. Harvard Business Review, December 2007.
Better Evaluation. Developmental Evaluation
Hallie Preskill, Tanya Beer, Evaluating Social Innovation, FSG.
Mark Cabaj,Five Good Ideas: Evaluation in disruptive times (Video of Talk) Maytree Foundation

blog type:

Issues & Ideas

Tools

tags:

developmental evaluation

emergence

impact measurement

Planning your Intended Change

Bryn Sadownik — Wed, 27 May 2020 20:35:58 +0000

By planning your intended change, you’re breaking down the big, awesome visionary change that is very challenging to measure, into more immediate and visible steps. In this way, impact measurement is more possible and useful. It’s also about breaking out what you can influence and then making the link between that and the bolder, systems-level change that you seek. Mapping out your intended change can really help you focus on what you should and could influence. It can also demonstrate your value and engage others in your mission.

This blog explores how to plan your intended change, one of the five Common Foundations to Impact Measurement. This is the first of a five part blog series that exmines the core elements of impact measurement advanced by Carleton's Centre for Community Innovation's Common Approach initiative. The topic this week is: Planning your Intended Change.

Good planning draws on research, experience and intuition. It also makes use of planning tools. A Theory of Change (and related mapping tools such as a Logic Model and Impact Map) helps people make connections between a desired change and the work on the ground to get there. It acts like a guidepost where you can look for early signs that change is happening. It also helps you interpret what you see and is useful in surfacing assumptions that you are making.

At a very basic level, an outcome is a description of what changes as a result of doing something. This is often described in terms of short-term, intermediate and long-term change. Your initiative has more control over short-term outcomes. Intermediate and long-term outcomes also result from other factors within the system that you seek to change.

Here are 3 really simplified examples:

This exercise becomes more useful if you not only describe what you would like to see and are doing, but how you see it happening and the assumptions that go along with that. This is the key difference between a Theory of Change and Logic Models/ Impact Maps. Great resources for planning your intended change include: University of Wisconsin Extension (Logic Models); Betterevaluation.org (Describe the Theory of Change); Social Impact Navigator (The Logic Model and Its Components); Innovation Network (Logic Model Workbook)

Here are a few tips for planning your intended change:

Do not articulate specific indicators and targets when you describe an outcome. It is important to start by having a picture of what you would like to see more broadly. Outcomes provide a clear vision of a desired change and help you to make the connections between what you do and that vision.
In articulating an outcome try to include the common parts of a sentence: the subject of the verb and the object. Outcomes are written as change statements (e.g. increase, decrease, or stay the same). Use an ‘active’ verb.
You may need to actively integrate outcomes that a (funding) partner has articulated into your own logic model or theory of change. If this is the case, consider the partner’s framework as one of the stakeholders that informs your theory of change.
If you are presented with an impact map / logic model template, do not get caught up in trying to figure out what goes into specific boxes. Collapse and make your own boxes if you need to.
Keep in mind that outcomes are beyond the immediate control of your program (project or initiative) but can be influenced by a program.
Planning should include all important elements and relationships (even if they aren’t in your control) and should be informed by different perspectives and voices.

Demonstrating Value has a number of tools, resources and archived blogs that can help you think through and articulate intended change. Here are a few:

Impact Mapping Worksheet. This resource includes a number of different approaches to planning intended change such as developing a logic model, theory of change and outcome mapping. Outcome mapping differs a little the other two in that it:

Integrates your connection and collaboration with other actors to create systems change.

Describes outcomes in terms of what you “expect to see,” “like to see” and would “love to see”.

Identifies the specific changes to behavior, actions and relationships that would suggest that change is happening.

Sustainable Livelihoods is a holistic framework that relates overall poverty reduction to how individuals build on their strengths.

Community Capitals Framework is similar framework as Sustainable Livelihoods but at a community level.

When it comes to planning the intended change of your program, initiative or social enterprise, proceed with what appeals and makes sense to you and the area you work in. Remember that what you create is only a model of the real world, and as you get a clearer picture of the outcomes and how your work relates to them, use that information to create a better model. If your initiative/business is emerging, rather than spend much time on developing a detailed plan, I recommend looking to innovative evaluation practices, such as development evaluation, that are well positioned to support nimble decision-making and learning so that you can quickly gain and interpret new evidence. For instance, see Developmental Evaluation at BetterEvaluation.org. If you are working towards colloborative change in a complex societal problem, ‘systems mapping’ and associated tools may be more useful, see FSG (Systems Thinking Toolkit).

blog type:

Issues & Ideas

Tools

tags:

logic models

impact mapping

It is time to revise the “Gold Standard” of Impact Measurement and Evaluation Design

Bryn Sadownik — Tue, 26 May 2020 17:57:58 +0000

A goal of impact measurement is to be able to clearly identify and measure the net, positive effects that result directly from the activities of an initiative, program or social enterprise. This could make it clearer for impact investors about what they invest in, what strategies and models are most effective, and how to replicate and scale what matters.

Clearly establishing net impact and the influence of a program’s activities on an outcome has been a huge challenge in this field. A ‘gold standard’ of research and evaluation, the randomized control trail, is often held up as the standard to which we should aspire. (For example, see NESTA’s Standards of Evidence.) In a nutshell, two groups of people are chosen at random, and only one participates in new program, while the other does not. (A variation of this is where multiple groups are involved). Participants in the groups are monitored for the same outcomes and if the ‘experimental’ group has a statistically significant difference in a measured outcome, we could infer that an intervention led to a specific impact. The ‘experiment’ should also be repeated, and the same result achieved each time. Statistical significance means that we have established that the difference in the data is not due to chance.

Unfortunately, outside of a science lab and the setting of clinical drug trails, it is quite challenging to structure experimental evaluation designs because of cost and practicality. Even the ability to conduct quasi-experimental evaluation designs (that use no control group) can be quite challenging. Is there any practical, statistical alternative to understanding the contribution that specific activities make in influencing an outcome?

Let’s unwind the history of statistics for a moment. Statistics evolved along two lines: Frequentist and Bayesian traditions. Both encompass rich, philosophical traditions, but with differences in how uncertainty and probability are viewed. Frequentist inference, the statistical practice embodied by the experimental ‘gold standard’ evaluation design, involves deducing whether an effect (relationship) that is seen in the data could instead be attributed to chance using a bunch of 'tests'. The other way that statistics evolved is based on the work of Thomas Bayes, an English, Nonconformist theologian and mathematician in the 18th century, who died before he could become famous. (Encyclopedia Brittania). In Bayesian inference, uncertainty is a reflection of imperfect knowledge that can be described in degree-of-belief probability statements, and which can be improved upon with additional evidence. It is not limited to testing a single hypothesis.

It is hard to describe more about the differences without getting mathematical very quickly, but a helpful place to start to learn more about this Bayesian statistics is: Bayesian Statistics: A Beginner's Guide For some reason, Bayesian statistics is not part of most foundational courses about statistics that many people (have to) take, but is often taught more specialized classes. Hopefully this can change.

Bayesian statistics can be very helpful in understanding relationships and causality in real-life complex systems. It allows us to combine data analysis with subjective knowledge that includes evidence we already know about the system and knowledge from stakeholders. This statistical tradition also allows us to communicate the degree of uncertainty we have in results in a richer way, which is very beneficial in focusing future data collection efforts to improve our understanding. The greater computational complexity of applying Bayesian statistics has been a barrier to use, but luckily this is now mitigated by the availability of easy-to-use software packages!

Many fields, particularly in the social sciences and natural resources management, are embracing Bayesian statistics and moving beyond the exclusive use of frequentist statistics. Here are a few great papers and examples:

Office of Planning, Research and Evaluation, 2017 Research Methods Meeting. Bayesian Methods for Social Policy Research and Evaluation
Charles Michalopoulos, MDRC, “Bayesian Methods in Social Policy Evaluations” and "A Bayesian Reanalysis of Results from the Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project"
Joseph E. Beck, Kai-min Chang , Jack Mostow and Albert Corbett, Carnegie Mellon School of Computer Science. “Does help help? Introducing the Bayesian Evaluation and Assessment methodology

While I don’t think it may be practical to integrate Bayesian statistics into all evaluation and impact measurement, stepping away from the ‘gold standard’ of experimental evaluation design, in favour of applying Bayesian Statistics is long overdue. It is much better suited to understanding how complex change happens and is one of the best tools to update our subjective understanding about impact with data and evidence.

blog type:

Issues & Ideas

tags:

impact measurement

The Common Foundations that Impact Measurement Tools Share

Bryn Sadownik — Wed, 13 May 2020 02:28:00 +0000

What do hundreds of tools, frameworks, guides, templates and other resources on impact measurement have in common?

A limited number of fundamental practices!

This is what The Common Approach to Impact Measurement, an initiative led by the Carleton Centre for Community Innovation, discovered in examining 500+ impact measurement resources, international standards and in consulting directly with social enterprise and social finance organizations. They have synthesized and defined these essential practices as The Common Foundations of Impact Measurement.

The Common Foundations of Impact Measurement practices are:

I described this amazing, bold project in a blog last summer, and since then I have become much more actively involved in this community of non-profits, social purpose organizations (SPOs), grant makers, investors and academics. The Common Approach to Impact Measurement seeks to improve the tools and practices of impact measurement on a number of different fronts that also include defining flexible indicator standards, a common set of organizational indicators, and a community-owned enabling infrastructure that would make it easier for social purpose organizations to manage data and learn from each other.

The Common Foundations are found within the Demonstrating Value Framework and the tools and resources that are available on our website. We are currently updating these resources to include icons that reference the Common Foundations, starting with our newly updated Demonstrating Value Workbook that has now expanded into the 2 complementary Activity Guides in the Getting Started Toolkit

In a series of blogs I will examine and reflect on each of the Common Foundations. Stay tuned! In the meantime, you can learn more about the Common Approach to Impact Measurement at:
carleton.ca/commonapproach/

blog type:

Tools

Free Webinar about the Common Approach to Impact Measurement

Bryn Sadownik — Wed, 11 Mar 2020 19:44:36 +0000

The Common Appoach to Impact Measurment, an initiative led by Carleton University's Centre for Community Innovation is convening a community of non profits, social purpose organizations (SPOs), grant makers, investors and academics to build a better way to measure social impact. The initiative is profiled in this blog posting last summer.

The Canadian Community Economic Development Network (CCEDNet) is hosting a free webinar on March 12 at 1:00 pm (Eastern Daylight Time) to introduce the Common Approach. In this webinar, they will explore how the Common Foundations were derived and how they can help evaluators, consultants, and trainers guide organizations to improved impact measurement. You can also find out more about a national initiative to identify and highlight Common Foundations “champions” who are willing to actively promote the five essential practices in the social purpose sector. Demonstrating Value is already a champion!

The presenter is Margerit Roger, M.Ed. of Eupraxia Training. Margerit is a Winnipeg-based program planner, evaluator and trainer who has worked with community-based organizations, industry, post-secondaries, labour organizations, and provincial and federal governments. Her impact evaluation work is grounded in Theory of Change, social impact analysis, and SROI in an effort to more effectively describe the critical “ripple effect” created by organizations that support vulnerable and marginalized populations. As a result, she feels strongly that the Common Approach and the Common Foundations provide an important strategic and communication tool for impact-oriented practitioners who work in the social purpose sector. Additional guests will be introduced during the webinar.

For more information and to register go to:

https://carleton.ca/commonapproach/cu-events/the-common-approach-and-common-foundations/

blog type:

Events

tags:

event

Mission Model Measure

Bryn Sadownik — Thu, 01 Aug 2019 18:16:01 +0000

This blog is the third in a series that highlights collaborative projects in Canada that have sought to make sense of impact measurement for social enterprises and other community-based organizations, and to advance new tools and common methods.

Mission. Model. Measure. is a new web guide developed through the Social Enterprise Ecosystem (S4ES) project, which is a partnership between the Canadian Community Economic Development Network, the Chantier de l'économie sociale, Buy Social Canada, Social Enterprise Institute and Social Value Lab. This partnership, which is funded primarily through the federal government, connects training, marketing, and impact measurement resources for social enterprises anywhere in Canada.

The site is directed to those who work in social enterprises. They can visit the site and take an impact practice self-assessment. This results in a report that sets out where their social enterprise is right now and what they could work on, benchmarking their practice relative to other social enterprise. The website also provides considerable guidance and tools, with a very extensive collection of impact practice resources that is well organized and easy to search.

The guidance section provides useful instruction on the topic of impact measurement organized in 10 topics and 50 sub-topics. Most of the guidance refers website users to a fee-based web training package ($999) offered through the Social Enterprise Institute. The program instructor is Jonathan Coburn, founding director of Social Value Lab, one of the partners of S4ES. The course is also informed by the Common Foundations set out in the Common Approach.

The Common Approach to Impact Measurement

Bryn Sadownik — Thu, 01 Aug 2019 00:05:16 +0000

This blog is the second of a series which highlights collaborative projects in Canada that have sought to make sense of impact measurement for social enterprises and other community-based organizations, and to advance new tools and common methods.

The Common Approach to Impact Measurement is a community of non profits, social enterprises, grant makers, investors and academics “building a better way to measure social impact.” It emerged from Ontario’s Social Enterprise Strategy 2016-2021 that led to the creation of an Impact Measurement Task Force. The task force resulted in a report which recommended a ‘common approach’ to impact measurement that is intended to provide a foundation for sector-wide measurement, and a reporting methodology that social enterprises can build upon and continue to evolve through insights and lessons learned. The Common Approach to Impact Measurement, based at Carleton University, was established to advance this agenda.

They have been working to articulate the Common Approach by convening stakeholders and defining the following: essential practices, flexible indicator standards (that relate social and environmental indicators to UN Sustainable Development Goals), and a community-owned enabling infrastructure that would make it easier for social enterprises to manage data and learn from each other. In addition, a common set of organizational indicators, is also being defined.

They have now released 25 Common Foundations (essential practices) grouped in 5 areas:

Plan your intended change
Use performance measures
Collect useful information
Gauge performance and impact
Report on results

Other work is on-going. You can find a recent update in this Powerpoint presentation.

The Common Approach seeks to strike a balance between the social enterprise sector's need for some common language and methods (to facilitate learning and encourage additional investment), with the need for measurement to be flexible and supportive. For instance, they have been working to relate the UN's Sustainable Development Goals to the work that social enterprises are doing in their communities, and making it more possible to identify and use indicators that can provide evidence that speaks to those goals. They are currently seeking to broaden their reach beyond Ontario, and Demonstrating Value is exploring opportunities to support their work.

Evaluation and Impact Measurement in the Social Economy

Bryn Sadownik — Wed, 31 Jul 2019 22:56:31 +0000

In recent years, many new initiatives have been launched in Canada that have sought to make sense of impact measurement for social enterprises and other community-based organizations, and to advance new tools and common methods. In a series of blogs, we highlight collaborative projects in Quebec, Ontario and as part of a national partnership to strengthen the social enterprise 'ecosystem'.

The first blog is about a project that has been underway since 2016 that is led by TIESS (Territoires innovants en économie sociale et solidaire), which is an umbrella group in Quebec that supports the social economy through the sharing and dissemination of knowledge, practices, experience and research. The project is named Evaluation and Impact Measurement in the Social Economy (Évaluation et mesure d’impact en économie sociale).

This project looked at the state of measurement practices and evaluation with a critical eye, and has strongly advocated for measurement to be flexible and sensitive to the context, needs and capacity of individual community-based organizations. The project resulted in a web guide for understanding what evaluation and impact measurement are, how they may relate to those working in the social economy (e.g. community-based organizations such as non-profits, social enterprises, coalitions, co-operatives, etc.), and guidance on how to develop and strengthen practices that meet the needs of the organizations. It also presents excellent summaries of current methods. The TIESS Web Guide can be found here. At this time it is only available in French, though it may be translated in the near future. In the meantime (if you are not fluent in French), Google Chrome provides a translation feature for any website.

A key part of the project is the Montreal Declaration on Evaluation and Social Impact Measurement. This declaration was drafted to promote a dialogue and common vision around the opportunities and challenges in the impact measurement field. It came about in part to challenge trends in the field that emphasize standardized, uniform evaluations/impact measurement that focus on ‘results’ over process, and do not take into account the context of the organization and feasibility of measurement. In the case of the later, what is possible to measure is not always important, and what is important is not always possible to measure. Leading with measurement risks distorting the work and mission of organizations. You can read the full declaration (translated to English) here. In a nutshell, the declaration holds that evaluation has to be useful to the organization, focused around learning, and participatory. The method should be flexible and based on context. The declaration also takes a position that funders and investors should bear the cost of any requirements that they impose around impact measurement (to a grantee) and disfavours investment tools such as social impact bonds in which investment in community is based on impact measurment to repay investors.

The Demonstrating Value Resource Society is among the first signatories to the Declaration. I was also honoured to be part of their final project forum held in April in Montreal. If you are interested, videos of the panel discussions are available here

Potential and Limits of Impact Measurement - Upcoming Forum in Montreal (April 9)

Bryn Sadownik — Tue, 12 Mar 2019 19:04:38 +0000

TIESS (L'organisme Territoires innovants en économie sociale et solidaire) is organizing a one-day forum to discuss impact measurement practices, as well as its challenges and opportunities. Several international initiatives and viewpoints will be presented in order to bring out points of convergence around which we can mobilize. The event also includes and unveiling of the Quebec Declaration on Evaluation and Social Impact Measurement.

To learn more about TIESS's work on evaluation and social impact measurement of the social economy, click here (in French only).

For more information and tickets, click here.

----------

Preliminary program

Welcome speech by Robert Beaudry, executive committee member of the City of Montreal, responsible for economic and commercial development, as well as government relations

Opening conference by Marthe Nyssens, Université Catholique de Louvain (Belgium)

“International experiences, common concerns” with:

•Yannick Blanc, La Fonda (France)
•Sébastien Pereau, Concertés (Belgium)
•Kate Ruff, Carleton University (Ontario)
•Bryn Sadownik, Vancity Community Foundation (British Columbia)
•Andrew Taylor, Ontario Nonprofit Network (Ontario)
•Marie J. Bouchard, ESG UQAM (Québec)

Group discussions.

Unveiling of the Quebec Declaration on evaluation and social impact measurement.

Closing cocktail.

----------

blog type:

Events