Equitable Evaluation Practices for Place-Based Development

This is the 11th installment of a 12-part blog series discussing doing evaluation in service of racial equity as part of a collaboration with the Kellogg Foundation. The views expressed here are those of the authors and do not represent the views of any partner organizations, including the Kellogg Foundation.

Evaluation, broadly, is used to generate shared knowledge about a program or intervention. Oftentimes, evaluation is more narrowly focused on assessing the merit of an intervention, determining the level of funding or support for an intervention, reviewing assumptions on which an intervention is based, and holding leaders and organizations accountable to the communities they serve.

The field of evaluation continues to evolve, and in recent years evaluators of color in particular have been more vocal about the need to address longstanding gaps in the field which have perpetuated harm and reproduced inequities through exclusionary processes. As evaluators, we want to move beyond undermining communities and doing harm and move toward exploring how evaluation can serve the goals of those further away from power who have historically been excluded from defining what counts as knowledge and how we define success. 

Historic Limitations to Program Evaluation

Program evaluation typically refers to being able to answer the question “What specific outputs and outcomes did an intervention produce?” This assumes that the intervention or program being measured can be readily described, has distinct boundaries, has a well understood link between cause and effect, and that observed changes can be easily attributed to a specific source. Yet, the reality is far from straightforward, particularly when dealing with place-based economic development interventions.

Place-based work is complex. It involves many partners and is influenced by a variety of factors including local context, individual personalities, economic and political shifts, multiple—sometimes overlapping— programs and strategies, and goals that can changeover time. Evaluating place-based work in service of equity requires thoughtful planning, targeting your activities, and understanding how systems interact with one another. In other words, it involves understanding that the context in which the work unfolds is just as important as clarifying how your coalitions’ work or program is going to make a difference. 

Evaluation Considerations for Place-Based Development

Strategically focusing your place-based evaluation efforts can help to ensure the knowledge gathered about your place-based program(s)or intervention(s) serves community and program goals. The following three considerations can help focus and inform your evaluation strategy:

1. Make sure your measurement and evaluation activities map to your key learning questions. Identifying key learning questions to guide your measurement and evaluation activities will ensure that each activity is serving the goals of your target population and project. Good learning questions will ensure that you are focused on the most important learning priorities. When possible, we recommend that you look for ways to build in community participation in the evaluation process. This can look different depending on the local circumstances. It is critical to take time to understand why community members agree to partner with you.  One way to do this is to form a community advisory board or include community stakeholders in the design of the evaluation or learning activities. To center equity from the start, consider who is involved in defining success and what perspectives are represented in the work. 
2. Focus on understanding the systems at work in your community. Document the physical boundaries and their origins, the key players in the region, the dynamics of power and history that underlie them, and the levers regional partners can employ to shape narratives and decisions. To do this, you can make use of a variety of tools to map the connections between actors, entities, etc. A variety of methods (e.g., social network analysis, outcome mapping) can be applied to get a better understanding the relationships between actors, institutions, and events that influence your work. 
3. Focus on documenting your work. Recognize that place-based work is dynamic, responsive to regional needs, and subject to change. Whether you are designing a new program or forming a regional coalition, it is very important to have a planful approach to documenting your work. Comprehensive documentation of your program's progression serves two vital purposes: ensuring long-term sustainability of your program and providing rich historical context for potential evaluations. As some place-based initiatives take years, or even decades, it’s possible that many people will work on them at different times. Documenting your work will help coalition members communicate over time about what has been done, what changes were made, why, and who was involved. Further, having detailed information about how your program is being conducted will give you data points to choose from, should you later evaluate the program. Additionally, when you can collaborate with community stakeholders to guide the documentation process you are also likely supporting local capacity building to document and share impact. 

Recognizing Types of Changes in Place-Based Development

When choosing what you will measure over time as part of your program or in preparation for an evaluation, consider keeping track of changes in the socio-economic and political environment, key systems, power structures, and specific groups of people affected by these factors. Traditionally, evaluations in place-based development have been largely shaped by federal or state reporting requirements that ask for jobs created or retained, or economic impacts accrued in a geographic area. These indicators tell limited and incomplete stories about outcomes of programs and interventions. They also fall short in relaying how efforts impact some populations and communities more negatively or positively compared to other populations and communities. With more complete understanding it is also possible to decipher who or what organizations have the power and influence to make key place-based development decisions. When considering what changes you will track over time, consider first what type of change you expect to see as a result of your efforts. This may help you choose which types of change are important for your organization to prioritize in monitoring and evaluation. You can also consider how time factors into what you expect success will look like.

Stachowiack et al. (2018) shared a helpful typology for characterizing changes in complex, place-based efforts that can help to organize your plans for what and how you will go about documenting changes and impact. They are described as:

• Early changes: shifts in the policy and decision-making environment that lay the foundation for systems and policy changes such as increased partnership quality, increased collaboration among partners, and increased awareness of an issue. 
• Systems changes: changes to institutions (or the formation of new institutions) that can affect the target region, population, or issue. These organizational changes may be formalized and likely to stay, or more informal experiments that could lay the groundwork for future formalized changes. Also consider tracking changes that happen within a single organization, multiple organizations with a common purpose, or multiple organizations with multiple purposes.
• Population changes: changes within the target population of the coalition or initiative which may be within geographic areas, specific systems, or have specific needs.

We have proposed helpful ways to approach equitable evaluation practices. Some of these approaches may be included in a formal evaluation, and we appreciate that it is unlikely that a single evaluation will be comprehensive enough to cover the multiple dimensions we reviewed. However, we also know that evaluation practices that advance racial and geographic equity are not the norm and thus we need to be intentional in embedding inclusive and participatory approaches that center communities and populations that have historically been excluded from knowledge production activities such as evaluation. 

Evaluation as a practice that centers equity has the potential to strengthen our understanding of place-based development work. The tools it can bring to bear help us to understand systems, identify root causes of problems, and understand regional context and community priorities. All of these things are necessary to build better regions for transformative and equitable development.