Choose Your Own Adventure: Six Tips to Conduct Equitable Evaluation

This is the seventh installment of a 12-part blog series discussing doing evaluation in service of racial equity as part of a collaboration with the Kellogg Foundation. The views expressed here are those of the authors and do not represent the views of any partner organizations, including the Kellogg Foundation.

RTI’s Equity Centered Methodology Framework and the Kellogg Foundation Guides on Doing Evaluation in Service of Racial Equity underline the importance of centering diversity, inclusion and equity throughout the research and evaluation process. These conceptual values are ideal in theory, but applying them in practice can be difficult, especially when we talk about them in esoteric philosophical ways.

Sharing real world examples of applying equitable evaluation can be a helpful way of communicating best practices and making the unknown seem less risky. We share concrete tips on conducting equitable evaluation from one of our projects focused on cocreating health programs through community partnerships.

1. Prepare for the evaluation by understanding context

You are an evaluator, and your role holds power, inherent biases, history, and epistemic privilege (i.e., you determine what knowledge is). These qualities can separate you from your community or social issue of focus. Your responsibility is to bridge that gap by understanding differences for opportunities. We recommend preparing for the evaluation by learning the cultural context of the program, organization, community and/or other surrounding policies.

For our project, we committed time to learn what the organization and community were about, what was their history, what was done in the past and how did it go, what did staff think, and where were the gaps and bright spots. We started meeting with key staff to hear about their experiences implementing the program and building relationships to raise awareness about our ongoing evaluation and its goals to cocreate collectively to ensure meaningful results that could improve programming.  

2. Engage diverse partners at the beginning and throughout the project cycle

In our blog series, we have emphasized the value of engaging community members throughout the evaluation process. Engaging partners from the beginning of the project and ongoing is critical to ensuring equity, diversity, inclusion, and belonging (EDIB). Engaging diverse partners is also important to capture representative perspectives across leadership levels and lived expertise.   

For our project, we engaged various levels of partners, including staff and leaders from the organization and community members with racialized backgrounds and who were impacted by health disparities. With staff, we leveraged department-wide meetings to initiate a feedback loop while we prepared materials for engaging community members through a formal advisory board.

3. Identify a shared purpose of the evaluation

Having a shared purpose with community members (or partners) can lead to more successful outcomes and demonstrate an equity-centered approach to align community needs with funder requirements. Here, you can invite input on whether your purpose is meaningful. What is the drive behind the evaluation? What is the motivation behind the work and how is it aligned (or not) with communities’ priorities? Where can we be more flexible and adaptive?

While our project had set expectations from the funder, we were able to identify many opportunities to adapt and extend the purpose of the evaluation. For instance, we asked staff who were involved in programming to share how they would use the results from the evaluation. Then, we asked community advisory board members whether there was anything missing in terms of generating results that would be relevant and responsive to the community. These key pieces informed our shared purpose in designing the evaluation.  

4. Frame questions that capture community concerns and priorities to the extent possible

Good questions ask about root causes of social issues and use asset-based language to refrain from harming a community with deficit framing. Even better questions are cocreated with the community; but this is not always possible. Generally, framing questions that can better capture community concerns is a key aspect of an equity-centered evaluation. In practice, that can feel impossible given project funding requirements and constraints.

For our project, we collaborated with a Community Advisory Board (CAB) to cocreate evaluation questions. One challenge we had with our discussions was scope creep, or running the risk of getting too far off topic from what the funder originally requested. This is where partnership skills are most essential to show transparency in the constraints you work in and negotiate where and how you may be able to develop questions that are most applicable to your community of focus. For us, that meant revisiting our community charter and reminding our CAB about the scope of the work, our mandated deliverables, and the reality of our timeline to develop questions that were feasible but still impactful for the community. When you build authentic connections, it is easier to be transparent and work with your partners to navigate those constraints.

5. Design the evaluation with EDIB values in mind

This component encompasses all phases of design, selecting instruments, collecting, and analyzing data. For equity-centered evaluation, flexibility and adaptations are key to developing data collection and analytical strategies that reflect values of EDIB.

As an example, you may have sent a survey out and not gotten any responses, particularly if you’re trying to reach staff or community leaders who are already overwhelmed with their own tasks. Collecting data in equity-centered research and evaluation doesn’t have to be prescriptive and can embed innovative practices that allow for changing how we typically do the work. You can instead ask, are there better ways of collecting this information? Is the language in this instrument meaningful to the people I am trying to reach? Am I meeting people where they are to convenience the process? Are there ways I can partner with communities instead?

In our project, we collected data during the pandemic, where our evaluation was not a priority. We initially administered surveys but knew we wouldn’t get the response rate we wanted if we didn’t create touch points with our participants. We started offering invites to join team meetings to facilitate the survey administration and facilitated group discussions for the survey questions instead. We had already built the relationships we needed to engage more deeply for adapted data collection approaches.

We also applied participatory techniques for data analysis. Once we obtained data from our survey, we summarized the results at a high level, demonstrating the growth and strength areas. We then shared preliminary findings with staff first to elicit reactions and additional input on the story behind the results. This was done using a virtual whiteboard at the department-wide meeting. Then we shared qualitative data from these discussions and quantitative data from our survey with our advisory board to provide recommendations for action.

6. Disseminate and use results beyond the research project

Now, you are done with your project. What’s next? As evaluators, we tend to disseminate results only to our funder to meet requirements and do not prioritize community-based dissemination efforts. This may be due to a variety of reasons, like time, funding, and other constraints. However, community-based dissemination that extends results beyond the project can build capacity in communities and demonstrate transparency in communication. This requires building in time and resources to translate complex concepts into brief products that integrate plain language guidelines. While this is not always possible, making it a practice to ask for this step to be a part of how you work helps to shift the norms and expectations in our field.

Sharing results more broadly with the community can also offer opportunities to assess data quality and validity, and whether the results resonate with communities’ experiences. We encourage you to consider diverse mediums to prioritize community-based dissemination for broader reach and impact.

Choosing Your Path to an Equity-centered Evaluation Practice

Our experience in this project and others have reinforced the fact that there are different ways to get started and decide how you want to enter the work. You get to choose your own path. You should not wait for the perfect project to implement these practices. There will be projects where you can only make changes in a couple of places. That is worth doing. All these instances are important because they can improve the evaluations we conduct, and because in the process we are improving and honing our own practice. As evaluation practitioners, we are pushing our field to evolve and improve what is seen as "standard" practice. In this way, evaluation in service of equity becomes the norm.