Considering Language Justice in Equity Centered Evaluation Practice

This is the 10th installment of a 12-part blog series discussing doing evaluation in service of racial equity as part of a collaboration with the Kellogg Foundation. The views expressed here are those of the authors and do not represent the views of any partner organizations, including the Kellogg Foundation.

Picture this scenario: A state agency has allocated funds to create a health-related communication initiative to raise awareness about vaccinations and increase uptake among marginalized communities. A lot of money was spent on creating an extensive website, printing materials, logo, and catchlines. However, once implemented in the community, the campaign had the opposite result. An evaluation after the launch and delivery of the materials found that community members couldn’t read in the language the materials were printed in, nor did they have the ability to access the website’s resources appropriately. The initiative had unintentionally excluded the very same marginalized communities it was trying to reach.  

This is a real-world example that illustrates the unintentional consequences of ignoring culturally appropriate language and failing to engage community members in a process to inform our work.  If we want our efforts to benefit communities, we need to reexamine how we are conceiving inclusion and the role that plays in our evaluation practice.  

Historically, evaluation as a field has only recently begun to focus on language justice. Much like the researcher’s positionality, language is not neutral and power dynamics related to language are always at play. As researchers, evaluators, and practitioners, we may not always have language at the forefront of our thinking. It is likely that many of us who do not live with a disability may even take for granted the language and accommodations we see in our everyday settings.  

Being mindful of language in evaluations reflects the cultural competencies needed to ensure evaluations do not reproduce inequities in their use of language.

Equity centered evaluations must be responsive to the diverse needs of cultural communities and other identities, which often involves tailoring to one’s language or language needs. As evaluators hold power in the evaluation process, we have a responsibility to mitigate these power dynamics as much as possible so we can foster authentic connections with community partners and others whom we want to benefit from our work.  

We view language as essential to justice and equity and refer to its embodiment in evaluation as language justice. Language justice refers to “the right everyone has to communicate in the language in which one feels most comfortable” with key tenets in transformational thinking to empower action and challenging dominant language narratives to advance social justice.  

Getting Started with Language Justice  

Embedding a language justice approach to the process of evaluation can happen in many ways. No task is too small, and your team can start making changes for more inclusive language now. Below, we provide recommendations on where to begin and questions to guide your decision-making. 

During Your Planning 

You know enough about your participants to create an inclusion criterion and screen them for participation in the actual study. Now, you have an obligation to learn more about the languages spoken in their communities. During the planning phase of your evaluation, it’s important to take the first step to understand the context you’re working in. Understanding the context of language is often overlooked in evaluations. What is the history of languages in this community? Are there any language groups that have been excluded in your topic area? What languages do these communities feel most comfortable in? Does your evaluation reflect those needs? What else can we do to center the perspectives of non-English speakers? Always try to challenge your viewpoints of “English as a dominant language” and instead encourage a willingness to learn more about the diverse contexts of communities you want to impact.  

Some agencies, like federal and state government entities for example, may have policies in place for language access and accommodations. Make this your starting point; don’t let this limit the efforts needed to better serve communities and engage them inclusively in your work.  

If we want to engage diverse language groups inclusively in our evaluations, we also need to make sure to budget the time, resources, and staffing to be successful when implemented. Other best practices include the creation of a language access plan as a standard within each evaluation plan. Accounting for the focus on language in the process of designing an evaluation will become a standard both with changing of professional norms, policies, and our own mindsets about what it takes to do this work.  

Throughout the Design 

How we frame our research questions, in terms of language, matters. Here, we consider language in the context of asset-based framing vs deficit framing. Researchers tend to focus on solving problems, and thus our research questions may be framed around the social issue we want to solve rather than the community assets that offer solutions. However, questions about social issues can be addressed all the same with asset-based framing. This is in line with RTI’s Equity Centered Methodology Framework, where we explain how asset-based framing of research questions can be a transformational practice to shift power to minoritized and marginalized communities. Does your research question tend to focus on the deficits of communities? How can you revamp the framing of the question to focus instead on assets or successes?  

When we collect data using traditional surveys or interviews, we often risk excluding a wide range of communities that deserve to have their voices heard because of language. Some people may be reluctant to participate in interviews because English is not their first language. Others may not have access to the appropriate accommodations needed to fully participate. Other ways to collect data that are more inclusive to diverse language communities include storytelling, photovoice, or other participatory methods that invite communities to share insights whether it be through imagery, stories, or building artifacts. Photovoice is one illustrative example of engaging historically excluded groups, such as individuals with disabilities, more inclusively in the research process. Additionally, the categories we choose to define people (e.g., gender, race, ethnicity, etc.) in our instruments affect how communities are accurately and respectfully perceived.  

Other components to consider are related to the facilitators or interviewers who will be collecting the data. Do they reflect the cultural communities you’re trying to reach? Are their staff equipped to communicate in languages for inclusive practices? Incorporating these activities can minimize issues of validity in qualitative and quantitative evaluation approaches.   

When You Communicate 

Communicating throughout any evaluation is important. But, if you’re not getting your message across to diverse audiences, then how does that compromise the utility of your efforts? What would you miss if you don’t center on language justice?  Considerations for language justice can be incorporated in how you communicate in meetings, evaluation results, and across partnerships. Here, we consider the accessibility of materials, the reading level of what we write, and other strategies to ensure inclusion of broader audiences. Remember that having one language only allows a certain group of people to access the information. What implications does that have for the goals of your evaluation?  

Some strategies you can apply for communication include: 

• Incorporate plain language guidelines to ensure materials are easy to understand across a broader audience. The CDC has several resources that can be helpful when you’re developing communication materials or meeting agendas.  
• Use readability tests, like the Kincaid reading level, to ensure a document or tool is easy to read and understand. We recommend a 6-7th grade reading level for any recruitment or data collection materials. 
• For virtual meetings, turn on closed captioning, include subtitles, and share transcripts with meeting notes after meetings are done. See the Language Justice Checklist for hosting in person events.  
• For communicating evaluation results in PDFs or PowerPoint, make sure to use alt text for people who use screen readers. Given what you’ve learned about the languages in these communities, consider whether there is a need to translate reports into other languages.  

Other resources are provided below to guide you in your journey to language justice:  

• Language Justice in Evaluation 
• Adopting Inclusive and Non Violent Language 
• Why You Should Use Inclusive Language 
• Considerations for LGBTQIA+ Communities 
• Beyond Plain Language: Creating Effective Communication Materials for People with Intellectual and Developmental Disabilities (rti.org) 
• Avoiding heterosexual bias in language (apa.org) 
• Other examples of guidelines on language access 

We recognize that changing how we practice evaluation can take time and opportunities will vary depending on the project. We encourage you to consider what is a good entry point for your work and to make use of these resources to consider the role of language in your practice.  As the Kellogg Foundation’s Practices Guides remind us, doing evaluation in service of racial equity must be an intentional process of learning and unlearning.