Conceptualizing better data work in non-profits: Prompt 3 – Participatory research, evaluation and design

Of the 5 prompts for Conceptualizing better data work in non-profits, Participatory (Action) Research, Evaluation, and Design is the approach I’ve already advocated for the most. In my most recent paper, we suggested that an opportunity for approaching data work in non-profits involves trying to “Build ways for youths and frontline staff to help create data systems and evaluate data work” (p. 649). And in Real Big Data we proposed using PAR as a strategy for young people and their communities to be involved in designing, developing, and using data systems, which could be done using participatory research and design processes.

First: what is PAR/PE/PD? While there are differences in focus across these practices, the common factor in each is that those most impacted by the outcomes of the process (in this case, the design of a data system) should be involved directly in making decisions about how the system is designed, how it is implemented, and how it gets evaluated. In a participatory approach, stakeholders are not only interviewed and asked for their opinions; instead, they help to design the focus of the study and the method/questions to be used.

I believe there are two primary ways that participatory approaches can be used in building better data systems in non-profits:

• To design a data system for an organization/entity in a way that involves stakeholders/clients as co-creators of the system.
• As the data system itself, whereby clients, staff, and other stakeholders are involved in determining what questions are important to answer, then designing and implementing studies to do so.

Let’s address these in order.

Designing Data Systems with Stakeholders

A participatory approach to designing a data system with stakeholders as co-creators might be a project that takes place once (the development process) and then later revised as necessary. A group of stakeholders is brought together. This should include, and perhaps be done by a majority, frontline staff and clients. However, other staff and even outside stakeholders are needed too. Including flexible funders that listen well may help funders understand why this process is important and to get on board with the results. Power relations because of positionality must be addressed and discussed directly, with those with greater power finding ways to mitigate the effects this can have on the group (e.g. is there a way to guarantee that clients won’t face negative impacts for their participation?).

The group must receive some training in how the research or design process works – enough to empower some informed decision-making. Likewise, some capacity-building about data is required so everyone feels they can play a smart role. Everyone that might join the group has personal experiences with data – start there! Critical perspectives should be part of this learning – including studying ways data systems cause harm, where data is going, and positive, alternative models for building data systems. Unearthing the positions people have about data, and discussing these as a set of values rather than facts is key (e.g. “I value keeping funders engaged” is better than “funders need data about how we improve kids grades”).

This work is time consuming but creates a solid foundation to work from. Further, without it, a PAR process is likely a symbolic farce. The group will be quietly dominated by those who sound smartest, or talk the most, or intimidate others. So this part of the process cannot be skipped or rushed.

From here, the group can develop any technical skills needed in research, evaluation, or design, often on the job. Someone part of the PAR team should have these skills – and be ready to teach them in a way that builds on participants’ existing experiences and knowledge. It is essential that this person be understood as a technical expert only – and that this expertise must be understood as only part of a decision-making process that also balances team members value commitments, ideas, and critical thinking.

Then the team can proceed with a research, evaluation, and design process. It must not be forgotten that a data system involves both the technical elements (what data do we collect? how? where is it stored?) and the process elements (who collects it? when? how?). A design team should develop both of these. The end result should be a data system that can be used by the organization, but which is designed by a stakeholder team to be respectful of all involved and the mission of the work. An organization that takes this work seriously will not treat this as merely advisory work, but will make certain to implement whatever is designed.

Ideally, a process of evaluation and revision of the data system will be part of the design outcomes. This evaluation/revision should consider including the design team or at least people who fill the same stakeholder roles. It may be necessary to do the educational work of above with future groups, so that they too can make well-informed evaluations.

PAR as the Data System Itself

A more radical approach to building a data system might use PAR as the data system. This could go a couple of ways.

1. Internal evaluator team: This team might be built in a similar fashion to the one above. Rather than design a data system to be used by the organization, however, this type of team might just work with the organization on a regular basis to determine important questions for study, then design and conduct an appropriate process. Results could be presented to stakeholders in the organization, with proposals for evolving the work or methods to communicate the work to outside parties.
2. External study of harmful systems or changes in social determinants: This relates to Prompt 4 in this series, the analysis of systems external to the organization that impact the lives of clients or stakeholders. Rather than focus on what clients are doing or how they are changing as a result of the program, a PAR team might try to understand the impacts of the external environment. This might be used to develop better understanding for improvement (e.g. teachers at a school working to understand the environment of students lives to understand why they aren’t showing up to school, then finding ways to improve the school’s response) or to advocate (e.g. government needs to change this harmful policy). This kind of approach does not directly produce outcomes data about clients, but rather focuses on addressing challenges clients face.
3. Participatory data as part of organizational processes: This one is a bit of a mind-bender and thought experiment. What if participatory processes around data simply became a part of how organizations conduct their work. For example, what if part of intake was a conversation between a worker and client(s) about what was important to know about and how they’d learn it – and then this became what the worker and client recorded together about their interactions. What if, instead of a database, this information was regularly examined by, visualized, and discussed by stakeholders involved? The ways an organization interacts with clients might just be part and parcel to how they create data – which would require some major shifts in how we think about service delivery too. A concrete example can be found in a long ago post of mine about case notes. This approach might also share a lot in common with the Reggio Emilia documentation approach.

As a data system itself, PAR could have serious potential to design alternative approaches to data, which might also evolve organizational practices. As far as I know, such a system is relatively rare, if it exists at all. There are a growing number of organizations using Participatory Evaluation, which is a time-bound way to approach this. However, as far as I know, these are typically one-off projects, rather than a regular system of operation that could be called an actual data system.

PAR Essentials

PAR is an appealing approach to improving data work. There are existing models for doing PAR and a growing number of people with experience doing and facilitating the work. However, I think there are a few additional essentials worth naming here that aren’t necessarily discussed in all PAR guides:

• Pay PAR participants for their work, even if it also helps them fulfill a personal driving purpose. This is real work and should be recognized as such by all stakeholders. Work that’s paid for is seen and understood as more than symbolic – because it costs, it is taken seriously. It helps participants prioritize this labor.
• Connect to participants personal motivations. Participating in research projects is not most people’s motivation. Collecting data does not sound fun to most people. Analysis sounds downright scary. And all of this is a lot of time commitment. It only really works if participants understand that it really matters, and find ways it really matters to them.
• Take education seriously and ensure it connects to the curiosity, experiences and motivations of participants. It should be done in a way that continuously reminds participants to draw on their own expertise and to participate fully.
• Remember that everything has an experiential politics, and the ones here are well-represented by popular education. This includes the politics of positionality too – and these must be discussed and addressed as part of the work.
• A competent researcher is NOT necessarily a competent PAR facilitator. In fact, most are terrible at it. PAR facilitation requires skills more in line with popular education and organizing than research. Find partners who can co-facilitate. Better yet, also find a participant who doesn’t feel knowledgeable about research to co-facilitate.
• The process must be supported by organizational leaders who really believe in the work the team is doing and who can explain the work competently to others, including the deeper purpose behind approaching things this way. The organization will be challenged for adopting this approach and leaders must become fierce and educated advocates, or will find themselves disempowering the team and returning to old modes of data work.

Participation can, and often is, symbolic. This is just a starting list of ways to make it work for an organizational data system. Starting, maintaining, and integrating the work of PAR teams into organizational data systems will be a new challenge because it is unfamiliar and requires a great deal of new learning and change within the organization. This is why my colleague Ross VeLure Roholt and I are working on trainings in what we call Transitioning Leadership. However, because it is an approach with a great deal of scholarly support and experienced facilitators floating about, and because it can easily be implemented iteratively over time, PAR is a great starting place for organizations wanting to do data work better.