How can data drive?

In today's dynamic landscape, data can serve to propel leaders and decision-makers toward informed choices as a driving companion. The prominence of data-driven decision-making in the business realm underscores its undeniable value. Data similarly lies at the core of our evaluation, steering all stakeholders forward. However, the decisions to make with the available data aren't always straightforward; this is because data can be elusive, complex to grasp, and nuanced to interpret. These characteristics are far from desirable in our metaphorical "driver." At ACED, the process of crafting evaluation plans is guided by a consideration of feasible, valuable, and optimal data types, which in turn, inform the selection of data collection frameworks. It is because of an understanding of data’s complex nature that we remain steadfast in our commitment to being data-driven at ACED.

Evaluation exists to inspire well-informed action toward the goals of a project or organization. This action must be underpinned by verifiable information. In highly goal-driven projects or organizations, there is a risk of overemphasizing individual stakeholder experience compared to structured data collections. While individual experiences hold significance, they must be contextualized within a broader spectrum of project impact data.

 

Data alone do not tell stories; rather, the meanings gleaned from data analysis are shaped by methodological decisions made throughout the collection and analysis process. The comprehensive body of data amassed during evaluation helps us tell a collaborative story of project or organizational success. Effective decision-making fueled by data necessitates both data availability, avoiding isolated decision-making, and the transformation of raw data into actionable information. More trustworthy data translates to better information, driving superior decision-making. In the spirit of trustworthiness, we examine the quality of each dataset collected using the principles of validity, reliability, and fairness, in alignment with leading organizations in the field.

Validity, akin to accuracy, seeks to articulate the alignment of the interpreted information with “true” values. Calculations of validity evidence are especially helpful when seeking to measure latent characteristics, such as the confidence of students in chemistry classrooms in their ability to solve problems. Reliability, akin to precision, seeks to demonstrate the consistency in data outcomes. There are a variety of accessible coefficient calculations that provide this support for interpreting datasets, so reliability evidence is almost always included in ACED evaluation plans. Fairness is a characteristic unfortunately often overshadowed by validity and reliability which seeks to identify whether there is consistent data interpretation across groups of any meaningful grouping, especially when considering historically excluded groups. Collecting evidence for fairness includes leveraging techniques such as measurement invariance testing (one of our areas of expertise!). At ACED, we meticulously consider each facet of data quality to fully understand trustworthiness and provide the most accurate degree of confidence in the data we collect, leading to the kind of decisions we can be proud of.

With an established toolbox for collecting evidence that provides support for the trustworthiness of our data interpretations, our commitment at ACED remains unwavering: to keep data alongside all stakeholders in the driver's seat, empowering us with informed insights rather than overwhelming us with evaluation outcomes, thereby driving continuous improvement.