Treffer: Validating a Measure of Problem-Framing Ability to Support Evidence-Based Teaching Practice.

In this evidence-based teaching practice paper, we report on development, implementation, and validation of the Design Skills Test (DST), an assessment of design problem framing ability. Methodology. The DST includes an authentic design scenario and a coding scheme to characterize 1) factual and conceptual information used to frame the problem in terms of needs/constraints; 2) design practices used (e.g., generating ideas, considering multiple stakeholders, remaining tentative); and 3) stylistic choices (e.g., organizing their response, depicting context). We developed three DST scenarios and tested them in a chemical engineering program over a three-year period (n=580). To make data analysis feasible, two undergraduate peer-learning facilitators analyzed each DST independently (14 PLFs contributed), following minimal training. Results. Using a validity-as-argument approach (Linn, 1994), we argue that the DST provides valid information about design problem-framing ability, provided the information is used for course improvement purposes. Inter-rater reliability for factual/conceptual codes was 65% to 83%; for practice codes 52% to 77%; and for stylistic codes 68% to 80%). Conclusions. Our findings indicate that the DST sheds light on students' design problem framing ability and provides valid evidence to help faculty evaluate the impact of incorporating design challenges, as not all design challenges support students to learn how to design. Given that professional engineering design practice relies on knowing how to frame problems, it is important for students to have opportunities to develop problem framing ability. Implications. While reliability with minimal training was lower than would be acceptable for research purposes, for instructional purposes, this represents a significant reduction of faculty time. To enhance reliability, we worked with instructional designers to develop an online, selfpaced training. [ABSTRACT FROM AUTHOR]