Treffer: Multimodal Learning Analytics of Collaborative Patterns during Pair Programming in Higher Education
Postsecondary Education
Weitere Informationen
Pair programming (PP), as a mode of collaborative problem solving (CPS) in computer programming education, asks two students work in a pair to co-construct knowledge and solve problems. Considering the complex multimodality of pair programming caused by students' discourses, behaviors, and socio-emotions, it is of critical importance to examine their collaborative patterns from a holistic, multimodal, dynamic perspective. But there is a lack of research investigating the collaborative patterns generated by the multimodality. This research applied multimodal learning analytics (MMLA) to collect 19 undergraduate student pairs' multimodal process and products data to examine different collaborative patterns based on the quantitative, structural, and transitional characteristics. The results revealed four collaborative patterns (i.e., a consensus-achieved pattern, an argumentation-driven pattern, an individual-oriented pattern, and a trial-and-error pattern), associated with different levels of process and summative performances. Theoretical, pedagogical, and analytical implications were provided to guide the future research and practice.
As Provided
AN0161748240;[jq69]08feb.23;2023Feb09.06:30;v2.2.500
Multimodal learning analytics of collaborative patterns during pair programming in higher education
Pair programming (PP), as a mode of collaborative problem solving (CPS) in computer programming education, asks two students work in a pair to co-construct knowledge and solve problems. Considering the complex multimodality of pair programming caused by students' discourses, behaviors, and socio-emotions, it is of critical importance to examine their collaborative patterns from a holistic, multimodal, dynamic perspective. But there is a lack of research investigating the collaborative patterns generated by the multimodality. This research applied multimodal learning analytics (MMLA) to collect 19 undergraduate student pairs' multimodal process and products data to examine different collaborative patterns based on the quantitative, structural, and transitional characteristics. The results revealed four collaborative patterns (i.e., a consensus-achieved pattern, an argumentation-driven pattern, an individual-oriented pattern, and a trial-and-error pattern), associated with different levels of process and summative performances. Theoretical, pedagogical, and analytical implications were provided to guide the future research and practice.
Keywords: Collaborative problem solving; Computer-supported collaborative learning; Pair programming; Computer programming education; Collaborative pattern; Multimodal learning analytics
Introduction
Grounded upon the sociocultural perspective of learning (Vygotsky, [66]), collaborative problem-solving (CPS) focuses on group members' knowledge construction and meaningful practices through continuous interactions and idea improvement with the technological and pedagogical supports (Hmelo-Silver & DeSimone, [20]; Stahl, [57]). Pair programming (PP), as a mode of CPS in computer programming education, asks students work together to solve challenging programming tasks, improve computational thinking, and enhance real-world problem-solving ability (Beck & Chizhik, [2]; Chittum et al., [7]; Sun et al., [63]). However, PP is a complex phenomenon, in which multiple modals (e.g., communication, behavior, socio-emotion, etc.) interact constantly to form different collaborative patterns and finally influence the quality of collaboration (Stahl & Hakkarainen, [59]). Considering the complex factors that may influence PP, it is necessary to investigate the collaborative patterns of PP as well as their associations with the collaborative quality. Recently, some research has explored students' collaborative patterns in CPS (e.g., Han & Ellis, [17]; Lin et al., [33]; Webb et al., [68]), but results varied regarding the relations between students' collaborative patterns and the quality of collaboration. More importantly, we found that most of the previous works merely analyzed the single dimension (e.g., cognitive process, interactive type) of CPS, but rarely examined the dynamic and temporal characteristics formed through multimodality during collaboration, which might lead to an incomplete understanding of the complexity in collaboration. To fill this gap, this research collected the multimodal process-oriented data (including verbal audios, computer screen recordings, facial expression recordings) and programming products data during students' PP in higher education and utilized multimodal learning analytics (MMLA) to detect and analyze students' collaborative patterns. Specifically, we identified clusters based on the assessment of collaborative processes and final products, and further examined the quantitative, structural, and transitional characteristics of different clusters to reveal the collaborative patterns. Based on the results, we provided theoretical, pedagogical, and analytical implications to promote future practice and research.
Literature review
Grounded upon the social, cultural, and situated perspectives of learning (Vygotsky, [66]), collaborative problem-solving (CPS) emphasizes students collaborate together to solve ill-structured problems, construct knowledge, and achieve shared goals (Damon & Phelps, [9]; Dillenbourg, [14]). Compared to the instructor-centered learning mode, CPS aims to achieve an active and constructive learning process through students' mutual interaction and knowledge co-construction (Brown et al., [4]; O'Donnell & Hmelo-Silver, [41]). Pair programming (PP), as a mode of CPS in computer programming education, requires two students engage in a coordinated way to solve programming problems and complete complex programming tasks (Bryant et al., [5]; Denner et al., [11]). Recently, PP has been widely used as a learning approach in higher education to promote active learning (Hawlitschek et al., [18]). PP emphasizes that knowledge is not considered as predefined and structural information delivered from instructors but is explored and constructed by students during the collaborative process of programming and debugging (Sun et al., [63]). Moreover, empirical studies have indicated that PP has potential in arousing novice learners' motivation and interests in computer science (Chittum et al., [7]), fostering their computational thinking skills (Romero et al., [53]), and improving their problem-solving abilities in reality (Beck & Chizhik, [2]).
However, PP is a complex phenomenon that involves multimodal interaction and coordination between individual student, student group, learning environment, and the knowledge artefact (Stahl & Hakkarainen, [59]). Specifically, the multimodality can be reflected through student pair's communication (Barron, [1]; Ouyang & Xu, [47]), behavior (Stahl, [58]), emotion (Kwon et al., [32]), interaction (Zemel & Koschmann, [71]), etc. Furthermore, the multimodality emerges during the collaboration with complex, multilevel, multilayered characteristics, which may influence the quality of collaborative learning (Byrne & Callaghan, [6]; Hilpert & Marchand, [19]). However, previous empirical research varied about the relations between students' collaborative patterns and the quality of collaboration (e.g., Han & Ellis, [17]; Lin et al., [33]; Webb et al., [68]). For instance, Lin et al. ([33]) detected 45 college students' CPS patterns in online forum based on their cognitive engagement; the manipulation-centered pattern demonstrated a deeper cognition of students in collaboration, while the discussion-centered pattern appeared more off-topic discussions. Webb et al. ([68]) identified 45 students' collaborative patterns in the third-grade mathematics course based on their interaction characteristics. There were groups that took turns to initiate a strategy, groups with students that generated their own strategies, and groups where one student took responsibility to generate the strategies. The results indicated that no single pattern was better than other patterns for leading students' success in collaboration. Moreover, these works mostly focused on the single aspect (e.g., cognitive process, interactive type) of CPS without considering the complexity, multimodality, and dynamics of collaboration, which might cause incomprehensive understandings of the collaborative patterns (Borge & Mercier, [3]). Overall, exploring collaborative patterns in PP, especially from a multimodal, dynamic, holistic perspective, is necessary to help researchers, instructors, and students unfold the complex factors that influence the collaborative quality as well as how they influence (Lu & Churchill, [34]; Perera et al., [50]).
From an analytical perspective, due to the complexity and multimodality of CPS, multidimensional, temporal, and fine-grained approaches are called for exploring students' collaborative patterns in computer programming education. Multimodal learning analytics (MMLA), as a new trend of learning analytics, leverage advances in multimodal data (e.g., speech, eye gaze, heart rate, body movement data) to capture and mining learning process and to address the challenges of investigating multiple, complex learning-relevant constructs in learning scenarios (Mu et al., [40]; Ochoa & Worsley, [42]; Wiltshire et al., [69]). Recently, relevant research has applied MMLA to reveal the complex, multimodal, and dynamic characteristics of CPS. For example, Sun et al., ([64]) utilized discourses analysis, click stream analysis, and video analysis to analyze 63 junior high school students' discourses, behaviors, and perceptions during collaborative programming. Kawamura et al., ([27]) modeled 48 students' wakefulness states on e-learning platforms and further detected drowsy students according to their multimodal data (i.e., face recognition, seat pressure, and heart rate). Wiltshire et al. ([69]) collected multimodal data (i.e., gesture, speech, mouse and keyboard movement) from 42 pairs of undergraduate students and used growth curve modelling to investigate how students' multimodal movement coordination dynamically changed during collaboration. Overall, compared to traditional statistical analysis (e.g., questionnaire data, performance assessment data), MMLA has the potential to reveal the complex, multimodal, dynamic collaborative patterns in PP from a multidimensional, temporal, and fine-grained perspective.
To address these gaps, the current study applied MMLA to examine students' collaborative patterns in a face-to-face, computer-supported PP environment in higher education. Specifically, we collected students' multimodal process-oriented data (including verbal audios, computer screen recordings, facial expression recordings) and programming products data. We proposed an analytical framework that integrated MMLA methods to identify students' collaborative clusters in PP and further revealed the characteristics of clusters. Specifically, two main research questions were proposed:
RQ1:
RQ2:
Methodology
Research context, participants, and programming procedures
The participants were 40 undergraduate students (23 males, 17 females) without prior programming foundation or experience. 20 pairs (2 students/group) were randomly assigned. Specifically, the 20 pairs included 5 male-only pairs, 6 female-only pairs, and 9 mixed pairs. The research dataset consisted of 19 datasets; data from one pair (i.e., a mixed pair) was damaged, which was excluded in this research. The research environment was a computer-supported collaborative problem solving activity. Two students in the same group sat opposite to each other and controlled a computer individually (see Fig. 1a). The computer screens were connected and shared by a remote screen control software. Student groups were asked to collaborate and learn programming on an online programming platform
Graph: Fig. 1 The research context
Two sections were designed to support student pairs' PP process (each section lasted 25 min). In the first section, students watched the instructional videos and learned to use the coding blocks (i.e., loop, if) on the platform by completing a series of programming tasks together. In the second section, group members collaborated to complete a final programming task within 25 min by using the coding skills they had learned. The final programming task included two requirements: (1) creating a five-by-five brick building with at least four bricks over water, and (2) the foundation of the building was first constructed with boulders and then with woods. Pairs were asked to use at least two
Data collection and dataset
The research dataset consisted of 19 datasets collected from 19 pairs of participants. This research collected the multimodal process-oriented data and programming product data of student pairs through two ways. First, video recorders (with audio) were used to capture student pairs' verbal communications and facial expressions. Second, computer screen videos (with audio) were recorded to capture student pairs' behavioral operations on the platform as well as their final programming products. Each dataset included audio recordings of verbal communication data of pairs (about 475 min in total), computer screen recordings of click stream data (about 475 min in total), video recordings of facial expression data (about 475 min in total), and the final products of pair programming task data.
The analytical framework, procedures and methods
An overall analytical framework was proposed to examine the multimodal characteristics of collaborative patterns. The framework included the first step of the assessment and clustering as well as the second step of collaborative pattern analysis. In the first step of assessment and clustering, K-means clustering was conducted to detect the collaborative clusters based on student pairs' process and summative assessment. In the second step of collaborative pattern analysis, Quantitative content analysis (QCA), click stream analysis (CSA), and video analysis (VA) were used to analyze student pairs'
Assessment and clustering
First,
Table 1 The process assessment framework of collaborative quality (Meier et al., [38])
Second,
Table 2 The summative assessment framework of collaborative product
Then, K-means clustering was used to extract the similar clusters of student groups' PP based on the process and summative assessment. K-means clustering, as an unsupervised algorithm, is designed to partition two-way, two-mode data (i.e., N objects with measurements on P variables) into K classes (MacQueen, [35]; Steinley, [61]). K-means clustering was run through R package
Collaborative pattern analysis
Quantitative content analysis (QCA), click stream analysis (CSA), and video analysis (VA) were used to analyze the process data of students' PP. The computer screen recording data and video recording data (with audio) were transcribed by two researchers to record students' verbal communications, operational behaviors, facial expressions in the same time scale. During the transcription, the unit of analysis for audio recording data was the unit of a sentence spoken by a student; the unit of analysis for the operation was a clickstream behavior conducted by a student when a student moved or clicked the mouse on the platform; and the unit of analysis for facial expression was one time of facial expression when a student was speaking or operating the computer. After the transcription, 19 datasets included 10,874 units of data (Mean = 572.32; SD = 48.25). There were a total of 3604 units of verbal data (Mean = 189.68; SD = 40.08), 2,356 units of behavior data (Mean = 124.00; SD = 51.33), and 4914 units of facial data (Mean = 258.63; SD = 38.39).
Based on the previous relevant literature (Díez-Palomar et al., [13]; Pekrun et al., [49]; Rogat & Adams-Wiggins, [52]; Sun et al., [63], [64]), a coding framework was proposed to analyze the process data of PP on the
Table 3 The coding framework
Next, three analytics methods were used to reveal the
From a structural perspective, epistemic network analysis (ENA) was used to demonstrate the structure of connections among the verbal communication, operational behavior, and facial expression dimensions in different clusters. ENA can detect and represent the accumulative connections between elements in coded data in dynamic networks (Csanadi et al., [8]; Shaffer et al., [56]). In this research, ENA was conducted on all codes of three dimensions. ENA Webkit (epistemicnetwork.org) was utilized to conduct ENA analysis and its visualization (Marquart et al., [37]). Referring to threshold value used in previous research (Shaffer et al., [56]), we set the threshold of edge weight as 0.25 in ENA and showed the strong and representative connections rather than all connections, in order to clearly interpret the structural characteristics among different clusters.
From a transitional perspective, process mining (PM) was used to detect and visualize the transitional processes of the verbal communication, operational behavior, and facial expression dimensions among different collaborative clusters. PM is a temporal data mining and analysis method that focuses exclusively on transitions between events or activities (Reimann, [51]; Schoor & Bannert, [55]). The software
Results
After the process and summative assessment of student's PP, the clustering results of K-means generated based on the distribution of corresponding standard scores. With the value of K as suggested by the elbow method (K = 4) (see Fig. 2), the optimal clustering results revealed four clusters of collaborative types, consisting of 5, 5, 6, and 3 student pairs for
Graph: Fig. 2 The optimal clusters of "K" with the elbow method
Graph: Fig. 3 The K-means clustering results (K = 4)
Among the four clusters, Cluster 1 had the highest score of collaborative processes (Mean = 38.60, SD = 2.33), followed by Cluster 2 (Mean = 31.80, SD = 3.71), Cluster 3 (Mean = 19.67, SD = 2.21), and Cluster 4 (Mean = 12.33, SD = 1.89). Cluster 1 also had the highest score of collaborative products (Mean = 21.80, SD = 2.04), followed by Cluster 4 (Mean = 13.67, SD = 1.89), Cluster 2 (Mean = 11.80, SD = 0.98), and Cluster 3 (mean = 11.67, SD = 1.49). In summary, Cluster 1 had the high performance in both process and summative assessment. Cluster 2, Cluster 3, and Cluster 4 had a low-level performance in summative assessment. Cluster 2 had the relatively high performance in process assessment, while Cluster 4 had a relatively low performance in process assessment.
From a quantitative perspective
From a quantitative perspective, ANOVA with the Bonferroni correction was conducted to test the significant differences between the four collaborative clusters on the three dimensions. Levene tests were conducted before ANOVAs and the results showed the homogeneity of variance. Moreover, post-hoc pairwise comparisons were conducted to further reveal significant differences between clusters (see Table 4). Considering that some codes (i.e., NR, RP, NE) were not normally distributed, a non-parametric test was conducted to cross-check the ANOVA results. The results showed that there were significant differences in the frequency of KC, CR, and PO (
Table 4 Results of code frequencies and one-way ANOVAs of four collaborative cluster types
*
From a structural perspective
From a structural perspective, the characteristics among the four clusters were reflected by the connection values and the centroid locations of the ENA plots (see Fig. 4). For all four clusters, most of the codes shared strong connections with MO in epistemic networks. Specifically, regular characteristics among the four clusters were reflected by four pairs of connected codes (connection values > 0.40), including OE – MO, ST – MO, AC – MO, and SR – MO. Moreover, OE – MO had the strongest connections (connection values > 0.85) among all the pairs in all four clusters. NR, NE, and AP were weakly associated with other codes (connection values < 0.25) in the four clusters.
Graph: Fig. 4 The epistemic network analysis of four collaborative clusters. The threshold of edge weight in the epistemic networks was set as 0.25 to show the representative connections and structural characteristics (i.e., connection value ≥ 0.25) among the verbal communication, operational behavior, and facial expression dimensions
Different characteristics were identified among the four clusters of collaborative types, reflected by the locations of the centroid in epistemic networks (shown as red nodes in Fig. 4). In Cluster 1, the centroid of the epistemic network was located at the upper left corner, mainly focusing on PO, KC, and CR (i.e., connection value of MO-PO = 0.55, connection value of MO-KC = 0.45, connection value of MO-CR = 0.31). In Cluster 2, the centroid of the epistemic network was located at the lower left corner, mainly focusing on AG, SR, FM, and QP (i.e., connection value of MO-AG = 0.68, connection value of MO-SR = 0.64, connection value of MO-FM = 0.42, connection value of MO-QP = 0.41, connection value of AG-SR = 0.32). In Cluster 3, the centroid of the epistemic network was located at the upper right corner, mainly focusing on RP, ST, and AC (i.e., connection value of MO-RP = 0.76, connection value of MO-ST = 0.64, connection value of MO-AC = 0.50). In Cluster 4, the centroid was located at the lower right corner, mainly focusing on OE, DB, NE, and NR (i.e., connection value of MO-OE = 0.99, connection value of MO-DB = 0.47, connection value of MO-NR = 0.17, connection value of MO-NE = 0.04). In summary, Cluster 1 concentrated on positively constructing knowledge and reaching consensus; Cluster 2 concentrated on arguing, asking question, simple replying and maintaining function; Cluster 3 concentrated on self-talking, adjusting code and running program; Cluster 4 concentrated on negatively expressing opinion, responding and debugging.
From a transitional perspective
From a transitional perspective, the characteristics among the four clusters were reflected by the code transitions in the process models (see Fig. 5). The regular characteristics of four clusters began by
Graph: Fig. 5 The process mining results of four collaborative clusters. In the process models, the boxes refer to the absolute frequencies of codes and the arrows refer to the observed directional transitions from code A to code B
Different transitional characteristics were found among four clusters (see Fig. 5). In Cluster 1, student pairs were more likely to start with two paths, including SR AC MO and QP RP MO/PO. Student pairs mainly ended with FM, which indicated that they regulated to maintain the function at the end of PP. Moreover, three loops appeared frequently in Cluster 1, including MO CR AC MO, MO ST AC MO, and MO OE DB MO. These results indicated that students tended to adjust coding blocks through self-talking and reaching consensus, and debug the programs through expressing new opinions. In Cluster 2, students usually started their collaboration with OE and then divided into two paths, namely OE FM AC MO/PO and OE DB MO. Compared to other three clusters, Cluster 2 ended with more codes (i.e., AG, ST, KC, MO, PO). Two loops often appeared during the PP processes, including MO KC (SR QP) AC MO and MO KC AC PO AG DB MO. These results indicated that students were more likely to constructed knowledge to drive the coding behaviors, but usually argued with each other when debugging programs. In Cluster 3, students had high probability to start their collaboration with QP, then moved to the path of NE OE DB or directly moved to AC. They mainly ended with FM, which also indicated that they regulated to maintain the function of pairs in the end. Two loops usually appeared in Cluster 3, including MO NR MO and MO CR DB MO. These results indicated that students sometimes replied to the peer negatively and sometimes reached a consensus to debug programs. In Cluster 4, the code transitions and loops started with MO and ended with ST, AC and DB. Specifically, a loop of MO OE DB AG MO appeared most frequently among all loops, which indicated that they expressed opinions to debug and solve problems but usually argued with each other. In addition, the loops of MO KC MO, MO NR MO and MO FM MO sometimes appeared, which also implied that pairs not only made regulations, but also had negative interactions when constructing knowledge.
Discussions and implications
This research applied MMLA to examine students' collaborative patterns in a face-to-face, computer-supported PP environment in higher education. Specifically, we collected students' multimodal process-oriented data and programming products data, and proposed an analytical framework integrating MMLA methods to detect and examine student pairs' collaborative patterns. Based on the process and summative assessment results, four clusters were detected from 19 pairs through K-means clustering, namely
From a theoretical perspective, this research contributed to the extant literature on CPS through revealing how complex connections among multimodality emerged into different collaborative patterns which in turn influenced the collaborative quality of final products. First, regarding the highly-performed collaborative pattern (i.e., Cluster 1), we found that opinion expression after a series of operations and trials could form a foundation for deep-level knowledge construction and group regulation to achieve high-quality collaboration (Ouyang & Chang, [43]; Park et al., [48]). Moreover, compared to negative emotions (i.e., Cluster 3, 4), students' positive emotions might contribute to the high quality of collaboration, like Cluster 1 did (Törmänen et al., [65]). Furthermore, consensus reaching in argumentation is also the key to achieve a high-quality of collaboration (Straus, [62]). Second, previous research verified that argumentation contributed to CPS through cognitive elaboration and knowledge construction (Stegmann et al., [60]), but constant argumentation without peers' consensus might result in divergence of opinions and inefficient collaboration (i.e., Cluster 2). Third, inconsistent with previous research that highlighted the role of self-talk in promoting self-regulation in CPS (DiDonato, [12]), the frequent use of self-talks (i.e., Cluster 3) might result in too much individual-oriented opinion expression and less group negotiation, which may in turn lead to the failure of collaboration. Students in Cluster 3 also spent most of the time on debugging, which somehow indicated that they encountered difficulties without successful programming in collaboration (Klahr & Carver, [29]). Fourth, compared to Cluster 3, students in Cluster 4 tended to express opinions together and appeared more programming running behaviors during debugging to achieve a relatively higher summative performance. Hence, running programming and debugging could together reflect students' persistence and productive struggle in PP that help them learn from failures (Kapur, [24]; Kim et al., [28]).
From a pedagogical perspective, instructors should concentrate on the collaborative process and provide appropriate scaffoldings and interventions to support a high quality of collaborative programming. First, instructors should provide scaffoldings to enhance student pairs' collaboration quality based on the characteristics of collaborative patterns. For example, students in Cluster 3 were more likely to be individual-orientated rather than group-orientated, which led to the low performance in both process and summative assessment; therefore, instructors can regulate their collaboration through some metacognitive scaffoldings (e.g., planning group's goal) and socio-emotional scaffoldings (e.g., encouraging students to collaborate) to achieve group cohesion within student pairs (Molenaar et al., [39]; Ouyang et al., [44]). In addition, students in Cluster 3 and Cluster 4 had constant debugging and frequent errors, which might indicate that they were not familiar with the programming skills; therefore, cognitive scaffoldings (e.g., task-relevant information or hint) can be provided to help them solve the problems in programming (Ouyang & Xu, [47]; Zhong & Si, [73]). Second, most of the students mainly expressed moderate emotions rather than positive emotions during the PP processes. However, positive social emotion plays an important role to motivate learning interest, lessen tension, and improve social cohesion in collaboration (Rogat & Adams-Wiggins, [52]), such as how Cluster 1 performed in this research. Hence, the engagement of instructors as social supporters during students' collaborative programming, might mobilize the collaborative atmosphere to reach the goal of high-quality PP (Ouyang & Scharber, [46]; Ouyang & Xu, [47]). Third, since constant argumentation and opinion divergence are the critical factors that resulted in low-quality PP (e.g., Cluster 2), instructors are supposed to pay attention to the conflicting moments in argumentation and make appropriate interventions (e.g., easing the atmosphere, providing new ideas) to guide the co-construction of knowledge and problem-solving (Barron, [1]). Overall, instructors should be aware of student pair' collaborative patterns as well as the complex characteristics, and support their work appropriately with varied scaffoldings.
From an analytical perspective, since CPS is a complex and adaptive phenomenon (Stahl & Hakkarainen, [59]), multimodal data collection and learning analytics are suggested for future works to explore the complex problems and phenomena in CPS (Jacobson et al., [23]; Ouyang et al., [45]). Compared to traditional performance evaluation (e.g., test score, product data) and self-report data (e.g., questionnaire, interview), process-based multimodal data and learning analytics methods provides us a holistic, complementary, fine-grained perspective to understand the complex nature of CPS (Hilpert & Marchand, [19]; Kapur, [25]). Recently, many research has used multimodal data (e.g., speed rate, gesture, body movement, eye movement) as well as learning analytics methods to examine the complex, synergistic, and dynamic collaborative patterns and characteristics in CPS (e.g., Mu et al., [40]; Ouyang et al., [45]; Wiltshire et al., [69]). Echoing this trend, this research collected student pairs' multimodal data (i.e., verbal audios, computer screen recordings, facial expression recordings, final products data) and applied multiple learning analytics methods (e.g., content analysis, epistemic network analysis, process mining) to investigate the collaborative patterns in PP as well as their quantitative, structural, and transitional characteristics. Furthermore, advanced and automated artificial intelligence (AI) algorithms (e.g., hidden Markov model, natural language processing, recurrence quantification analysis) are advised to analyze the complexity and dynamics of collaboration in the future research (Gorman et al., [16]; Hoppe et al., [21]). Compared to traditional learning analytics methods, AI-driven methods have potential to analyze multimodal and nonlinear data and extract the complex and dynamic structure of CPS (de Carvalho & Zárate, [10]). Overall, due to the complexity of CPS, it is critical to capture the fine-grained process data and utilize multimodal learning analytics to reveal the collaborative patterns as well as their implicit characteristics (Kapur, [25]; Reimann, [51]).
Conclusions, limitations, and future directions
Since it is challenging for novice programmers to succeed in collaborative programming, it is necessary to investigate how their multimodality can form different collaborative patterns and how different patterns contribute to the quality of collaborative programming. Using MMLA, the current research collected and analyzed multimodal data to understand the collaborative patterns during student pairs' PP in higher education. The results detected four collaborative patterns associated with different levels of process and summative performances. Based on these findings, the current research proposed theoretical, pedagogical, and analytical implications to guide future practice and research. There are two limitations in the current research, which lead to future research directions. First, since the current study aimed to explore collaborative clusters and patterns, the research design may lead to a threat to validity (Drost, [15]; Humphry & Heldsinger, [22]), which should be addressed in future research. For example, regarding the internal validity, we did not control the gender distribution of student pairs, which might partially influence the collaborative processes. In addition, although participants did not have prior programming foundations or experiences, no pre-test was set to measure and control students' prior programming knowledge. Moreover, the difficulty of the programming tasks may also have impacts on student collaboration. Regarding the external validity, the sample size of student pairs had a limited range of demographic backgrounds. Therefore, future CPS research is supposed to strictly control internal validity (e.g., gender, prior knowledge, task) and expand the sample size and pair structure and arrangement to test, validate, or modify the implications. Second, this MMLA research merely collected students' discourse, online behaviors, and facial expression from video data to analyze the CPS processes, and there is a lack of other multimodal data, such as physiological and psychological data. In addition, the facial expressions were coded manually rather than automated identification based on software, which might reduce the data analysis efficiency and accuracy. Therefore, AI-driven data collection and analysis methods as well as more modalities of data (e.g., physiological, eye tracking data) can provide further insights into CPS research. Overall, it is valuable to examine different collaborative patterns of novice programmers through MMLA, in order to tease out fine-grained and complex features, which serves as a data-driven evidence for promoting the quality of computer programming in higher education.
Acknowledgements
The authors would like to thank students who participated in this research.
Author contributions
WX designed and conducted data analysis, and wrote the manuscript draft; YW facilitated research design, collected and coded the data; and FO designed and supervised the research and revised the manuscript. All authors read and approved the final manuscript.
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was supported by National Natural Science Foundation of China (62177041); Zhejiang Province educational science and planning research Project (2022SCG256); Zhejiang University graduate education research Project (20220310).
Availability of data and materials
The data was available upon request from the first author.
Declarations
Competing interests
The authors declare that they have no competing interests.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
1 Barron B. Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences. 2000; 9; 4: 403-436. 10.1207/S15327809JLS0904_2
2 Beck L, Chizhik A. Cooperative learning instructional methods for CS1: Design, implementation, and evaluation. ACM Transactions on Computing Education. 2013; 13; 3: 10-32. 10.1145/2492686
3 Borge M, Mercier E. Towards a micro-ecological approach to CSCL. International Journal of Computer-Supported Collaborative Learning. 2019; 14; 2: 219-235. 10.1007/s11412-019-09301-6
4 Brown JS, Collins A, Duguid P. Situated cognition and the culture of learning. Educational Researcher. 1989; 18; 1: 32-42. 10.3102/0013189X018001032
5 Bryant S, Romero P, Du Boulay BAbrahamsson P, Marchesi M, Succi G. The collaborative nature of pair programming. Extreme programming and agile processes in software engineering. 2006; Springer: 53-64. 10.1007/11774129_6
6 Byrne D, Callaghan G. Complexity theory and the social sciences. 2014; Routledge
7 Chittum JR, Jones BD, Akalin S, Schram ÁB. The effects of an afterschool STEM program on students' motivation and engagement. International Journal of STEM Education. 2017; 4; 1: 11-26. 10.1186/s40594-017-0065-4
8 Csanadi A, Eagan B, Kollar I, Shaffer DW, Fischer F. When coding-and-counting is not enough: Using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning. 2018; 13; 4: 419-438. 10.1007/s11412-018-9292-z
9 Damon W, Phelps E. Critical distinctions among three approaches to peer education. International Journal of Educational Research. 1989; 13: 9-19. 10.1016/0883-0355(89)90013-X
de Carvalho WF, Zárate LE. A new local causal learning algorithm applied in learning analytics. The International Journal of Information and Learning Technology. 2020; 38; 1: 103-115. 10.1108/IJILT-04-2020-0046
Denner J, Green E, Campe S. Learning to program in middle school: How pair programming helps and hinders intrepid exploration. Journal of the Learning Sciences. 2021; 30; 4–5: 611-645. 10.1080/10508406.2021.1939028
DiDonato NC. Effective self-and co-regulation in collaborative learning groups: An analysis of how students regulate problem solving of authentic interdisciplinary tasks. Instructional Science. 2013; 41; 1: 25-47. 10.1007/s11251-012-9206-9
Díez-Palomar J, Chan MCE, Clarke D, Padrós M. How does dialogical talk promote student learning during small group work? An exploratory study. Learning, Culture and Social Interaction. 2021; 30: 100540. 10.1016/J.LCSI.2021.100540
Dillenbourg PDillenbourg P. What do you mean by collaborative learning?. Collaborative-learning: Cognitive and computational approaches. 1999; Elsevier: 1-19
Drost EA. Validity and reliability in social science research. Education Research and Perspectives. 2011; 38; 1: 105-123. 10.3316/informit.491551710186460
Gorman JC, Grimm DA, Stevens RH, Galloway T, Willemsen-Dunlap AM, Halpin DJ. Measuring real-time team cognition during team training. Human Factors. 2020; 62; 5: 825-860. 10.1177/0018720819852791
Han F, Ellis RA. Patterns of student collaborative learning in blended course designs based on their learning orientations: A student approaches to learning perspective. International Journal of Educational Technology in Higher Education. 2021; 18; 1: 1-16. 10.1186/s41239-021-00303-9
Hawlitschek A, Berndt S, Schulz S. Empirical research on pair programming in higher education: A literature review. Computer Science Education. 2022. 10.1080/08993408.2022.2039504
Hilpert JC, Marchand GC. Complex systems research in educational psychology: Aligning theory and method. Educational Psychologist. 2018; 53; 3: 185-202. 10.1080/00461520.2018.1469411
Hmelo-Silver CE, DeSimone CHmelo-Silver C, Chinn CA, Chan C, O'Donnell A. Problem-based learning: An instructional model of collaborative learning. The international handbook of collaborative learning. 2013; Routledge. 10.4324/9780203837290
Hoppe HU, Doberstein D, Hecking T. Using sequence analysis to determine the well-functioning of small groups in large online courses. International Journal of Artificial Intelligence in Education. 2021; 31: 680-699. 10.1007/s40593-020-00229-9
Humphry SM, Heldsinger SA. Common structural design features of rubrics may represent a threat to validity. Educational Researcher. 2014; 43; 5: 253-263. 10.3102/0013189X1454215
Jacobson MJ, Kapur M, Reimann P. Conceptualizing debates in learning and educational research: Toward a complex systems conceptual framework of learning. Educational Psychologist. 2016; 51; 2: 210-218. 10.1080/00461520.2016.1166963
Kapur M. Productive failure. Cognition and Instruction. 2008; 26; 3: 379-424. 10.1080/07370000802212669
Kapur M. Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. International Journal of Computer-Supported Collaborative Learning. 2011; 6; 1: 39-56. 10.1007/s11412-011-9109-9
Kassambara, A, & Mundt, F. (2017). Package 'factoextra'. Extract and visualize the results of multivariate data analyses. [Software]. R Package Version 1.0.7.
Kawamura R, Shirai S, Takemura N, Alizadeh M, Cukurova M, Takemura H, Nagahara H. Detecting drowsy learners at the wheel of e-learning platforms with multimodal learning analytics. IEEE Access. 2021; 9: 115165-115174. 10.1109/ACCESS.2021.3104805
Kim C, Vasconcelos L, Belland BR, Umutlu D, Gleasman C. Debugging behaviors of early childhood teacher candidates with or without scaffolding. International Journal of Educational Technology in Higher Education. 2022; 19; 1: 1-26. 10.1186/s41239-022-00319-9
Klahr D, Carver SM. Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology. 1988; 20; 3: 362-404. 10.1016/0010-0285(88)90004-7
Kodinariya TM, Makwana PR. Review on determining number of cluster in K-Means clustering. International Journal. 2013; 1; 6: 90-95
Krippendorff K. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research. 2004; 30; 3: 411-433. 10.1093/hcr/30.3.411
Kwon K, Liu YH, Johnson LP. Group regulation and social-emotional interactions observed in computer supported collaborative learning: Comparison between good vs. poor collaborators. Computers & Education. 2014; 78: 185-200. 10.1016/j.compedu.2014.06.004
Lin PC, Hou HT, Wu SY, Chang KE. Exploring college students' cognitive processing patterns during a collaborative problem-solving teaching activity integrating Facebook discussion and simulation tools. The Internet and Higher Education. 2014; 22: 51-56. 10.1016/j.iheduc.2014.05.001
Lu J, Churchill D. Using social networking environments to support collaborative learning in a Chinese university class: Interaction pattern and influencing factors. Australasian Journal of Educational Technology. 2014; 30; 4: 1-15. 10.14742/ajet.655
MacQueen JLe Cam LM, Neyman J. Some methods for classification and analysis of multivariate observations. Proceedings of the Fifth Berkeley symposium on mathematical statistics and probability. 1967; University of California Press: 281-297
Malmberg J, Järvelä S, Järvenoja H. Capturing temporal and sequential patterns of self-, co-, and socially shared regulation in the context of collaborative learning. Contemporary Educational Psychology. 2017; 49: 160-174. 10.1016/j.cedpsych.2017.01.009
Marquart, C. L, Hinojosa, C, Swiecki, Z, Eagan, B, & Shaffer, D. W. (2018). Epistemic network analysis [Software]. Version 1.6.0. https://epistemicnetwork.org.
Meier A, Spada H, Rummel N. A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Learning. 2007; 2: 63-86. 10.1007/s11412-006-9005-x
Molenaar I, Sleegers P, van Boxtel C. Metacognitive scaffolding during collaborative learning: A promising combination. Metacognition and Learning. 2014; 9; 3: 309-332. 10.1007/s11409-014-9118-y
Mu S, Cui M, Huang X. Multimodal data fusion in learning analytics: A systematic review. Sensors. 2020; 20; 23: 6856. 10.3390/s20236856
O'Donnell AM, Hmelo-Silver CEHmelo-Silver CE, Chinn CA, Chan CKK, O'Donnell AM. Introduction: What is collaborative learning? An overview. The international handbook of collaborative learning. 2013; Routledge: 93-111
Ochoa X, Worsley M. Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics. 2016; 3; 2: 213-219. 10.18608/jla.2016.32.10
Ouyang F, Chang YH. The relationships between social participatory roles and cognitive engagement levels in online discussions. British Journal of Educational Technology. 2019; 50; 3: 1396-1414. 10.1111/bjet.12647
Ouyang F, Chen Z, Cheng M, Tang Z, Su C-Y. Exploring the effect of three scaffoldings on the collaborative problem-solving processes in China's higher education. International Journal of Educational Technology in Higher Education. 2021; 18; 35: 1-22. 10.1186/s41239-021-00273-y
Ouyang F, Dai X, Chen S. Applying multimodal learning analytics to examine the immediate and delayed effects of instructor scaffoldings on small groups' collaborative programming. International Journal of STEM Education. 2022; 9; 1: 1-21. 10.1186/s40594-022-00361-z
Ouyang F, Scharber C. The influences of an experienced instructor's discussion design and facilitation on an online learning community development: A social network analysis study. The Internet and Higher Education. 2017; 35: 34-47. 10.1016/j.iheduc.2017.07.002
Ouyang F, Xu W. The effects of three instructor participatory roles on a small group's collaborative concept mapping. Journal of Educational Computing Research. 2022; 60; 4: 930-959. 10.1177/07356331211057283
Park JBH, Schallert DL, Sanders AJZ, Williams KM, Seo E, Yu LT, Vogler JS, Song K, Williamson ZH, Knox MC. Does it matter if the teacher is there? A teacher's contribution to emerging patterns of interactions in online classroom discussions. Computers and Education. 2015; 82: 315-328. 10.1016/j.compedu.2014.11.019
Pekrun R, Goetz T, Titz W, Perry RP. Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist. 2002; 37; 2: 91-106. 10.1207/S15326985EP3702
Perera D, Kay J, Koprinska I, Yacef K, Zaïane, O. R. Clustering and sequential pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and Data Engineering. 2009; 21; 6: 759-772. 10.1109/TKDE.2008.138
Reimann P. Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning. 2009; 4; 3: 239-257. 10.1007/S11412-009-9070-Z/FIGURES/4
Rogat TK, Adams-Wiggins KR. Interrelation between regulatory and socioemotional processes within collaborative groups characterized by facilitative and directive other-regulation. Computers in Human Behavior. 2015; 52: 589-600. 10.1016/j.chb.2015.01.026
Romero M, Lepage A, Lille B. Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education. 2017; 14; 42: 1-15. 10.1186/s41239-017-0080-z
Rozinat, A, & Günther, C. W. (2012). Disco [Software]. Version 3.1.4. https://fluxicon.com/disco/
Schoor C, Bannert M. Exploring regulatory processes during a computer-supported collaborative learning task using process mining. Computers in Human Behavior. 2012; 28; 4: 1321-1331. 10.1016/J.CHB.2012.02.016
Shaffer DW, Collier W, Ruis AR. A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics. 2016; 3; 3: 9-45. 10.18608/jla.2016.33.3
Stahl G. Studying virtual math teams. 2009; Springer. 10.1007/978-1-4419-0228-3. 1178.00016
Stahl G. Group practices: A new way of viewing CSCL. International Journal of Computer-Supported Collaborative Learning. 2017; 12; 1: 113-126. 10.1007/s11412-017-9251-0
Stahl G, Hakkarainen KCress U, Rosé C, Wise AF, Oshima J. Theories of CSCL. International handbook of computer-supported collaborative learning. 2021; Springer: 23-44. 10.1007/978-3-030-65291-3_2
Stegmann K, Wecker C, Weinberger A, Fischer F. Collaborative argumentation and cognitive elaboration in a computer-supported collaborative learning environment. Instructional Science. 2012; 40; 2: 297-323. 10.1007/s11251-011-9174-5
Steinley D. K-means clustering: A half-century synthesis. British Journal of Mathematical and Statistical Psychology. 2006; 59; 1: 1-34. 2242281. 10.1348/000711005X48266
Straus D. How to make collaboration work: Powerful ways to build consensus, solve problems, and make decisions. 2002; Berrett-Koehler Publishers
Sun D, Ouyang F, Li Y, Chen H. Three contrasting pairs' collaborative programming processes in China's secondary education. Journal of Educational Computing Research. 2020; 59; 4: 740-762. 10.1177/0735633120973430
Sun D, Ouyang F, Li Y, Zhu C. Comparing learners' knowledge, behaviors, and attitudes between two instructional modes of computer programming in secondary education. International Journal of STEM Education. 2021; 8: 54. 10.1186/s40594-021-00311-1
Törmänen T, Järvenoja H, Mänty K. All for one and one for all—How are students' affective states and group-level emotion regulation interconnected in collaborative learning?. International Journal of Educational Research. 2021; 109: 101861. 10.1016/j.ijer.2021.101861
Vygotsky LS. Mind in society: The development of higher psychological processes. 1978; Springer
Wang L, Geng F, Hao X, Shi D, Wang T, Li Y. Measuring coding ability in young children: relations to computational thinking, creative thinking, and working memory. Current Psychology. 2021. 10.1007/s12144-021-02085-9
Webb NM, Ing M, Burnheimer E, Johnson NC, Franke ML, Zimmerman J. Is there a right way? Productive patterns of interaction during collaborative problem solving. Education Sciences. 2021; 11; 5: 214. 10.3390/educsci11050214
Wiltshire TJ, Steffensen SV, Fiore SM. Multiscale movement coordination dynamics in collaborative team problem solving. Applied Ergonomics. 2019; 79: 143-151. 10.1016/j.apergo.2018.07.007
Xu W, Geng F, Wang L. Relations of computational thinking to reasoning ability and creative thinking in young children: Mediating role of arithmetic fluency. Thinking Skills and Creativity. 2022; 44: 101041. 10.1016/j.tsc.2022.101041
Zemel A, Koschmann T. Recalibrating reference within a dual-space interaction environment. International Journal of Computer-Supported Collaborative Learning. 2013; 8; 1: 65-87. 10.1007/s11412-013-9164-5
Zheng L, Zhen Y, Niu J, Zhong L. An exploratory study on fade-in versus fade-out scaffolding for novice programmers in online collaborative programming settings. Journal of Computing in Higher Education. 2022; 34: 489-516. 10.1007/s12528-021-09307-w
Zhong B, Si Q. Troubleshooting to learn via scaffolds: Effect on students' ability and cognitive load in a robotics course. Journal of Educational Computing Research. 2021; 59; 1: 95-118. 10.1177/0735633120951871
By Weiqi Xu; Yajuan Wu and Fan Ouyang
Reported by Author; Author; Author