Treffer: Mapping Self-Regulated Learning Events and Actions in Online Teacher Professional Development with Process Mining Techniques

This study aimed to visualize self-regulated learning (SRL) behaviors performed by users from an online teacher professional development platform called the EdHub Library using the pm4py algorithm in Python to parse event data during the first 30 days of the school year and the first 90 days of the COVID-19 pandemic in March 2020. Process mining techniques were implemented to visualize the user profiles of frequent users in case-based decompositions and identify critical SRL events and actions of all users in activity-based decompositions for three school years (2018, 2019, and 2020). SRL events and actions were measured for all pageviews.

As Provided

AN0169743333;i4g01oct.22;2023Aug05.12:07;v2.2.500

MAPPING SELF-REGULATED LEARNING EVENTS AND ACTIONS IN ONLINE TEACHER PROFESSIONAL DEVELOPMENT WITH PROCESS MINING TECHNIQUES 

<sbt id="AN0169743333-2">INTRODUCTION</sbt>

This study aimed to visualize self-regulated learning (SRL) behaviors performed by users from an online teacher professional development platform called the EdHub Library using the pm4py algorithm in Python to parse event data during the first 30 days of the school year and the first 90 days of the COVID-19 pandemic in March 2020. Process mining techniques were implemented to visualize the user profiles of frequent users in case-based decompositions and identify critical SRL events and actions of all users in activity-based decompositions for three school years (2018, 2019, and 2020). SRL events and actions were measured for all pageviews.

At the beginning of the COVID-19 pandemic, educators' in-person professional development (PD), school research, and teacher evaluations were severely disrupted (Sparks, 2020). Given the constraints imposed by COVID-19, online teacher PD helps address educators' needs by offering synchronous and asynchronous opportunities that support educators' pedagogical practice and collaboration efforts (Hartshorne et al., 2020). While evaluations of online teacher PD platforms exist in the literature, educators have turned their attention to free online resources to support remote teaching and PD since the beginning of the pandemic in March 2020 (Cavanaugh et al., 2020). Educators apply self-regulated learning (SRL) behaviors to browse materials that meet their PD needs and objectives in online environments. SRL comprises a wide range of cognitive, metacognitive, motivational, emotional, and behavioral competencies that allow individuals to monitor and regulate their cognition, motivation, and behavior concerning their learning goals and contexts (Greene, 2018). The literature shows that SRL skills are best exhibited in learners' choices and learning tactics during their learning experience (Fan et al., 2021).

The EdHub Library: Online Teacher PD

The EdHub Library has been part of the Network of Educator Effectiveness (NEE) in Missouri since the fall of 2014. NEE provides teacher evaluation solutions and training in school districts. School districts in NEE have access to over 500 asynchronous online PD resources organized by teacher standards. NEE school districts have access to a video library of best practices in classroom teaching, examples of scoring classroom observations, a catalog of self-paced online modules, copyrighted assessment instruments, journal reflection activities, and annual calibration training sessions for school administrators (Leung, 2021).

The EdHub Library is embedded within existing classroom observation data collection and reporting tools. EdHub provides PD materials that target different users, including principals, teachers, and instructional staff. The EdHub Library underwent a significant interface redesign in 2017, providing educators with improved mechanisms for searching and browsing resources by teacher standards (Leung, 2021). Before the interface redesign, users were required to navigate to the instructional modules at five levels sequentially. In the first generation of the platform, users could not align the content with teacher standards as they navigated to deeper sections of the library.

The redesign effort involved implementing a three-level hierarchical structure that allowed for the straightforward navigation and alignment of PD content with teacher standards. The EdHub homepage represents the firstlevel page, which includes topic categories and their alignment with the standards. As users navigate from the homepage to a topic, the second- level pages contain all the modules within a subject with a corresponding alignment with standards. As users select a module from the topic listing on the second-level page, the third-level pages correspond to modules that outline the description of the material, learning objectives, several self-paced activities, and external resources. Breadcrumb navigation is consistently provided on second- and third-level pages to allow users to return to the EdHub homepage. Users must return to the EdHub homepage to look for content based on standards.

In the second generation of the platform, educators can search for PD materials through the homepage directory, dedicated teacher standard site maps, and the search engine, as shown in Figure 1. Regardless of how users look for materials, content alignment with teacher standards is evident throughout the library's information architecture. For example, the homepage outlines the topic titles, summaries, and teacher indicators. Dedicated teacher standard site maps allow users to drill down on specific modules related to a particular standard or topic. The site maps also enable users to access modules directly without accessing the topic pages from the homepage. The search engine component organizes the module results in topic groups with previews of teacher standards.

Theoretical Framework

Although Zimmerman (1989, 2000) was one of the first researchers to develop three models of SRL, including the triadic analysis, cyclical phases, and multilevel SRL, Pintrich et al. (1993) argued that SRL models lacked connections between motivation and cognition to account for a significant number of SRL processes. In Pintrich's (2000) model, SRL consists of four phases of regulation: (1) fore-thought, planning, and activation, (2) monitoring, (3) control, and (4) reaction and reflection. The four phases generally represent the sequence in which learners perform a task but do not always represent linear or hierarchically structured phases as learners progress through the task (Pintrich, 2004). As learners go through the four phases, they attempt to regulate themselves in four areas: (1) cognition, (2) motivation, (3) behavior, and (4) context. The first three areas of cognition, motivation, and behavior represent the three areas of psychological functioning (Snow et al., 1996). In contrast, the context area considers the task's social context, conditions, and perceptions as learners attempt to regulate their psychological functions.

According to a review of SRL models by Panadero (2017), Pintrich's SRL model is the only comprehensive model describing learners' attempts to monitor, control, and regulate their learning contexts and overt behaviors. In the case of the EdHub Library, Pintrich's (2000) SRL model covers the motivation and behavioral areas of regulation for different users (i.e., principals, teachers, and instructional staff) accessing PD materials. For example, principals may be motivated to access specific library areas related to classroom observation training and professional development materials. Unlike principals, teachers are not required to access particular materials targeted at administrators. However, they may be motivated to access content related to teaching practices (e.g., classroom management) or subject-specific content (e.g., a unit of instruction examples). Regardless of user roles and motivation, analyzing temporal data with process mining techniques addresses the control phase in Pintrich's SRL model concerning how users select and adapt cognitive strategies while navigating PD materials of their interest.

Problem Statement

Dede et al. (2009) stated that effective online teacher professional development (oTPD) programming could be evaluated in terms of (1) program design, (2) program effectiveness, (3) technical design, and (4) learner interactions. The program design aspect examines the delivery mechanisms for accessing the PD curriculum. The program effectiveness aspect evaluates the effectiveness of the intended short-term outcomes of the oTPD against teachers' self-reported data. The technical design aspect evaluates the effects of communication tools on teachers' short-term goals. The learner interactions aspect considers teachers' quality and level of participation using communication and collaboration tools.

The present studies that evaluated learner interactions in oTPD relied on self-reported data but failed to capture the temporal characteristics of educators' SRL activities. Temporal or trace data better measure educators' cognition and metacognition than incomplete or biased self-reports (Fan et al., 2021). The characteristics of temporal data refer to the passage of time and the temporal sequence of events and actions educators perform during online PD sessions. This particular study examines the learner interactions aspect of oTPD by extracting the temporal data of educators who have 24/7 access to self-paced online PD materials during specific times of the school year (i.e., preparing for the new school year) and in emergency circumstances (i.e., the beginning of the COVID-19 pandemic).

Purpose and Significance of the Study

This study aims to visualize SRL events and actions performed by users during essential times of the school year (e.g., the first 30 days of the school year) and critical times from the start of the COVID-19 pandemic in March 2020 using a prominent online teacher PD platform. Establishing these SRL activities is essential for understanding educators' priorities and information needs at the beginning of the school year and during the pandemic. This study contributes to the literature by examining educator interactions and PD resources.

Regarding process mining, case- and activity- based decomposition techniques allow mapping SRL events and actions of specific users with significant page views and all users from the event log data, respectively. First, case-based decompositions were based on four frequent users who navigated the platform during the first 30 days of three school years (2018, 2019, and 2020) and 90 days at the pandemic's beginning. Second, activity-based decompositions were based on all users' SRL events and actions during the first 90 days of the COVID-19 pandemic in March 2020. This study explored the following research questions:

RQ1: What are the most accessed resources for 2018, 2019, and 2020 academic years and 90 days into the COVID-19 pandemic?

RQ2: What are the frequent library users' SRL events and actions in the first 30 days of 2018, 2019, and 2020 academic years and 90 days into the COVID-19 pandemic?

RQ3: What are all users' SRL events and actions during the first 90 days of the COVID- 19 pandemic?

The remainder of this paper is organized as follows. The literature review section provides the related concepts and methodologies for measuring SRL. The methodology section describes the data collection and process mining procedures used to visualize the follow-through diagrams of users' event-based data. The findings section summarizes the study's results organized by the research question. The discussion section describes the interpretation of the findings, implications, limitations, and future directions of the present study. Finally, the conclusion section concludes the article and highlights its contributions.

LITERATURE REVIEW

Zimmerman (1989) was one of the first researchers to develop SRL models to explain SRL behaviors. SRL is defined by Zimmer-man (2000) as learners' beliefs about their capability to engage in proper actions, thoughts, feelings, and behaviors to pursue worthwhile academic goals while self-monitoring and self-reflecting on their progress toward goal completion.

Elements, Development, and Measurement of SRL

Zimmerman (2000) stated that the SRL process occurs in three stages: (1) forethought, (2) volition control, and (3) self-reflection. In the first stage of forethought, learners prepare for their work before performing their studying. In the second stage of volition control, learners control their attention and willingness during the learning process. In the third stage of self-reflection, learners assess their performance and learning strategies to achieve learning outcomes. Furthermore, learners exhibited four levels as they progressed toward SRL: (1) observation, (2) emulation, (3) self-control, and (4) self-regulation. At the first level, learners rely on observational learning as a starting point by observing others modeling the desired behavior to perform a particular task. At the second level, emulation involves learners' attempts to model the desired behavior in their way with some guidance. At the third level, learners internalize their experiences at the emulation level and attempt to perform tasks independently. At the fourth level, learners can adapt their behavior independently of the mentor by using their experience to guide their behavior.

Many previous studies have focused on the use of self-report instruments to measure SRL in four ways: (1) questionnaires, (2) interviews, (3) think-aloud protocols, and (4) learning diaries (ElSayed et al., 2019). Such measurements depend on the learners' memory retrieval capacities, perceptions, and viewpoints. Although these measurements are intended to assess SRL as aptitude learners possess, they do not track their ability to modify their behavior during the learning experience. Regarding the questionnaires used in measuring SRL, the instruments measure the generalized ability to self-regulate, types of learning strategies, and academic motivation. In Panadero's (2017) review of SRL models, five validated instruments were developed to measure SRL behaviors: (1) the Academic Self-Regulation Scale (A-SRL), (2) Learning and Study Strategies Inventory (LASSI), (3) Confidence and Doubt Scale, Online Motivation Questionnaire (OMQ), (4) Online Self- Regulated Learning Questionnaire (OSLQ), and (5) Motivated Strategies for Learning Questionnaire (MSLQ).

Measurement of SRL Using Learning Analytics and Educational Data Mining

Instead of depending on learners' perceptions and assessing SRL as an aptitude, the measurement of SRL has shifted toward assessing SRL events as time-based or task-related, accurately tracking learners' SRL behaviors in contextualized online environments. ElSayed et al. (2019) and Araka et al. (2019) investigated the types of data, frequently measured SRL components and behaviors, techniques for measuring SRL in online environments, and interventions to support SRL scaffolds. First, seven data types were used to measure the SRL obtained from online settings, devices, and human subjects. The data used to measure SRL included log data, assessment results, physiological data, demographic information, chat and forum conversations, self-reported data from instruments, and video recordings. Second, ElSayed et al. (2019) identified nine frequently measured SRL categories based on goal setting, reviewing records, emotion regulation, learning strategies, self-evaluation, seeking information, organizing and transforming, environmental structuring, and rehearsing memorizing. Third, SRL studies have implemented learning analytics and educational data mining techniques for clustering, classification, and temporal data mining. Less frequent types of research used in SRL include social network analysis, burst detection, anomaly detection, summarization, and principal component analysis. Fourth, SRL interventions aim to support students' cognition, behaviors, and motivation in dashboard visualizations, software agents, and learner feedback (Araka et al., 2019). Learning analytics techniques were used to enhance online activities by providing insights to learners in learning management systems (LMSs), massive open online courses (MOOCs), and personal learning environments (PLEs). Using software agents in online environments offers learners assistive guidelines to support SRL development when engaging in tasks. Web-enabled prompts were the last intervention to enhance learners' SRL abilities by providing instructors with learner feedback to assess the use of SRL strategies.

In MOOCs, Maldonado-Mahauad et al. (2018) applied process mining techniques to discover different learning strategies and three kinds of students based on the interaction sequence patterns that students performed. The first type was the sampling learner with the lowest performance in video lectures and the fewest attempts to solve assessments. The sampling learner initially started a course with low activity and explored different course materials. The second type of student was the targeting learner, with higher engagement rates with assessments than the sampling learner but with similar engagement rates with video lectures. The targeting learner was more strategic or goal-oriented, and efforts were devoted to increasing assessment performance. The third type was the comprehensive learner, who considered more SRL because of the high frequency of SRL behaviors exhibited during a course session. The comprehensive learner was characterized by following the recommended course activity sequence and investing more time in the video lectures. Unlike previous learners, the comprehensive learner had the highest observed interaction sequence patterns.

Greene and Azevedo (2009) codified macro and microlevel SRL strategies for high and middle school students using think-aloud protocols to capture their behaviors in a technology- enabled environment. Azevedo et al. (2008) described four macrolevel SRL processes associated with students' success while engaging in the tasks. The four macrolevel SRL processes are (1) planning, (2) monitoring, (3) strategy use, and (4) interest. Students choose microlevel SRL processes associated with macrolevel SRL processes to complete the task. The four microlevel SRL processes in the first macrolevel SRL planning process are planning, goals, prior knowledge activation, and recycling goal in working memory. In the second macrolevel SRL process of monitoring, the seven microlevel SRL processes are judgment of learning, feeling of knowing, self-questioning, content evaluation, identifying adequacy of information, monitoring progress toward goals, and monitoring the use of strategies. In the third macrolevel SRL process of strategy use, 18 microlevel SRL processes include the selection of informational sources, coordination, reading new paragraphs, reviewing notes, memorization, free search in a hypermedia environment, goal-directed search, summarization, taking notes, drawing a diagram, re-reading, making inferences, hypothesizing, knowledge elaboration, mnemonic, evaluating content as an answer to goal, finding the location in the environment, and skipping to the next objective. In the fourth macrolevel SRL process of interest, the interest statement is a microlevel SRL process where the learner expresses interest in the task or content domain.

METHODOLOGY

A process is a collection of activities users execute to accomplish a goal (Berti et al., 2019). Process mining is a technique that converts event-log data into insights and actions. In the context of the study, process mining provides inspectable learning events that users perform in the first 30 days of the beginning of the school year and within 90 days of the onset of the pandemic. The methodology section is divided into three parts: (1) description of the data, (2) exploratory data analysis, and (3) process mining.

Data Description

Google Analytics extracted web analytics data from three academic years (2018, 2019, and 2020). A total of 505,788 records were extracted for the following variables: (1) date timestamp, (2) client ID, (3) page, (4) pageviews, and (5) user type. The UA Dimensions and Metrics Explorer (n.d.) reports the date timestamp in the Central Standard Time zone. The client ID variable is a unique identifier that defines each website user based on the browser, device, and cookie combination. Although Google Analytics collects IP addresses from users, any identifiable information is anonymized, and client IDs cannot be traced back to users' personal information (Google, n.d.). The page variable identifies the specific resources accessed by the users. The pageviews variable refers to the number of times users viewed a website. The user type variable specifies new and returning users based on their client IDs.

The dataset was divided into three subsets for three school years (August 1 through July 31): 142,062 records for the 2018 academic year, 108,982 for the 2019 academic year, and 254,744 records for the 2020 academic year. A fourth subset of the 2020 academic year was obtained from March 1, 2020, to May 31, 2020, to observe 90 days of resource transactions from the pandemic's beginning.

Exploratory Data Analysis

Exploratory data analysis was performed over three academic years, and the pandemic subset data to explore the most accessed resources and user types (i.e., new and returning users). In addition, the ratios of user returns were calculated to understand the users' PD needs. User return ratios were derived by dividing the number of new users by the number of returning users organized by academic year and pandemic subsets.

Process Mining Techniques

The dataset was prepared for process mining by converting Google Analytics variables into a log format to allow the pm4py algorithm to parse event data into visualizations in direct follow-through graphs (Fraunhofer Institute for Applied Information Technology, n.d.). Direct follow-through graphs are maps where the nodes represent the event or activity, and the directed edges between the nodes (i.e., resource pages) connect the source event or activity to the target event or activity. Although other process mining algorithms exist (i.e., Alpha Miner, Heuristic Miner, and Inductive Miner), these algorithms tend to reduce and aggregate the complexity of event data, producing incomplete process models (Leemans et al., 2019). For this reason, direct follow-through graphs were implemented by considering frequency as the primary metric to measure the number of times the target event followed the source event. This specific study implemented a frequency metric to visualize and measure the total frequency of all events using the pageviews variable to describe all event relationships accurately.

The time: timestamp variable defined the date and time when a resource or page (concept: name) was accessed by the client ID (case:concept: name). The pageviews variable was converted as the cost measure because it indicates how many times a resource was accessed by a new or returning user (org: resource). Four direct follow-through maps were generated to visualize the total frequency of pageviews for frequent users at the beginning of each school year and for all users within 90 days of the pandemic. Google Analytics variables were converted for process mining, as shown in Table 1.

Because event logs are detailed records of user actions on a web server, two standard process mining decomposition techniques are case-based, vertical partitioning, and activitybased, horizontal partitioning. Case-based decompositions extract events from specific users, whereas activity-based decompositions partition events based on the activities being investigated. In this study, case-based decompositions were defined by the most frequent users with client IDs that contained their highest number of pageviews for the first 30 days of each school year. Activity-based decompositions were based on the activities of all users that occurred within 90 days of the pandemic in March 2020. Case- and activity-based decompositions provide inspectable SRL events and actions in direct follow-through maps by observing the number of times resources were accessed and their navigation patterns (i.e., content, search, and standard) based on pageviews. Case- and activity-based follow-through diagrams are accessible from srl.javierleung.com

FINDINGS

The findings are organized by the research questions.

TABLE 1

<div class="table-size-normal table-border"><table border="1"><tr><td>Conversion of Google Analytics to Process Mining Variables</td></tr><tr><td>Google Analytics</td><td>Process Mining</td></tr><tr><td>date timestamp</td><td>time: timestamp</td></tr><tr><td>client ID</td><td>case: concept: name</td></tr><tr><td>page</td><td>concept: name</td></tr><tr><td>pageviews</td><td>cost</td></tr><tr><td>user type</td><td>org: resource</td></tr></table></div>

RQ1: What are the most accessed resources for 2018, 2019, and 2020 academic years and 90 days into the COVID-19 pandemic?

In the 2018-2019 school year, the most accessed PD resources were (1) student growth and development modules, (2) teacher standard 1 (content knowledge) indicator site map, (3) classroom observation training materials, (4) teacher standard 4 (critical thinking) indicator site map, and (5) teacher indicator 2.4 (social-emotional learning) video examples. In the 2019-2020 school year, the most accessed PD resources were the (1) student growth and development modules, (2) teacher standard 1 indicator site map, (3) teacher standard 5 (classroom environment) indicator site map, (4) teacher standard 7 (effect of instruction) indicator site map, and (5) content engagement modules. In the 2020-2021 school year, the most accessed PD resources were (1) student growth and development modules, (2) classroom observation training materials, (3) remote learning modules, (4) teacher standard 1 indicator site map, and (5) teacher standard 7 indicator site map. At the beginning of the pandemic in March 2020, the most accessed PD resources were (1) classroom observation training materials, (2) student growth and development modules, (3) teacher standard 1 indicator site map, (4) teacher standard 7 indicator site map, and (5) teacher standard 4 indicator site map.

During the past three school years, the EdHub Library has attracted 124,785 new users and 150,976 returning users, with an approximate user-return ratio of 80%. Because of the interface redesign, the library has added 26,191 users. Table 2 summarizes the number of new and returning users for each school year and the first 90 days of the pandemic. In the first two school years of 2018 and 2019, approximately 80% of the users returned to the library to access PD. Even though the user base for the 2020-2021 school year and the first 90 days of the COVID-19 pandemic were significantly less than in previous school years, new users exceeded the total number of returning users as the pandemic unfolded in March 2020.

TABLE 2 User Return Ratio for Each School Year and 90 Days Into the Pandemic

<div class="table-size-normal table-border"><table border="1"><tr><td>Academic Year</td><td>New Users</td><td>Returning Users</td><td>User Return Ratio</td></tr><tr><td>2018&#8211;2019</td><td>62,589</td><td>79,473</td><td>0.79%</td></tr><tr><td>2019&#8211;2020</td><td>48,883</td><td>60,099</td><td>0.81%</td></tr><tr><td>2020&#8211;2021</td><td>13,313</td><td>11,404</td><td>1.17%</td></tr><tr><td>COVID-19 pandemic March 2020 (90 days)</td><td>12,731</td><td>12,651</td><td>1.00%</td></tr></table></div>

RQ2: What are the frequent library users' SRL events and actions in the first 30 days of 2018, 2019, and 2020 academic years and 90 days into the COVID-19 pandemic?

In case-based decompositions, four client IDs with the highest pageviews were identified in the first 30 days of each school year and 90 days at the beginning of the pandemic in March 2020. Each client ID's SRL events and actions were mapped to describe their navigation behaviors in the library. Table 3 describes four use cases from four client IDs and their respective pageviews.

The following four use cases were organized by client ID. Each use case was described using the information architecture of the library, from the EdHub Library homepage to subsequent instructional modules and activities on third-level pages. In the direct followthrough diagrams for the four use cases, the nodes shaded in purple indicate the increased traffic to particular resources on the first- (i.e., homepage) and second-level pages (i.e., topic categories). The SRL events and actions described how users leveraged the three search mechanisms among the homepage directory, teacher standard site maps, and the search engine. In addition, the SRL events and actions showed users' behaviors in accessing instructional modules on third-level pages across different topics to supplement their learning.

TABLE 3 Four Users From Case-Based Log Partitioning

<div class="table-size-normal table-border"><table border="1"><tr><td>30-day and 90-day Case-Based Log Partitioning</td><td>Client ID of Users</td><td>Pageviews</td><td>User Type</td></tr><tr><td>2018 Academic Year (08/01/2018&#8211;09/01/2018)</td><td>883818586</td><td>376 pageviews</td><td>New user</td></tr><tr><td>2019 Academic Year (08/01/2019 &#8211; 09/01/2019)</td><td>1157641077</td><td>253 pageviews</td><td>New user</td></tr><tr><td>2020 Academic Year (08/01/2020&#8211;09/01/2020)</td><td>1750362365</td><td>117 pageviews</td><td>New user</td></tr><tr><td>2020 Pandemic (03/01/2020&#8211;05/31/2020)</td><td>812965228</td><td>152 pageviews</td><td>Returning user</td></tr></table></div>

Use Case #1: Academic Year 2018, Client ID 883818586

Use Case # 1 shows a new user's SRL events and actions for the first use case (see Appendix, Use Case #1: Academic Year 2018 Client ID 883818586). By default, all learning events started on the EdHub Library homepage as the first-level page. For second-level pages, the user accessed the (1) teacher standard 1 indicator site map, (2) scoring classroom observation videos, (3) educational leadership topics, (4) getting started with EdHub, (5) content knowledge topics, (6) classroom management topics, (7) professional practices topics, and (8) beginning teacher support topics. For third-level pages, the user accessed modules in (1) social-emotional learning, (2) classroom and community culture, (3) second-grade and high school math video examples, (4) mentoring, and (5) classroom management of time, space, and transitions.

This user's actions revealed the preferred behavior for searching content through the EHub Library homepage and the teacher standard 1 indicator site map to access third-level pages for principal professional development modules and video examples for math and language arts for teacher indicator 1.1 (content and academic language). However, this user was unsuccessful in searching for content from the all-teacher standards site map page and then returned to the EdHub homepage.

Use Case #2: Academic Year 2019, Client ID 1157641077

Use Case #2 shows a new user's SRL events and actions for the second use case (see Appendix, Use Case #2: Academic Year 2019 Client ID 1157641077). By default, the EdHub Library homepage was the main starting point. This user relied on two resource pages for second- level pages to access content: (1) all teacher standards site map and (2) getting started with NEE topics. Interestingly, this user performed a successful search query for the principal indicators, resulting in getting started with the NEE module. In addition, this user leveraged the all-teacher standards site map to access third-level pages in (1) classroom observation scoring videos, (2) professional development planning modules, and (3) school building evaluation processes modules. Other third-level pages were accessed from the homepage, including dyslexia and learning and content engagement modules.

Use Case #3: Academic Year 2020, Client ID 1750362365

Use Case #3 shows a new user's SRL events and actions for the third use case (see Appendix, Use Case #3: Academic Year 2020 Client ID 1750362365). Although the EdHub Library homepage was the landing page for most actions the user performs, the beginning teacher support module was the starting page. This action indicated that the beginning teacher support page had been bookmarked previously, with subsequent access to the EdHub Library homepage. This user accessed several second- and third-level pages as follows: (1) teacher standard 7 indicator site map, (2) teacher standard 5 indicator site map, (3) classroom observation scoring video examples, (4) building instructional skills topics, and (5) beginning teacher support for administrators. It is worth noting that accessing third-level pages from the homepage is impossible because of the hierarchical design structure. However, this particular user could have bookmarked instructional modules from third-level pages or left the administrator page open from the last session, and then performed new searches by returning to the homepage.

Use Case #4: Academic Year 2020, Pandemic Client ID 812965228

Use Case #4 shows the SRL events and actions for the fourth use case of a returning user (see Appendix, Use Case #4: Academic Year 2020 Pandemic Client ID 812965228). Like the previous use case, this user could have bookmarked or left the administrator page open from the last session and then performed content searches on the homepage. For the second-level pages, the user accessed the (1) teacher standard 1 indicator site map, (2) teacher standard 5 indicator site map, (3) all teacher standards site map, (4) beginning teacher support, (5) NEE recertification training materials, and (6) building instructional skills topics.

From the second-level pages, the user accessed modules in the student learning and growth topic related to the emotional health of students and nine effective instructional strategies. In addition, this user accessed beginning teacher support topics and, subsequently, accessed formative assessment modules. While this user's behavior occurred between the beginning teacher support and formative assessment topics, this user exhibited SRL behaviors by accessing modules in two different topics to supplement the learning effort.

RQ3: What are all users' SRL events and actions during the first 90 days of the COVID-19 pandemic?

In activity-based decompositions, 25,382 events were discovered and visualized the navigation behaviors across all users in the first 90 days of the COVID-19 pandemic. The output of the process mining algorithm includes a large direct follow-through diagram that describes five prominent events that occurred across all users. These events showed users' navigation behaviors among the individual activities within the modules.

Event #1: Introduction to NEE.

Event #1 shows the SRL events and actions for all users navigating between the assessment and feedback activities in the effective feedback to students module (see Appendix, Event #1: Introduction to NEE). The first linear behavior was observed in the administrator library module when accessing three activities in a sequence as follows: (1) professional growth, (2) educators' use of adequate words, and (3) the first five days of the school year. The second linear behavior was the sequence of two activities from the professional practices module: what is good mentorship and standards for developing a successful mentoring program?

Event #2: Beginning Teacher Support for New Teachers

Event #2 shows the SRL events and actions for all users accessing the beginning teacher support module (see Appendix, Event #2: Beginning Teacher Support for New Teachers). In this event, users navigated the sequence of five instructional activities in the beginning teacher support module. Users also tended to review the two activities in the module related to the classroom environment and cognitive engagement.

Event #3: Social-Emotional Learning

Event #3 shows the SRL events and actions for all users accessing three out of the four activities in the social-emotional learning module (see Appendix, Event #3: Social-Emotional Learning). In this event, users followed the sequence of activities and returned to review the previous activities in promoting students' social competence and social-emotional learning needs modules.

Event #4: Cognitive Engagement

Event #4 shows the SRL events and actions for all users navigating between the two modules for introduction to cognitive engagement and prior knowledge (see Appendix, Event #4: Cognitive Engagement). Users did not follow the sequence of the three activities provided by these modules. In this event, users only accessed two activities in the introduction to cognitive engagement and prior knowledge modules. These two activities involved practical implementation of examples of engaging students cognitively and stimulating prior knowledge. Six activities were provided in the introduction to cognitive engagement module, but users only accessed the practical aspect of the module for building cognitive engagement into lessons. In this particular event, users were selective of PD materials ready for classroom implementation.

Event #5: Critical Thinking

Event #5 shows all users' SRL events and actions navigating the introduction to teaching critical thinking module (see Appendix, Event #5: Critical Thinking). In this event, users followed the sequence of five activities provided in the module and scored the practice videos in language arts and math.

DISCUSSION

The EdHub Library allows for macro and microlevel SRL processes using the homepage, dedicated teacher standard site maps, and the search engine. Users in the library exhibited strategies for selecting PD (i.e., strategy use) as the predominant macrolevel SRL process. Users also performed a combination of microlevel SRL processes to select and evaluate PD materials online. Even though the analyses were limited to navigation sequences in a technology-enabled environment based on log data, the navigation sequences indicated a combination of three microlevel SRL processes in the macrolevel SRL process of strategy use, including free search in a hypermedia environment, goal-directed search, and rereading.

The direct follow-through diagrams from the case- and activity-based decompositions showed three navigation behaviors: contentoriented, search-oriented, and standard-oriented. Content-oriented behaviors exhibited users' actions in navigating across the three levels of the library (i.e., homepage, topics, and modules). In search-oriented behaviors, users relied on the search engine from the EdHub Library homepage to search for all available PD materials. Standard-oriented behaviors showcased how users narrowed choices in selecting PD materials from the dedicated teacher standard site maps.

Most SRL events were content-oriented, where users looked for specific PD content. A few SRL events showed standard-oriented behaviors when using dedicated teacher standard site maps for teacher standard 1 (content knowledge), teacher standard 4 (critical thinking), teacher standard 5 (classroom environment), and teacher standard 7 (effect of instruction). Users seldom used the search-oriented approach but intentionally looked for PD aligned with teacher standards. Additionally, users who relied on content- and standard-oriented behaviors over search-oriented behaviors may have been discouraged from browsing lengthy search results. The literature notes that long search results discouraged users from seeking information in two instances: lack of clear search aims and increased cognitive load to evaluate search results (Badii et al., 2018; Kashyap et al., 2005; Otto et al., 2004).

The case-based direct follow-through maps established new and returning user profiles based on pageview frequencies. Although these user profiles did not describe particular behaviors across all users in the library, the user profiles established the navigation behavior characteristics of frequent users. User profiles mainly exhibited more content- and standard-oriented than search-oriented behaviors. Users may have used the search engine to initially perform a high-level overview of the PD materials in the library. The case-based direct follow-through maps also showed how users supplemented their knowledge with different instructional modules and activities from other topic areas.

The activity-based direct follow-through maps showed SRL events and actions from new and returning users in five topic areas of the EdHub Library: (1) introduction to NEE, (2) beginning teacher support for new teachers, (3) social-emotional learning, (4) cognitive engagement, and (5) critical thinking. Most events in the five topic areas exhibited content-oriented behaviors where users navigated linearly or cyclically between online activities within the modules.

Implications and Future Research

The findings have implications for PD developers, user experience designers, and researchers. This study highlights the need for PD developers to establish critical connections between concepts in teaching practices and concrete implementation examples of instructional strategies. Based on the observed SRL events and actions in the direct follow-through diagrams, instructional modules of core teaching practices can list the recommended resources to support the practical implementation of instructional strategies in the classroom. The library can aid the SRL development of educators, especially those with low self-efficacy, by connecting core concepts of teaching practices with practical examples for implementing instructional strategies. The findings also help designers of oTPD platforms create efficient interfaces with a strong information scent that aids phases in SRL.

While the navigation sequences indicated a combination of three microlevel SRL processes regarding free search in a hypermedia environment, goal-directed search, and rereading, other SRL processes could have occurred during library sessions. For example, educators may have taken digital notes, relied on curation tools to monitor progress, or worked in collaborative groups to assess alignment. Similar to the studies by Lim et al. (2021) and Greene and Azevedo (2009), the future direction of the present study involves using think-aloud data to visualize detailed follow- through diagrams that capture the microlevel SRL processes beyond log data for different types of users. In addition, future research on oTPD can further explore the relationship between SRL and information architectures that aid educators in four areas of SRL (i.e., cognition, motivation, behavior, and context).

LIMITATIONS

Two study limitations were the assignment of client IDs by Google Analytics and the lack of distinction between users (e.g., principals, teachers, and instructional staff). Google Analytics may have assigned two different client IDs to the same user. For example, during the pandemic, school principals could have used the same login credentials to access the EdHub Library from two locations—one from home and the other from their school district offices—using two separate IP addresses and browsers. School leaders could have accessed resources in two places as they decided to hold in-person or emergency remote teaching. Due to the built-in privacy mechanism in Google Analytics, the administrative console does not allow distinguishing between principals, teachers, and instructional staff.

CONCLUSION

This study accomplished two goals: (1) identifying the most accessed resources for three academic years (2018, 2019, and 2020) and the beginning of the COVID-19 pandemic, and (2) mapping SRL learning events and actions for frequent users and all library users in the first 30 days of the school year and 90 days into the pandemic of March 2020. The findings emphasized how educators self-regulated their behaviors when using three navigation approaches (i.e., content, search, and standard) as they navigated through the three levels of the EdHub Library's information architecture. This study documented the initial efforts to map SRL events and actions from an extensive online professional development library characterized predominantly by content- and standard- oriented behaviors. This study identified microlevel SRL processes (i.e., free search in a hypermedia environment, goal-directed search, and rereading) in the macrolevel SRL process of strategy use from a prominent online teacher professional development platform.

APPENDIX

Case- and activity-based follow-through diagrams are accessible on srl.javierleung.com

Case-based follow-through diagrams:

• Use Case #1: Academic Year 2018 Client ID 883818586

https://srl.javierleung.com/Case_based/ Case_1.html

• Use Case #2: Academic Year 2019 Client ID 1157641077 https://srl.javierleung.com/Case_based/ Case_2.html

• Use Case #3: Academic Year 2020 Client ID 1750362365 https://srl.javierleung.com/Case_based/ Case_3.html

• Use Case #4: Academic Year 2020 Pandemic Client ID 812965228 https://srl.javierleung.com/Case_based/ Case_4.html Activity-based follow-through diagrams:

• Event #1: Introduction to NEE https://srl.javierleung.com/Activity_based/ Activity_1.html

• Event #2: Beginning Teacher Support for New Teachers https://srl.javierleung.com/Activity_based/ Activity_2.html

• Event #3: Social-Emotional Learning https://srl.javierleung.com/Activity_based/ Activity_3.html

• Event #4: Cognitive Engagement https://srl.javierleung.com/Activity_based/ Activity_4.html

• Event #5: Critical Thinking https://srl.javierleung.com/Activity_based/ Activity_5.html

• Correspondence concerning this article should be addressed to: Javier Leung, leungj@missouri.edu

DIAGRAM: FIGURE 1 The EdHub Library Homepage

University of Missouri