Treffer: Introducing Prescriptive and Predictive Analytics to MBA Students with Microsoft Excel
Postsecondary Education
Weitere Informationen
Managers are increasingly being tasked with overseeing data-driven projects that incorporate prescriptive and predictive models. Furthermore, basic knowledge of the data analytics pipeline is a fundamental requirement in many modern organizations. Given the central importance of analytics in today's business environment, there is a growing demand for educational pedagogies that give students the opportunity to learn the fundamentals while also familiarizing them with how such tools are applied. However, a tension exists between the introduction of real-world problems that students can analyze and extract insight from and the need for prerequisite knowledge of mathematical concepts and programming languages such as Python/R. As a consequence, this paper describes an application-focused course that uses Microsoft Excel and mathematical programming to introduce MBA students with nontechnical backgrounds to tools from both prescriptive and predictive analytics. While students' gain proficiency in managing data and creating optimization and machine learning models, they are also exposed to broader business concepts. Teaching evaluations indicate that the course has helped students further develop their practical skills in Microsoft Excel, gain an appreciation of the real-world impact of data analytics, and has introduced them to a discipline they originally believed was best suited for more technically focused professionals.
As Provided
AN0174668198;[2wzs]01jan.24;2024Nov14.10:02;v2.2.500
Introducing Prescriptive and Predictive Analytics to MBA Students with Microsoft Excel
Managers are increasingly being tasked with overseeing data-driven projects that incorporate prescriptive and predictive models. Furthermore, basic knowledge of the data analytics pipeline is a fundamental requirement in many modern organizations. Given the central importance of analytics in today's business environment, there is a growing demand for educational pedagogies that give students the opportunity to learn the fundamentals while also familiarizing them with how such tools are applied. However, a tension exists between the introduction of real-world problems that students can analyze and extract insight from and the need for prerequisite knowledge of mathematical concepts and programming languages such as Python/R. As a consequence, this paper describes an application-focused course that uses Microsoft Excel and mathematical programming to introduce MBA students with nontechnical backgrounds to tools from both prescriptive and predictive analytics. While students' gain proficiency in managing data and creating optimization and machine learning models, they are also exposed to broader business concepts. Teaching evaluations indicate that the course has helped students further develop their practical skills in Microsoft Excel, gain an appreciation of the real-world impact of data analytics, and has introduced them to a discipline they originally believed was best suited for more technically focused professionals. Supplemental Material: Supplemental materials are available at https://doi.org/10.1287/ited.2023.0286.
Keywords: machine learning; mathematical optimization; prescriptive and predictive analytics; Excel
1. Introduction
The emergence of business analytics and artificial intelligence (AI) has fundamentally changed the requirements of a management education due to strong industry demand for personnel that have proficiency in business intelligence, quantitative analysis, and data engineering ([112], [24], [67]). In fact, the field of data science, which did not formally exist in the early 2010s ([81]), is projected to grow approximately 28% through 2026 ([87]). A nascent area is the manager who has enough low-level technical knowledge to understand the array of sophisticated analytics techniques that can be applied to data stores while also having a high-level perspective to see the managerial implications of the resulting analysis ([29]). Although this seems to be an incredible opportunity for MBA programs, which have recently seen a decline in the number of applications ([98]), many business schools are struggling with how to incorporate analytics training into their MBA programming ([94]).
Although many management schools offer graduate degrees in data science, incorporating modernized analytics modules into the MBA curriculum is challenging. From a program design perspective, there are no accreditation standards for developing machine learning and business analytics courses ([107]). Furthermore, many business-focused AI textbooks put too much emphasis on the application of analytics and do not give sufficient detail as to how the algorithms actually work ([62]). Alternatively, most academic AI resources are written for technical and/or mathematically oriented learners ([107]). Thus, there is tension between creating MBA courses that balance the need to have students understand analytics algorithms with teaching them how to interpret and communicate their findings ([49]). There is also pressure to ensure that other functional areas of business knowledge are well represented in the curriculum (e.g., marketing, accounting, and finance), to offer unique courses in new and promising industry segments (e.g., digital transformation, sustainability, and entrepreneurship), and to incorporate extracurricular and networking opportunities ([48], [45]). As a consequence, there may be scant room for offering multiple courses on the topic of business analytics. The previous difficulties are further exacerbated by three main course-level challenges.
1.1. Students Require Prerequisite Knowledge
Prescriptive and predictive analytics are grounded in several quantitative disciplines such as algebra, mathematical programming, numerical optimization, statistics, and data mining. Furthermore, many introductory analytics courses assume students have some programming experience in Python/R or teach students how to code ([73]). However, strong quantitative skills are typically lacking in MBA students ([103]) and because programming is difficult to learn, a considerable time investment must be made ([71], [39]). Consequently, in some courses geared toward nontechnical learners, exercises and assignments often rely on partially constructed solutions if they contain difficult mathematical concepts or snippets of code that students do not fully comprehend ([49]). Expecting that students have the required technical knowledge to succeed is clearly an impediment to encouraging them to seek out analytics training. Conversely, proficiency with Excel spreadsheets continues to be a technical area that graduate programs should place more emphasis on ([30], [88]) even though most MBA programs require applicants to have a working knowledge of Excel on admittance ([82]).
1.2. Experiential Education is Valuable Even for Management Training
Businesses must look beyond the data scientist to develop talent such that managers can contribute to producing data-driven solutions ([66]). However, to fully understand what data analysts do, what type of insights are possible, and the many pitfalls that may arise during the data engineering and modeling process, students must be trained to perform prescription/prediction tasks. It has also been shown that business students both enjoy, and gain a deeper understanding of, quantitative material when a hands-on approach is used ([107]). Nevertheless, when providing MBA students with opportunities to implement analytics models for difficult data-grounded problems, it is essential to have them also acquire skills related to problem conceptualization, model formulation, and the dissemination of insight ([31]). Thus, the ability to craft application-driven learning experiences is crucial to having MBA students understand how to better manage a team of analytics professionals that work together to make data-informed decisions that improve the efficiency and operations of organizations ([57]).
1.3. Modern Analytics Workflow
There are many aspects to data-driven decision making that must be touched on even for management-level knowledge. For instance, the education literature suggests that students should become proficient at assessing data quality, identifying relevant managerial questions, determining whether the data can be used to address the proposed questions, managing data sets, performing quantitative analyses, and interpreting model outputs ([103]). Although these concepts aptly summarize the data acquisition, data preparation, and the model engineering (i.e., train, test, evaluate) pipeline, there is also a fundamental difference between building analytics models that are used to extract managerial insight, and deploying a trained model to generate business value ([99]). Thus, MBA students should be aware of how analytics models can be deployed in production environments. Although this involves an introduction to machine learning operations (i.e., ML Ops), it also puts more emphasis on the integration of domain specific knowledge, both in the implementation of the AI tool and in determining what high-level insights can be derived from its deployment ([49]).
This paper describes an application-driven course that has been developed to introduce MBA students to optimization and machine learning concepts using the unifying language of mathematical programming. Although this lens has been introduced at a PhD level ([8]), it is argued that this perspective is also valuable for teaching at the MBA level. As a consequence, students are not required to have prerequisite programming knowledge nor do they need to demonstrate advanced mathematical proficiency. Furthermore, all analytics models can be implemented with Microsoft Excel, which strengthens their competency with the software. The course provides students with many opportunities to apply the tools they learn and repeatedly emphasizes the business insight that can be obtained from rigorous quantitative analysis. Thus, it differs from the literature in that it presents more detail than many courses designed for nontechnical learners ([102]) while also focusing on managerial issues such as quantitative storytelling, the model engineering process, and deployment; these topics are typically omitted from graduate-level curricula ([65]). In sum, the course teaches students how to obtain and communicate high-level analytics-based insight for real-world applications using experiential learning activities.
2. Background and Motivation
In the first year of their MBA at the Schuilch School of Business, students are required to complete two mandatory courses that relate to data analytics and operations management. The first is entitled
The elective half-year course, entitled
From a technical perspective, MAOR has historically focused on teaching mathematical optimization, decision trees, and Monte Carlo simulation. However, prior to the redesign of the curriculum, enrollment was low (10–15 students) and the examples were somewhat outdated. In fact, there had not been any substantial revision to the course content for many years and several topics overlapped with existing courses. More specifically, AI and its integration with society emerged as a strategic focus of the Schuilch School of Business and the university more broadly ([109], [110]). As a consequence, other courses were revised to meet this objective. For instance, the coverage of decision trees was moved to
The philosophical underpinnings behind the revised version of MAOR come from several sources including the author's own research and educational background, a review of the relevant literature ([101], [51], [19]), as well as discussions with colleagues within and outside the institution. The following five principles helped guide the redesign:
The revised course was first taught in Winter 2019 with 26 students (12 completed course evaluations) and Fall 2021 with 28 students (15 completed course evaluations). Although the course is typically offered in-person, the Fall 2021 version was fully online due to the COVID-19 pandemic.
3. Learning Objectives and Course Design
The purpose of MAOR is to provide students with an introduction to modern analytics through the analysis of real-world problems focusing on operations management. The course discusses data (acquisition, validation, and preparation), model engineering (training, testing, and evaluating prescriptive/predictive models), and model deployment. It consists of a mix of mini-lectures (videos) together with discussions of real-world applications (in-class sessions) focusing on decision situations. The central theme of the course is to have all topics motivated by circumstances that require evidenced-based decision making. Thus, students develop a sense of how to approach complex problems of a technical nature. Furthermore, a ubiquitous feature in modern management is the central role that data plays in driving core business processes. Consequently, students learn how to effectively analyze data sets and implement analytical tools using a mathematical programming framework.
The course is organized into two modules. The first module introduces students to optimization technology for prescriptive analytics and includes topics such as linear programming, sensitivity analysis, mixed-integer linear programming, goal programs, and nonlinear programming. Many application areas are covered including inventory modeling, price optimization, production, transportation, process analysis, scheduling, and supply chain management. Several topics in the first half of the course are somewhat standard and are covered in MBA-level operations courses such as decision analysis ([92]) and spreadsheet modeling ([85]). This is purposeful: not only are these topics taught at a level where larger-scale optimization models for more realistic-sized business problems are solved, but they act as a scaffold for creating prediction models using optimization technology.
The second module begins with a discussion of data acquisition, preparation, and wrangling and subsequently takes the view that machine learning is the application of optimization technology on historical data stores. As a result, supervised and unsupervised learning models are formulated as mathematical programming problems ([32]). Topics include regression (least squares, quantile, least absolute deviation, k-nearest neighbors), regularization (best-subset selection, ridge, lasso), classification (logistic, support vector machines, k-nearest neighbors), unsupervised models (k-means clustering, principal component analysis), machine learning operations, and modern examples of AI systems. Many applications are introduced including human resource management, worker/student performance, revenue management, healthcare process analysis, quality management, and product forecasting. Moreover, the optimization framework is applied to several data sets to obtain managerial insight. The limitations and benefits of using an exact approach are discussed (e.g., certificate of optimality, added expressiveness of the mathematical language), as well as the process of deployment (giving analytics-based recommendations versus deploying a trained model). Finally, the link between prescription and prediction is emphasized (i.e., predict-then-optimize) to connect the perspectives. All computational analysis is performed in Microsoft Excel using the native solver tool and with OpenSolver, an open-source alternative.
The course objectives are as follows:
3.1. Course Materials
There are no required textbooks for this course. This is because of financial considerations, the lens MAOR uses to introduce concepts in business analytics (i.e., the unifying language of mathematical programming), and because, as discussed in the Introduction, AI textbooks are typically too technical for business students or they do not present enough detail with respect to the underlying algorithms. Instead, students are provided with a detailed slide deck (∼100 slides per meeting session), several handouts, and multiple Excel spreadsheets that have worked examples of all problems discussed in class. In addition, a package of completed problems is provided for self-study (exams). An example slide deck covering mixed-integer linear programming (MILP) models (Week 5) and machine learning regression (Week 9) is included in the supplementary materials, as well as a handout on sensitivity analysis (Week 2) and three Excel files (one example is from Week 5 and two examples are from Week 9).
3.2. Computational Considerations
When solving problems motivated by real-world applications, it is often necessary to formulate large-scale models (e.g., hundreds or thousands of decision variables and/or constraints). For instance, constructing a mathematical program to solve a transportation problem (prescription) with 50 nodes may require thousands of decision variables. Alternatively, creating a quadratic program to solve a soft-margin, support vector machine (prediction) requires there to be at least as many decision variables as the number of data points and features ([32]).
Unfortunately, Microsoft Excel solver is limited to 200 decision variables and 100 constraints. Thus, to solve moderate- to large-scale problems, the open-source add-in OpenSolver is used ([64]). Not only is it easy to install, but it has no limit on the number of variables or constraints that can be incorporated into a model. The interface is similar to the native solver tool (Figure 1) and the advanced version has access to several powerful nonlinear solvers. This is especially important when formulating machine learning models as their objective functions are frequently nonlinear (e.g., logistic regression, ridge regression). Furthermore, its open-source status means that OpenSolver can be used in industry for noncore projects. Finally, there are Excel connectors for industrial solvers ([34]) which implies that students may see similar interfaces if they are employed in organizations that have licenses for these software tools.
Graph: Figure 1. Screenshot of the OpenSolver Model Dialog Box for the Best-Subset Regression Example Discussed in Week 9Note. The example is introduced in Section 3.3.2 and the Excel spreadsheet, SubsetModel, is described in Appendix B.2 and is provided as a supplementary file, Week9−Regularized Regression.xlsx.
Nevertheless, it's important to note that OpenSolver is not a panacea. Although there has been significant speed improvements in constrained optimization solvers, linear and quadratic programs in particular, state-of-the-art implementations of machine learning models as optimization problems still only contain hundreds to a few thousand data points ([9]). Although this is sufficient for many applications and for an introductory MBA course on analytics ([78]), the result is that the course does not delve into problems where big data sets are required ([25]). Thus, it is recommended that all models, especially assigned material, be run on several machines (MacOS, Windows) to determine how long it takes OpenSolver, or the native solver tool, to find an optimal solution. Then, an instructor can modify the problem (e.g., reduce the size of the data set, use a different solver) if the runtime is too long.
In Appendix A, more detail is provided regarding the use of OpenSolver, features that have been particularly important for the course, and issues that instructors should be aware of.
3.3. Weekly Sessions
The course is delivered over a 13-week semester. There are 12 two-hour meeting sessions, a midterm, and a final exam. An example course outline is provided in Table 1. Prior to each meeting session, students are required to watch a 20- to 40-minute video that contextualizes the upcoming in-class session. The video focuses on introducing a specific prescriptive/predictive technique, the settings under which it is appropriate, what/how data are used, potential pitfalls, and discusses how the model can be implemented in Excel. During the meeting session, one to three real-world applications that use the technique are introduced. Each example is structured similarly: problem definition, identifying relevant data, data preparation and transformation, model engineering, discussion of managerial insights, and the communication of business knowledge.
Table 1. Example Course Outline
This method of pedagogy is known as the flipped classroom ([14], [3]) and is particular advantageous in this setting as it allows students to learn at their own pace, makes course content more easily accessible, and meeting sessions are almost entirely devoted to (i) introducing a real-world application; (ii) solving a data-driven analytics problem; and (iii) contextualizing the example within the MBA curriculum to build an understanding of the challenges that may be associated with encountering the problem outside the classroom (e.g., issues with data acquisition, combining multiple competing perspectives). Furthermore, for each problem, a spreadsheet associated with the implementation of the solution is provided. Apart from being extensively used during class, it is a great resource for self-study in that students can review the steps associated with data wrangling, identify the actions needed for implementing the analytics model, and experiment with alternative formulations (e.g., adding/removing constraints, different types of regularization). This helps to improve their Excel proficiency and provides a template for producing spreadsheets with a similar structure (e.g., assignments/exams).
3.3.1. Detailed Outline—Prescriptive Analytics Session: MILP Models (Week 5).
From a technical perspective, the purpose of this week's module is to have students realize that mathematical programs can include many types of decision variables (continuous, integer, discrete). The implication is that very general models can be created whose decisions only depend on what type of managerial insight one would like to learn. That is, models are not limited to one type of decision variable but, instead, depend on the business decisions that need to be understood/made. The prelecture video touches on this idea noting that the computational complexity of the underlying algorithm is no more difficult than solving an integer or binary program (the previous week's class). What's difficult about this class of problems, however, is that to faithfully represent the real-world situation, the different types of decision variables need to be linked. Thus, a new class of constraints is required. To understand how these
During the two-hour class discussion, two to three examples are introduced that pertain to some aspect of operations management. They can include make versus buy decisions ([2]), building plan design ([75]), production and transportation, and volume discounts ([96]). Each example is derived from a research project and/or a consulting experience of the instructor or their colleagues, which means that there is a rich discussion of the setting that motivates each problem. All examples follow the same rubric: problem description, definition of decision variables and objective function, creation of constraints, introduction of the linking constraints, formulation in Excel, and analysis of the optimal solution.
The in-class examples give students the opportunity to formulate and solve a mathematical programming problem, and thus, they get hands-on experience at model building in a low-stress environment. Furthermore, the problems incorporate
From a managerial point of view, the flipped classroom environment allows for a significant interaction. Furthermore, after solving each problem, there is a rich conversation of next steps (e.g., how to interpret the results, follow-up analyses that could be performed to demonstrate the robustness of the findings, the ways in which the results can be communicated to management). Finally, operational insights motivated by the solution are considered. For example, in the production and transportation example, a dog food manufacturer must decide which production facilities to operate (binary decision variables) and which customers to serve from which operating plant (continuous decision variables). The optimal solution has that a small subset of all production facilities are used. Furthermore, the students observe that some customers receive dog food from production facilities in a distant city even though there is an option to rent a facility in the same city, whereas other customers are not served at all. The example highlights how the high fixed cost of renting a production facility is driving the model to minimize the number of rented plants. In some cases, the total cost of production is such that it makes sense not to serve a customer. However, if variable costs were higher and fixed costs were lower, the insights may differ. Thus, model interpretability is important as it provides visibility into why an analytics tool produced a particular solution.
For a detailed discussion of the
3.3.2. Detailed Outline—Predictive Analytics Session: Regression (Week 9).
From a technical perspective, the purpose of this week's module is to have students realize that mathematical programs can be formulated to solve prediction tasks. This concept is introduced in the previous two sessions with a discussion of empirical risk minimization and the presentation of ordinary least squares (OLS) regression as an unconstrained quadratic optimization problem. However, it is expanded on in the prelecture video where mathematical programming formulations of regularized regression models (lasso, ridge, and best subset selection) are introduced.
To be more concrete, suppose that a data set has been collected with
where
where
In the prelecture video, apart from presenting the mathematical programming formulation for each regression model, insight is given as to the motivation for the constraints, when each model should be applied, and several real-world studies that utilize the different regularization techniques. In the subsequent in-class session, an example is introduced where students apply the models to a historical data set of academic performance associated with high school students in Portugal. The data set must first be cleaned and some of the variables transformed before it is separated into a training and testing set. The final data set contains 33 features and 395 records, which is split using the 80/20 rule. Then, students implement each regularized model in Excel including two benchmarks: (i) predicting the mean response of the training set and (ii) OLS regression. After implementation, students evaluate the results and how changes to the budget parameter affects model performance. The quantitative discussion concludes with students having to choose the best-performing prediction model, motivating that choice (e.g., lowest root mean squared error), and identifying the most important features for predicting grades, that is, home life situation, presence of both parents, parental education levels, rural versus urban location, school reputation, whether the student has university aspirations, number of absences, and the amount of study time.
After the quantitative analysis, students must indicate how they would go about presenting these findings to parents and school administration alike. This discussion focuses on what is controllable (e.g., number of absences, study time) versus intractable factors that cannot be easily changed (e.g., home life situation, presence of both parents, parental education levels, rural versus. urban location). The classroom conversation eventually leads to a broader dialogue on how to incentivize individuals, such as employees, to achieve desirable performance ([91], [53]). The discussion also introduces behavioral nudges ([97], [28]), a more recent mechanism that is used to indirectly influence behavior, such as Uber rewarding badges to incentivize drivers to work longer hours and Deliveroo using push notifications to nudge to their food delivery workers' into working faster ([68]).
Two additional models are introduced during the in-class session. The first is the formulation of quantile regression which is motivated by the desire to predict the median response (e.g., house prices) and to create prediction intervals. The corresponding model is a linear program:
where
Finally, an implicit assumption regarding both quantile and regularized OLS regression is that the approximator is a linear function of the features. Although more complex models can be introduced in advanced classes, such as mathematical programs for decision tree learning ([26], [1]), their formulation is too complex for the MBA audience. Instead, a nonparametric model (kNN regression) can be formulated where the function class does not need to be specified:
where
The session concludes by presenting an overview of the data wrangling and model engineering process, and contrasts the role of data in creating predictive/prescriptive models. For a detailed discussion of the
3.3.3. Discussion Sessions (Week 8 and Week 12).
There are two sessions that do not follow the flipped classroom structure. These sessions present high-level content on the data acquisition and preparation process, and model engineering (train, test, evaluate), as well as details associated with deploying analytics models in practice (i.e., user insight vs. the creation of an AI tool). Week 8 acts as a transition session between the presentation of prescriptive (Weeks 1–7) and predictive (Weeks 9–12) models. Students are introduced to noteworthy concepts associated with the process of creating prediction tools such as transforming data, model engineering, assessing out-of-sample performance, data leakage, model bias, and the bias-variance tradeoff. This is especially important for MBA graduates if and when they manage multiple analytics professionals (e.g., data scientists, machine learning engineers) on large AI projects. Furthermore, the final session (Week 12) focuses on the end result of analysis, that is, whether the models are used to obtain managerial insight or are deployed. Concrete suggestions on effective communication strategies are given such as ensuring that recommendations and business insights are clearly communicated, the importance of creating visual aids, the use of outlines to systematically organize complex ideas, and how to balance the amount of detail that is presented (e.g., main text versus appendix). In addition, a few real-world examples of customer-facing AI tools are showcased. The classroom conversation highlights the purpose of each tool and the potential ethical issues associated with its use. A summary of the class content is given in Appendix C.
3.4. Assignments and Exams
There are five deliverables in MAOR: three assignments worth 12% each and two noncumulative exams worth 32% each. Using Table 1 as a guide, the first assignment covers weeks 2–4, the second covers material from weeks 5–7, and the last assignment focuses on weeks 8–11. Because of time constraints, machine learning operations is not assessed. Finally, the first exam focuses entirely on prescriptive modeling, whereas the second exam covers data engineering and predictive modeling.
All work is submitted individually, although students are encouraged to consult with their peers on assignments. Deliverables have the same format: they contain two to three questions that introduce a modern problem, provide real data, and require both technical and managerial answers. Each question has students perform data preprocessing and wrangling, necessitates the implementation of one or more prescriptive/predictive models, requires the evaluation of the proposed models, and asks several high-level questions about the implications of the results. Deliverables are submitted online, and the only difference between assignments and exams is that the latter has a shorter time interval for completion. That is, all deliverables are take-home, open-book assessments that are completed using Microsoft Excel. For more details, see Appendix B.4 and the online appendix, which includes a sample exam and the corresponding Excel spreadsheet.
3.5. Consulting Project
There is currently no culminating project in MAOR. The reason is that it is a second-year elective course and, during this year, MBA students are required to complete an intense capstone project that liaises with a real company and conducts a comprehensive strategic assessment of all functional organizational areas. Because students analyze many aspects of a business and make recommendations to management in a strategic action plan, adding another consulting project to their workload would be placing an undue burden on their already packed schedules. Furthermore, there may also be conflicts of interest that would greatly increase the logistical complexity of MAOR.
Nevertheless, in many courses, consulting projects with real companies are incorporated into the curriculum ([15]). This can be an extremely valuable learning experience as it exposes students to the analytics pipeline (problem definition, data acquisition, model engineering) while emphasizing soft skills such as the writing of technical reports and presenting analytical findings to management. To facilitate these types of experiences, instructors can collaborate with their department's career center or use online platforms such as Riipen (https://app.riipen.com), which connects educators to industry partners so that a real-world project can be designed as an in-course assignment (both approaches have been successfully used in other courses taught by the author).
4. Student Feedback and Course Modifications
To determine the effectiveness of the course curriculum, feedback from the online student evaluation system is examined. The program solicits both quantitative and qualitative responses on the efficacy of the instructor, the utility of the course content, and how closely the student's experiences match with their expectations. Changes to the course, driven by both formal/informal student feedback and structural modifications induced by the COVID-19 pandemic, are also discussed.
4.1. Student Evaluations
To determine the success of the curriculum, a subset of the quantitative course evaluation questions that most directly relate to content design and delivery were selected. This includes the structure of the weekly modules, the effectiveness of the deliverables, and the ability of the instructor to achieve the stated course objectives. The mean course score, department score, and faculty score are recorded with the best assessment being seven and the worst being one (i.e., a seven-point scale is used as focus groups at the school indicated that students desired a more fine grained set of distinctions compared with the five-point scale). The same set of questions are given to all courses offered by the business school. Results for the following 10 questions are presented in Table 2.
Table 2. Student Evaluations Scores (Seven-Point Scale) for 10 Questions Most Directly Related to Content Delivery
1
For every question, significantly higher evaluations for MAOR are observed compared with other courses offered by the department and within the business school. The final three rows of Table 2 present the average scores for all 10 questions. Student
Informal discussions with students in MAOR indicate that the favorable evaluations are because of the perceived relevance of the topics covered in the course and the accessibility of the curriculum (e.g., Excel versus Python, emphasis on obtaining business insight versus exploring mathematical proofs/structure). The scores from Table 2 suggest that students are very much engaged with, and interested in, the material and that from the student perspective, the course design is effective at developing problem-solving and critical thinking skills. Finally, the analysis underscores the fact that although some aspects of the course are novel (in particular, representing machine learning models as mathematical programming problems) and, in many cases, there are few resources on the topic, students are satisfied with the support they have received (lecture slides, videos, and in-class sessions) and are intellectually stimulated rather than overwhelmed by the content.
Although student evaluations can be a biased indicator of quality ([100]) and their validity in evaluating teaching effectiveness has been repeatedly questioned ([40]), the results do indicate that the course has been successful at achieving the desired objectives. Furthermore, research suggests that there is a correlation between high course ratings in student evaluations one year, and a subsequent increase in a course's overall enrollment in the subsequent year ([17], [111]). Although MAOR has not been consistently offered, the quantitative evaluations indicate that it may be quite popular once teaching schedules stabilize.
In addition to the quantitative analysis of student course evaluation scores, a thematic analysis of student comments was performed ([16]). The relevant question, and student responses to that question, are given in Table 3 where each quote best captures multiple students' sentiment. The qualitative analysis further suggests that the course has been successful at achieving the learning outcomes and following the five philosophical principles that helped guide the redesign. The results indicate that MAOR has made students more excited about the techniques and applications related to the field of data analytics and it is apparent that, after taking the course, student's still believe that the concepts they learned will be useful in their future careers. A common theme is that student's appreciated the balance between the technical and managerial content, enjoyed the opportunity to implement analytics models, and valued the materials that were provided to them. In fact, the ability to complete assignments in Excel was especially important.
Table 3. Qualitative Comments from Student Evaluations
4.2. Student-Driven Course Updates
After the Winter 2019 version of the course, a theme emerged that slightly changed the curriculum for Fall 2021. The following comment summarizes the motivation behind these changes:
"I think more time could be spent on predictive analysis (machine learning), as it is more useful and difficult to learn. I feel it needs more time to teach and learn this topic."
In the Winter 2019 version of MAOR, there was considerably less discussion of empirical risk minimization (Week 7) and machine learning operations (Week 12), whereas dimensionality reduction was not covered at all (Week 11). After adding these topics, the same sentiment was not expressed in the Fall 2021 course evaluations. Nevertheless, additional changes are being considered. In particular, efforts are focused on determining whether goal programming (Week 6) should be replaced. The original motivation for including this topic is because deviational variables are seen in quantile regression (Week 9). However, the discussion of soft constraints can be incorporated into some of the examples associated with linear programming (Weeks 3–5) and be reinforced in Week 9.
Two potential candidates have emerged as replacements: stochastic programming and an additional session on data wrangling. With stochastic programming, more emphasis can be placed on how data are used in prescriptive analytics. There are also connections with the discussion of empirical risk minimization and the structure of supervised learning models. Finally, a similar lecture is given to students in a different graduate program with identical technical backgrounds. Alternatively, by including another session on data acquisition/wrangling, more emphasis can be placed on how to determine what data are required to answer complex business problems. Furthermore, additional detail can be given as to how to clean and visualize data for model engineering.
A few technically proficient students expressed their desire to see Python incorporated into the course curriculum. This feedback has not, and will not, be incorporated for two main reasons. First, a majority of students enjoyed the coverage of Excel (see, for example, Table 3). These students indicated that they were happy to improve their proficiency with spreadsheet modeling and, more importantly, would not take the course if coding knowledge became a prerequisite. This suggests that, for the most part, the course is appealing to the desired audience. Second, there are currently two masters programs offered by the business school that require students to complete more mathematically and computationally challenging analytics courses. Knowledge of Python/R is expected and MBA students are welcome to take those courses as electives if they desire.
4.3. Case-Based Assignments
Although the course deliverables are motivated by real-world applications, none of them are case based. To further bolster the managerial content of MAOR, assignments and/or exams could be derived from case studies. For instance, there are a plethora of cases at Harvard Business Publishing that require mathematical programming and machine learning methods. In fact, the author is exploring whether future iterations of the course will replace the assignment component with case studies. One of the reservations associated with making this change is that, typically, case studies are completed by groups of MBA students. As a consequence, not all students may get hands-on experience preparing and analyzing the data, implementing the prescriptive/predictive models, validating their effectiveness, and reporting the results; this is a serious concern.
4.4. Hybrid Delivery
In the Winter 2019 version of MAOR, there were no prerecorded video lectures. Instead, meeting sessions were three hours long, and all content was presented in a classroom setting. However, halfway through the semester, COVID-19 shut down all in-person lectures. As a result, prelecture videos were introduced as meeting sessions (over Zoom) and were shortened to two hours. This format remained for the Fall 2021 version of the course, where meeting sessions were also delivered over Zoom. The following comment summarizes the consensus observed with this format:
"Liked the class discussions and that [the] materials were provided prior to class."
Because of the almost unanimous support of the course structure, a hybrid delivery will be adopted going forward. As discussed in Section 3.3, the videos will cover more technical content, whereas the in-person sessions will be devoted to examples that use the methods in real-world applications.
5. Discussion and Conclusion
Mathematical programming constitutes a fundamental quantitative skill that MBA graduates should possess, and because of this, is taught pervasively in operations management programs throughout North America ([63], [61]). At the same time, data analytics is a nascent but in-demand field that has been shown to produce tangible business benefits ([44]). This paper introduces a new course geared toward MBA students that teaches them the fundamentals of prescriptive and predictive analytics using the unifying perspective of mathematical optimization. The curriculum is primarily based on a flipped classroom design ([3]) where technical content is presented as short prelecture videos and real-world examples are solved collectively during meeting sessions. There are many experiential learning components as students get hands-on opportunities to solve quantitative problems with Excel through in-class examples, assignments, and exams. However, particular focus is placed on ensuring that managerial concepts in operations management are emphasized while also describing how to communicate analytics-driven insight to business managers. Finally, an overview of the analytics pipeline is presented, several AI tools (e.g., fraud detection, revenue management) are introduced, and broader ethical issues associated with their deployment are discussed.
Student feedback indicates that this course has achieved its learning objectives and has appropriately balanced the presentation of technical material with insightful commentary on managerial decision-making. They also suggest that the flipped classroom format has been successful and that the use of Excel has been well received. Interestingly, not one student commented on feeling mathematically unprepared, which indicates that the course may have strong appeal to nontechnical learners who want to gain a better understanding of the relationship between business problems and the application of analytical tools used for decision-making. Finally, it was observed that students appreciated the emphasis on real-world problems; this allows for rich discussions that emphasize the connections between analytics and modern topics in operations and supply chain management. This emphasis also means that, as compared with typical introductory operations courses offered in many MBA programs, larger-sized and potentially more intricate models can be formulated.
The course aims to produce "data-savvy" managers ([60]). To do so, MBA students must learn the fundamentals of analytics and receive hands-on training to frame problems and interpret the results while also connecting these quantitative skills to the broader business discipline ([105]). In sum, they must relate the low-level implementation of mathematical models and computational algorithms, to the high-level processes that govern the deployment of AI tools to extract insight, make data-driven decisions, and are used by organizations to interact with their customers. By unifying the presentation of prescriptive and predictive analytics through the use of mathematical optimization, the course de-emphasizes mathematical knowledge and puts the focus on business-level problem solving abilities and critical thinking skills. Furthermore, providing different perspectives—from micro (model engineering) to macro (analytics pipeline) processes—and incorporating multiple real-world examples allows MBA students to gain the confidence to contribute to analytics teams. This is imperative when working on large organizational projects that are far more intricate and technically complex. Although many business schools have different viewpoints on how best to prepare MBA students for such a position and an industry that increasingly values analytics and AI ([48]), this course presents, what are hopefully, valuable insights on how such a curriculum can be designed and delivered.
Acknowledgments
I thank Amber Moore whose encouragement directly motivated the creation of this work, Andre Cire for careful review of the original manuscript, and the editor, associate editor, and anonymous reviewers whose comments and suggestions considerably improved the article.
Appendix A. Overview of OpenSolver's Functionality
OpenSolver is an Excel add-in that extends the native solver tool in two notable ways: (i) there is no limitation on the number of decision variables and constraints that can be included in an optimization model; and (ii) there are several nonlinear solvers. It is also free and open source ([64]). Students in MAOR are required to download the advanced version of OpenSolver to gain access to the linear and nonlinear solvers. After downloading the zip file and extracting the folder, students open the application by double-clicking on
The OpenSolver interface (Figure A.1) is intuitive and similar to the native Excel Solver tool (see the tutorial at https://opensolver.org/using-opensolver/). In fact, many students find it convenient to build larger models using the native solver tool and solve them with OpenSolver's algorithms. Apart from the OpenSolver Model dialog box which is used to build the model, two key aspects of its functionality are highlighted:
Graph: Figure A.1. OpenSolver Interface in Excel's Data Tab on Windows
Graph: Figure A.2. OpenSolver Solver Engine Dialog Box
Graph: Figure A.3. OpenSolver Options Dialog Box
As is apparent, using OpenSolver to solve optimization problems in Excel is no harder than learning how to formulate models with the native solver tool. Thus, while considerable time is spent discussing Solver and the reports (e.g., sensitivity) it generates (Week 2), little time is spent reviewing the layout of the OpenSolver Model dialog box. Instead, as previously described, the novel features of the software are highlighted. In addition, there are several unique issues that are reviewed:
Appendix B. Description of Excel Examples Included in the Supplementary Files
In this section, details regarding the Excel files included as part of the online appendix are discussed. The first three examples are associated with the discussion of course content in Week 5 (see Section 3.3.1) and Week 9 (see Section 3.3.2), whereas the fourth example is associated with the motivation behind the questions on the final exam. For each application, some background is given (i.e., the problem setting, managerial issues, and the data that has been collected), the methodological focus is highlighted, instruction regarding the formulation of the model is provided, the use of Excel/OpenSolver is described, and some in-class discussion questions are posed.
B.1. Production and Transportation Example
Although this problem is fairly typical for MBA-level operations courses, details regarding this particular application are motivated by a personal relationship with an executive at Masters Best Friend. Consequently, some background information regarding the company is given, as well as the motivation behind the particular problem. Some simplifying assumptions have been made (e.g., production cost/capacity are identical across facilities), and all data are fictitious. The managerial decisions represent whether certain production facilities should be rented whether certain orders should be satisfied, and how many units should be transported from each rented production facility to each customer that is served. Then, there is a brief class discussion on the data that has been obtained, how one would go about collecting this data in practice, and additional information that one may need to procure in order to have the model faithfully represent the real-world setting.
Although a detailed explanation of the steps associated with deriving the optimization model is presented in the supplementary file
From a technical perspective, the purpose of this example is to demonstrate the generality of the type of constraints that can be used to link decision variables in mathematical programming problems. Earlier in the session, an example with Big-M constraints is introduced that uses a single binary variable to toggle a single continuous decision variable to be some positive number or zero. In this example, each binary variable toggles multiple decision variables to be positive or, collectively, all zero. Furthermore, it reinforces the idea that a single model can be created to answer multiple managerial questions by incorporating multiple types/classes of decision variables. However, in order for the model to produce sensible results, the different variables must be linked.
The Excel spreadsheet (Figure B.1) in the online appendix that corresponds to this example is entitled
Graph: Figure B.1. Screenshot from the Production and Transportation Example Highlighted Using OpenSolver's Show Model FeatureNote. The option to Show/Hide model is provided as a button on the OpenSolver tab.
The model can be implemented and solved using both the Excel Solver tool or OpenSolver. Figure B.1 presents a screenshot of the Excel spreadsheet where the model has been highlighted using OpenSolver's Show/Hide Model button. The details of how the model is formulated in OpenSolver is given in Figure B.2. The interface and implementation is similar to the native Excel Solver tool. The only difference is that adding constraints does not require an extra dialog box.
Graph: Figure B.2. Screenshot of the OpenSolver Model Dialog Box for the Production and Transportation Example
In addition to examining how Solver came to the optimal solution (interpretability) and the sensitivity of the solution to changes in model parameters (as discussed in Section 3.3.1), student feedback is solicited on how one would go about showing the robustness of these results and how best to present these findings to executives. Additional ideas for classroom discussion include the following:
B.2. Regularized Regression Example
The setting and data set are derived from the paper by [22]. Nevertheless, it is important to emphasize what managerial questions this type of analysis is attempting to answer, how pervasive these types of inquiries are (i.e., predicting student/employee performance given a large data set of features), and how difficult it is to draw causal conclusions from the results. From a technical perspective, the purpose of this example is to demonstrate how to apply optimization models for predictive purposes using a machine learning (i.e., train, test, evaluate) approach. To this end, a detailed explanation of the steps associated with formulating optimization models of ordinary least squares, lasso, ridge, and best-subset selection regression is given in the first half of the supplementary file
The Excel file has nine spreadsheets. The first two contain the raw data set and a reference to where the data were obtained. The raw data are split into a training (80%) and testing (20%) set. In each of these spreadsheets, columns A:AF contain the raw data: there are 315 instances in the
Four different predictive models are trained (see the spreadsheets
where
The optimization model in
Running the optimization models provide estimates of
Graph: Figure B.3. Screenshot from the Regularized Regression Example Demonstrating the Predictive Performance of the Regression Models (OLS, Ridge, Lasso, Subset) and the Benchmark (Average)
Although the regularized models (either ridge or lasso) exhibit slightly better performance than OLS, the test set metrics do indicate that the outcome variable can be fairly accurately predicted (e.g., there is a significant difference between the benchmark's performance metrics—see
B.3. k-Nearest Neighbor (kNN) Regression Example
The setting and data set are identical to what is described in the
The Excel file has five spreadsheets. The first two contain the raw data set and references where the data were obtained. In contrast to the
The final spreadsheet,
A screenshot from the OpenSolver Model dialog box is presented in Figure B.4. Because the model is linear, the COIN-OR CBC (Linear Solver) engine can be selected. After formulating the model and selecting "Save Model," the "Solve" button should be pressed (Figure A.1). The prediction for the test set instance is in cell F3 and can be compared with the actual grade (cell I2). To this end, running the model with different values of
Graph: Figure B.4. Screenshot from the kNN Regression Example Demonstrating the OpenSolver Model
The resulting discussion focuses on data (see Section 3.3.2) and the differences between parametric and nonparametric prediction models. Other function classes are briefly introduced (e.g., decision trees, neural networks) and the tradeoff between these prediction machines are discussed, such as capacity versus interpretability. For instance, some ideas for classroom discussion include the following:
B.4. Sample Final Exam
There are three questions associated with this final exam. The answer key (
Graph: Figure B.5. Screenshot of the Excel Spreadsheet from Question 1 of the Final Exam
Graph: Figure B.6. Screenshot of the Excel Spreadsheet from Question 2 of the Final Exam
Graph: Figure B.7. Screenshot of the Excel Spreadsheet from Question 3 of the Final Exam
Appendix C. Discussion Sessions
C.1. Week 8: Introduction to Prediction
C.1.1. Video.
Defining predictive analytics (the training of a computational model using a historical data set that generalizes a decision rule against a given performance metric by anticipating unseen future inputs), clearly defining the prediction problem (what objective are we serving? what are we trying to do for the end-user? why is the problem important?), acquiring a data set (how do we decide what data to collect and how much is needed?), secure storage (where do we store the data?), and privacy compliance (how should sensitive information be protected?).
C.1.2. Class.
Data wrangling (reformatting particular attributes and correcting errors in the data set, what to do with missing values), feature engineering, irrelevant data and outliers, overview of model engineering (train, test, evaluate), data leakage, the difference between prediction (estimating an outcome based on the association between a dependent variable and set of independent features) and causality (identifying mechanisms through which certain responses are observed), evaluating model performance (e.g., root mean squared error, coefficient of determination, confusion matrix, precision, recall), and modeling bias ([90]).
C.2. Week 12: Introduction to Machine Learning Operations
C.2.1. Video.
Overview of the analytics pipeline and the differences between recommendations for business insight versus deploying an AI tool for widespread adoption/interaction.
C.2.2. Class.
How to present/write technical insights for business audiences if the use case is for generating analytics-based recommendations ([35]). Some AI tools are deployed, however, and real-world examples include COMPASS to predict recidivism ([50], [86]), recommendation algorithms ([74], [5]), AI tools used for automating medical diagnoses ([6], [33]), fraud detection ([93], [36]) vehicle routing ([38], [27]), and revenue management and pricing ([20], [4]). Because of time constraints, only two applications are covered during the in-class session. However, because of the high-level presentation associated with these examples, some of this content has been taught in other courses.
For each application, the AI tool and its managerial purpose is described, the concept of model drift (predictive) and the need for reparameterization (prescriptive) is introduced, and quality management (detecting issues, retraining) is discussed ([58]). Finally, students are introduced to the ethical issues surrounding AI systems such as the possibility that decisions can reinforce existing forms of prejudice and amplify inequality ([77]).
References
1 Aghaei S, Gomez A, Vayanos P. (2020). Learning optimal classification trees: Strong max-flow formulations. Preprint, submitted May 13, https://arxiv.org/abs/2002.09142.
2 Agrawal VK. (2006). Why, when, and what to outsource, Chapter 11. Schniederjans A, Schniederjans D, Schniederjans M, eds. Outsourcing Management Information Systems (Idea Group Publishing, Hershey, PA).
3 Akçayir G, Akçayir M. (2018). The flipped classroom: A review of its advantages and challenges. Comput. Ed. 126 : 334–345.
4 Ammirato S, Felicetti AM, Linzalone R, Volpentesta AP, Schiuma G. (2020). A systematic literature review of revenue management in passenger transportation. Meas. Bus. Excell.
5 Anand R, Sabeenian R, Gurang D, Kirthika R, Rubeena S. (2021). AI based music recommendation system using deep learning algorithms. Proc. IOP Conf. Series Earth Environment Sci., vol. 785, 012013. https://iopscience.iop.org/article/10.1088/1755-1315/785/1/012013/pdf.
6 Babier A, Mahmood R, McNiven AL, Diamant A, Chan TC. (2020). Knowledge-based automated planning with three-dimensional generative adversarial networks. Medical Phys. 47 (2): 297–306.
7 Bertsimas D, Dunn J. (2017). Optimal classification trees. Machine Learn. 106 (7): 1039–1082.
8 Bertsimas D, Dunn J. (2019). Machine Learning Under a Modern Optimization Lens (Dynamic Ideas, Charlestown, MA).
9 Bertsimas D, King A. (2016). OR forum—An algorithmic approach to linear regression. Oper. Res. 64 (1): 2–16.
Bertsimas D, Wiberg H. (2020). Machine learning in oncology: methods, applications, and challenges. JCO Clinical Cancer Inform. 4:885–894.
Bertsimas D, Allison K, Pulleyblank WR. (2016). The Analytics Edge (Dynamic Ideas, Charlestown, MA).
Bertsimas D, Orfanoudaki A, Wiberg H. (2021). Interpretable clustering: An optimization approach. Machine Learn. 110 (1): 89–138.
Bertsimas D, Copenhaver M, Wang Y, Pawlowski C. (2019). Machine learning under a modern optimization lens. Accessed April 5, 2023, https://bit.ly/3XRFjbO.
Bishop J, Verleger MA. (2013). The flipped classroom: A survey of the research. Proc. ASEE Annual Conf. & Exposition, 23–1200. https://peer.asee.org/the-flipped-classroom-a-survey-of-the-research.
Boutilier JJ, Chan TC. (2023) Introducing and integrating machine learning in an operations research curriculum: An application-driven course. INFORMS Trans. Ed. 23(2):64–83.
Braun V, Clarke V. (2006). Using thematic analysis in psychology. Qual. Res. Psych. 3 (2): 77–101.
Brown CL, Kosovich SM. (2015). The impact of professor reputation and section attributes on student course selection. Res. Higher Edu. 56 (5): 496–509.
Brusco M (2022) Logistic regression via excel spreadsheets: Mechanics, model selection, and relative predictor importance. INFORMS Trans. Ed. 23(1):1–11.
Carillo KDA. (2017). Let's stop trying to be "sexy": Preparing managers for the (big) data-driven business era. Bus. Processing Management J. 23(3):598–622.
Carrier E, Fiig T. (2018). Future of airline revenue management. J. Revenue Pricing Management 17:45–47.
Corbett-Davies S, Goel S. (2018). The measure and mismeasure of fairness: A critical review of fair machine learning. Preprint, submitted August 14, https://arxiv.org/abs/1808.00023.
Cortez P, Silva AMG. (2008). Using data mining to predict secondary school student performance. Accessed April 5, 2023, http://www3.dsi.uminho.pt/pcortez/student.pdf.
Cousineau M, Verter V, Murphy SA, Pineau J. (2023). Estimating causal effects with optimization-based methods: A review and empirical comparison. Eur. J. Oper. Res. 304(2):367–380.
Dean MD. (2020). Using the learning assistant model in an undergraduate business analytics course. INFORMS Trans. Ed. 20 (3): 125–133.
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. (2009). Imagenet: A large-scale hierarchical image database. Proc. IEEE Conf. on Comput. Vision and Pattern Recognition (IEEE, Piscataway, NJ), 248–255.
Dunn JW. (2018). Optimal trees for prediction and prescription. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA.
Ellegood WA, Solomon S, North J, Campbell JF. (2020). School bus routing problem: Contemporary trends and research directions. Omega 95 : 102056.
Fishbane A, Ouss A, Shah AK. (2020). Behavioral nudges reduce failure to appear for court. Science 370 (6517): eabb6591.
Fitzgerald S. (2012). MBA in business analytics: Big data are a big deal. Accessed April 5, 2023, https://bit.ly/3PRb7ZL.
Formby SK, Medlin D, Ellington VB. (2017). Microsoft Excel®: Is it an important job skill for college graduates? Inform. Systems Ed. J. 15 (3): 55.
Fry MJ. (2008). Teaching problem solving techniques to MBA students through supply chain management. INFORMS Trans. Ed. 8 (2): 65–74.
Gambella C, Ghaddar B, Naoum-Sawaya J. (2021). Optimization problems for machine learning: A survey. Eur. J. Oper. Res. 290 (3): 807–828.
Gomolin A, Netchiporouk E, Gniadecki R, Litvinov IV. (2020). Artificial intelligence applications in dermatology: Where do we stand? Frontiers Medicine 7 : 100.
Gurobi Optimization LLC (2022). Gurobi optimizer reference manual. Accessed April 5, 2023, https://www.gurobi.com.
Hager PJ, Scheiber HJ, Corbin NC. (1997). Designing & Delivering: Scientific, T echnical, and M anagerial P resentations (John Wiley & Sons, Hoboken, NJ).
Hasan I, Rizvi SAM. (2022). AI-driven fraud detection and mitigation in e-commerce transactions. Proc. Data Analytics and Management: ICDAM 2021, vol. 1 (Springer, Berlin), 403–414.
Herrington JD. (2010). MBA: Past, present and future. Acad. Ed. Leadership J. 14 (1): 63.
Holland C, Levis J, Nuggehalli R, Santilli B, Winters J. (2017). Ups optimizes delivery routes. Interfaces 47 (1): 8–23.
Holzmen S, Gooch S. (2016). Learning to code at a business school. Accessed April 5, 2023, https://bit.ly/3oKwAYw.
Hornstein HA. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Ed. 4 (1): 1304016.
Huggins E, Bailey M, Guardiola I. (2020). Case article—Converting nfl point spreads into probabilities: A case study for teaching business analytics. INFORMS Trans. Ed. 21 (1): 45–48.
Jurney R. (2017). Agile Data Science 2.0: Building Full-Stack Data Analytics Applications with Spark (O'Reilly Media, Newton, MA).
Kaushal A, Altman R, Langlotz C. (2020). Healthcare AI systems are biased. Sci. Amer. 11 : 17.
Kelly J. (2022). Data analytics is a hot, sexy sector for career growth. Accessed April 5, 2023, https://bit.ly/3bqV1ar.
Kirkpatrick L. (2020). Is an MBA degree really worth it? Accessed April 5, 2023, https://bit.ly/3cXlNHQ.
Kopcso D, Pachamanova D. (2018). Case article—Business value in integrating predictive and prescriptive analytics models INFORMS Trans. Ed. 19(1):36–42.
Kraft MA, Rogers T. (2015). The underutilized potential of teacher-to-parent communication: Evidence from a field experiment. Econom. Ed. Rev. 47 : 49–63.
Kurczy S. (2018). MBAs who code. Accessed April 5, 2023, https://bit.ly/3d1djj3.
Langley P. (2019). An integrative framework for artificial intelligence education. Proc. Conf. AAAI Artificial Intelligence (AAAI Press, Palo Alto, CA), vol. 33, 9670–9677.
Larson J, Mattu S, Kirchner L, Angwin J. (2016). How we analyzed the compas recidivism algorithm. ProPublica 9 (1): 3.
Laursen GH, Thorlund J. (2016). Business Analytics for Managers: Taking Business Intelligence Beyond Reporting (John Wiley & Sons, Hoboken, NJ).
LeClair D. (2018). Integrating business analytics in the marketing curriculum: Eight recommendations. Marketing Ed. Rev. 28 (1): 6–13.
Lee DK, Zenios SA. (2012). An evidence-based incentive system for Medicare's end-stage renal disease program. Management Sci. 58 (6): 1092–1105.
Leong TY, Cheong ML. (2008). Teaching business modeling using spreadsheets. INFORMS Trans. Ed. 9 (1): 20–34.
Liberatore MJ, Nydick RL. (1999). The teachers' forum: Breaking the mold—A new approach to teaching the first mba course in management science. Interfaces 29 (4): 99–116.
Lin-Siegler X, Ahn JN, Chen J, Fang FFA, Luna-Lucero M. (2016). Even Einstein struggled: Effects of learning about great scientists' struggles on high school students' motivation to learn science. J. Ed. Psych. 108 (3): 314.
Longwood University (2021). What can I do with an MBA in data analytics? Accessed April 5, 2023, https://bit.ly/3QdSKOs.
Luigi (2019). The ultimate guide to model retraining. Accessed April 5, 2023, https://mlinproduction.com/model-retraining/.
Manyika J, Silberg J, Presten B. (2019). What do we do about the biases in AI? Accessed April 5, 2023, https://bit.ly/3oK0KLF.
Manyika J, Chui M, Brown B, Bughin J, Dobbs R, Roxburgh C, Byers AH. (2011). Big data: The next frontier for innovation, competition, and productivity. Accessed April 5, 2023, https://mck.co/3vxARlU.
Mark D. (2021). What is a typical MBA program curriculum? Accessed April 5, 2023, https://bit.ly/3Sl6Tv6.
Marr B, Ward M. (2019). Artificial Intelligence in Practice (Wiley, New York).
Martinez de Albeniz Margalef V. (2011). Linear programming basics. Case study, Harvard Bus. Rev. https://store.hbr.org/product/linear-programming-basics/IES442.
Mason AJ (2012) Opensolver-an open source add-in to solve linear and integer programmes in excel. Klatte D, Lüthi H-J, Schmedders K, eds. Selected Papers Internat. Conf. Oper. Res. (OR 2011) (Springer-Verlag, Berlin), 401–406.
Mike K, Hazan T, Hazzan O. (2020). Equalizing data science curriculum for computer science pupils. Proc. 20th Koli Calling Internat. Conf. on Comput. Ed. Res. (ACM, New York), 1–5.
Miller S, Hughes D. (2017). The Quant Crunch: How the Demand for Data Science Skills Is Disrupting the Job Market (Burning Glass Technologies, Boston).
Mitra S, Goldstein Z, Kapoor BL. (2021). Predictors of choosing business analytics concentration and consequent academic performance. INFORMS Trans. Ed. 21 (3): 130–144.
Möhlmann M. (2021). Algorithmic nudges don't have to be unethical. Accessed April 5, 2023, https://bit.ly/3oPRky9.
Molnar C. (2020). Interpretable machine learning. Leanpub Publishing, https://books.google.ca/books?hl=en&lr=&id=jBm3DwAAQBAJ&oi=fnd&pg=PP1&dq=Molnar+C+(2020)+Interpretable+machine+learning.&ots=EgzP0kIJS5&sig=WEB5YhdpOkRUCLFDyvwLzWjppvI&redir_esc=y#v=onepage&q=Molnar%20C%20(2020)%20Interpretable%20machine%20learning.&f=false.
Montoya M. (2022). MS Excel as a modern data analysis tool? Accessed April 5, 2023, https://bit.ly/3oKpDqm.
Murray S. (2014). Why MBA students are learning to code. Accessed April 5, 2023, https://bit.ly/3BATJnU.
Naderi B, Roshanaei V, Baron O. (2022). An interactive spreadsheet model for teaching classification using logistic regression. Working paper. Accessed April 5, 2023, https://bit.ly/3KenIoq.
Nagpal A, Gabrani G (2019) Python for data analytics, scientific and technical applications. 2019 Amity Internat. Conf. Artificial Intelligence (AICAI) (IEEE, Piscataway, NJ), 140–145.
Naumov M, Mudigere D, Shi HJM, Huang J, Sundaraman N, Park J, Wang X, et al. (2019). Deep learning recommendation model for personalization and recommendation systems. Preprint, submitted May 31, https://arxiv.org/abs/1906.00091.
Ngowtanasuwan G. (2012). An application of mathematical model for decision in building plan design. Procedia Soc. Behav. Sci. 68 : 537–548.
Olsen-Phillips P. (2022). What to look for in a business analytics curriculum. Accessed April 5, 2023, https://bit.ly/3BzLVTj.
O'Neil C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Broadway Books, New York).
Pachamanova D, Tilson V, Dwyer-Matzky K. (2022). Case article—Machine learning, ethics, and change management: A data-driven approach to improving hospital observation unit operations. INFORMS Trans. Ed. 22 (3): 178–187.
Patrick J, Montazeri A, Michalowski W, Banerjee D. (2019). Automated pathologist scheduling at the ottawa hospital. INFORMS J. Appl. Analytics 49 (2): 93–103.
Paul JA, MacDonald L. (2020). Analytics curriculum for undergraduate and graduate students. Decision Sci. J. Innovative Ed. 18 (1): 22–58.
Press G. (2013). A very short history of data science. Accessed April 5, 2023, https://bit.ly/3SjkyTG.
Questrom School of Business (2022). Preparation for the online MBA. Accessed April 5, 2023, https://bit.ly/3zn1NWx.
Regan PJ. (2005). Professional decision modeling: Details of a short MBA practice course. INFORMS Trans. Ed. 6 (1): 35–52.
Roth WS, Matherne BP. (2021). Using moneyball to introduce students to data analytics: Illustrating the data analytics life cycle. INFORMS Trans. Ed., ePub ahead of print July 13, https://doi.org/10.1287/ited.2021.0252ca.
Rotman School of Management (2022). Rsm2408h: Modelling and optimization for decision making. Accessed April 5, 2023, https://bit.ly/3PPPuJr.
Rudin C, Wang C, Coker B. (2018). The age of secrecy and unfairness in recidivism prediction. Preprint, submitted November 2, https://arxiv.org/abs/1811.00731.
Schroeder B. (2021). The data analytics profession and employment is exploding—Three trends that matter. Accessed April 5, 2023, https://bit.ly/3SlBOY8.
Seal KC, Leon LA, Przasnyski ZH, Lontok G. (2020). Delivering business analytics competencies and skills: A supply side assessment. INFORMS J. Appl. Analytics 50 (4): 239–254.
Shah R. (2019). Stand up for best practices: Misuse of deep learning in nature's earthquake aftershock paper. Accessed April 5, 2023, https://bit.ly/3OLMbBG.
Shepperd M. (2021). CS5702 modern data book. Accessed April 5, 2023, https://bit.ly/3BzXQ3q.
Sommer SC, Loch CH. (2009). Incentive contracts in projects with unforeseeable uncertainty. Production Oper. Management 18 (2): 185–196.
Stern School of Business (2021). Syllabus for decisions models and analytics (Fall 2021, MBA Program). Accessed April 5, 2023, https://bit.ly/3JqVkOS.
Soviany C. (2018). The benefits of using artificial intelligence in payment fraud detection: A case study. J. Payments Strategy Systems 12 (2): 102–110.
Stine J, Trumbore A, Woll T, Sambucetti H. (2019). Implications of Artificial Intelligence on Business Schools and Lifelong Learning (Academic Leadership Group, Cambridge, MA).
Telenti A, Jiang X. (2020). Treating medical data as a durable asset. Nature Genetics 52 (10): 1005–1010.
Terwiesch C, Cachon G. (2019) Matching Supply with Demand: An Introduction to Operations Management, 4th ed. (McGraw-Hill Education, New York).
Thaler RH. (2018). From cashews to nudges: The evolution of behavioral economics. Amer. Econom. Rev. 108 (6): 1265–1287.
Thomas P. (2022). MBA applications at some of the country's best colleges fell this year. Accessed April 5, 2023, https://on.wsj.com/3zMvbXN.
Visengeriyeva L, Kammer A, Bär I, Kniesz A, Plöd M. (2020). Why you might want to use machine learning. Accessed April 5, 2023, https://ml-ops.org/content/motivation.
Wachtel HK. (1998). Student evaluation of college teaching effectiveness: A brief review. Assessment Evaluation Higher Ed. 23 (2): 191–212.
Warner J (2013) Business analytics in the MBA curriculum. Proc. Northeast Bus. Econom. Assoc. https://web.p.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=e6e875a2-6e06-4800-8bd8-aa5eb949737b%40redis.
Way T, Papalaskari MA, Cassel L, Matuszek P, Weiss C, Tella YP. (2017). Machine learning modules for all disciplines. Proc. 2017 ACM Conf. Innovation Tech. Comput. Sci. Ed., 84–85. https://dl.acm.org/doi/pdf/10.1145/3059009.3072979.
Weathers D, Aragón O. (2019). Integrating analytics into marketing curricula: Challenges and effective practices for developing six critical competencies. Marketing Ed. Rev. 29 (4): 266–282.
Wilder C, Li S. (2019). Case—You can't take it with you. INFORMS Trans. Ed. 20 (1): 37–40.
Wilder C, Ozgur C. (2015a). Business analytics for business analysts in manufacturing. Handbook of Research on Organizational Transformations Through Big Data Analytics (IGI Global, Pennsylvania), 97–105.
Wilder CR, Ozgur CO. (2015b). Business analytics curriculum for undergraduate majors. INFORMS Trans. Ed. 15 (2): 180–187.
Xu JJ, Babaian T. (2021). Artificial intelligence in business curriculum: The pedagogy and learning outcomes. Internat. J. Management Ed. 19 (3): 100550.
York BN, Loeb S, Doss C. (2019). One step at a time the effects of an early literacy text-messaging program for parents of preschoolers. J. Human Resources 54 (3): 537–566.
York University (2022a). Fostering the future of artificial intelligence: Report from the York University task force on AI & society. Accessed April 5, 2023, https://bit.ly/3vyc80S.
York University (2022b). York university strategic research plan (2018-202): Toward new heights. Accessed April 5, 2023, https://bit.ly/3vwMJok.
Yu KW, Mincieli L, Zipser N. (2021). How student evaluations of teaching affect course enrollment. Assessment Evaluation Higher Ed. 46 (5): 779–792.
Zhan J, Her YW, Hu T, Du C. (2018). Integrating data analytics into the undergraduate accounting curriculum. Bus. Ed. Innovation J. 10 (2).
By Adam Diamant
Reported by Author