Treffer: Data Structures and Algorithms for Progressive Data Analysis

Many data analysts today work in higher-level scientific languages. These popular scientific languages (SLs), such as Python, R, Julia, and MATLAB, are used to implement data pipelines to carry out analyses. They offer multiple layers of libraries, with standard data types such as vectors, arrays, tensors, data tables (also called Data Frames), and various types of graphs (aka networks). These data types support arithmetic operations, linear algebra, statistical operations, input and output, and other more specialized operations. These libraries and data types are fundamental to any data analysis and computation, and every data analyst relies on them.However, these standard libraries are not currently designed for progressive computation. A progressive language or platform should offer data structures and algorithms similar to SLs to achieve the same level of service and avoid the current situation where each PDA application has to re-implement all of them. This chapter explores some of the issues and challenges involved in implementing these data structures and algorithms for a progressive approach. It discusses how general algorithms and data structures that support arithmetic, linear algebra, basic statistics, and input/output methods should be adapted to become progressive. The chapters on data management, visualization, machine learning, and evaluation build on some of the mechanisms described here. Many of the issues discussed in this chapter are speculative. Our goal is to provide tips that can help the community tackle challenges now and in the future.