Details

Autor(en) / Beteiligte

• Lamb, Andrew
• Shen, Yijie
• Heres, Daniël
• Chakraborty, Jayjeet
• Kabak, Mehmet Ozan
• Hsieh, Liang-Chi
• Sun, Chao

Titel

Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine

Ist Teil von

• Companion of the 2024 International Conference on Management of Data, 2024, p.5-17

Ort / Verlag

New York, NY, USA: ACM

Erscheinungsjahr

2024

Quelle

ACM Digital Library

Beschreibungen/Notizen

• Apache Arrow DataFusion is a fast, embeddable, and extensible query engine written in Rust that uses Apache Arrow as its memory model. In this paper we describe the technologies on which it is built, and how it fits in long-term database implementation trends. We then enumerate its features, optimizations, architecture and extension APIs to illustrate the breadth of requirements of modern OLAP engines as well as the interfaces needed by systems built with them. Finally, we demonstrate open standards and extensible design do not preclude state-of-the-art performance using a series of experimental comparisons to DuckDB. While the individual techniques used in DataFusion have been previously described many times, it differs from other industrial strength engines by providing competitive performance and an open architecture that can be customized using more than 10 major extension APIs. This flexibility has led to use in many commercial and open source databases, machine learning pipelines, and other data-intensive systems. We anticipate that the accessibility and versatility of DataFusion, along with its competitive performance, will further the proliferation of high-performance custom data infrastructures tailored to specific needs assembled from modular components. While the individual techniques used in DataFusion have been previously described many times, it differs from other industrial strength engines by providing competitive performance and an open architecture that can be customized using more than 10 major extension APIs. This flexibility has led to use in many commercial and open source databases, machine learning pipelines, and other data-intensive systems. We anticipate that the accessibility and versatility of DataFusion, along with its competitive performance, will further the proliferation of high-performance custom data infrastructures tailored to specific needs assembled from modular components.