Velox in the Age of Machine Learning

In this talk, Pedro will present Velox community updates and new developments in Meta’s internal adoption of Velox. He will discuss how Velox has been focused on accelerating important AI/ML workloads related to training data preparation and training data loading at Meta, and highlight some of the active development/research areas where traditional data management systems fall short when supporting novel AI/ML workloads – and what the Velox community is doing to address them.

Pedro Pedreira
Velox Lead & Software Engineer at Meta

Training Data Loading with Velox

AI has become the first consumer of data. To ensure data-hungry trainers are consistently fed with training datasets, data loading platforms have quickly evolved, growing in popularity and scale. This workload brings novel challenges to data management related to new data shapes (extreme width), new batching and shuffling requirements, and tensor consumption. In this talk, we will discuss important data management features for AI/ML data, and how Velox is helping accelerate large-scale training data loading workloads.

Masha Basmanova
Co-creator of Velox & Software Engineer at Meta

PySpark Meets Velox: Redefining Efficiency in ML Workloads

AI applications within Meta, including recommendation systems and GenAI, are pushing the limits of our batch compute engines. As demand has grown exponentially over the past few years, bespoke solutions have emerged, but they often present challenges in terms of maintenance, reusability, efficiency, and reliability. In this talk, we introduce PySpark/Velox, a new offering at Meta that combines Python’s ease-of-use, Spark’s scalability, and Velox’s efficiency. We will discuss the architecture, various use cases and results, and how it fits into our existing stack.

Konstantinos Karanasos
Software Engineer at Meta

Kk Pulla
Software Engineer at Meta

Metalake – Meta’s Mutable Data Lake for Scalable Pre-training Data Loading

While off-the-shelf data lake solutions provide robust management for large datasets, Meta faces unique efficiency challenges, such as efficiently merging full-column updates at petabyte scale. Metalake, Meta’s internal mutable data lake, is designed to address these needs, enabling logical deletes, efficient rich update semantics, all while optimizing compute efficiency.

In this talk, we’ll dive into how Metalake, powered by Velox and PySpark, accelerates feature injection pipelines and scales model training with minimal compute overhead. We’ll present how Velox-based optimizations like shallow copy partitions, auto compaction, and encoding preserved merge-time rebatching eliminate traditional join performance bottlenecks, and significantly reduce compute costs.

Darren Fu
Software Engineer at Meta

State of the Union: Apache Gluten

Get the latest updates on Apache Gluten, the open-source project accelerating Spark with Velox. This session will cover recent developments, performance improvements, and roadmap plans, highlighting how Gluten enhances query execution efficiency in Spark.

Binwei Yang
Co-creator of Apache Gluten & Software Engineer at IBM

From Open Source to Enterprise: Real-World Applications of Velox and Apache Gluten in Dataproc’s NQE

Google Dataproc’s Performance Boost, powered by the integration of Velox and Apache Gluten, is revolutionizing big data analytics for enterprise customers. This session delves into the real-world impact of our Velox, showcasing how we’ve scaled these open-source technologies to deliver unprecedented performance gains. We’ll explore the architecture of our Dataproc integration, highlighting how Velox’s vectorized execution and Apache Gluten’s glue layer combine to significantly reduce query latency and resource consumption. Furthermore, we’ll detail our key improvements to the Spark-Velox integration including broadcast hash join optimizations that dramatically improve the performance of join-heavy workloads. We also focused on the addition and stabilization of comprehensive ORC and Parquet test cases, ensuring robust and reliable performance for a critical data format.

To help customers quickly understand the potential benefits, we’ve developed a qualification tool that analyzes Spark event logs and accurately predicts the performance gains achievable with Velox pushdown. Through case studies with marquee customers, we’ll demonstrate the tangible benefits of NQE, including accelerated data processing, improved cost efficiency, and enhanced scalability. Join us to learn how we’re pushing the boundaries of big data analytics with Velox and Apache Gluten in Google Dataproc, and how our extensions and innovative tools are contributing to the broader open-source community.

Abhishek Modi
Sr. Staff Software Engineer at Google

Techniques for addressing UDF incompatibility between spark and Gluten/Velox

In this talk, Shiyu talks through challenges encountered in the productionization of Gluten/Velox around compatibility between Spark’s and Velox’s json and re (regular expression) UDF’s and then presents two techniques in solving these incompatibilities between UDF’s.

Shiyu Gan
Software Engineer at Bytedance

Enhancing Pinterest’s Data Platform: A 2025 Update on Gluten Integration

In this lightning talk, we will explore the advancements Pinterest has made to its query platform using Gluten over the past year. We’ll begin with a brief overview of how Gluten is integrated into Pinterest’s infrastructure, followed by a discussion on the improved querying speeds and increased efficiency resulting from this integration. We will conclude with our plans for future developments and areas we want to improve on.

Felix Loesing
Software Engineer at Pinterest

Accelerate queries on Apache Hudi using Gluten and Velox

In this session, we’ll explore how Hudi-rs, the native Rust implementation of Apache Hudi, enables powerful integration with Velox to dramatically improve query performance. We’ll cover the technical design behind this integration, highlighting key architectural decisions that optimize data access patterns and query execution. Attendees will learn about the performance benefits achieved through benchmarks and see how this integration can be applied to their own data environments.

Shiyan Xu
Founding Team Member at Onehouse & Apache Hudi PMC member

Presto C++: The next decade in SQL processing

In this talk, we will cover the journey of Presto C++’s (Prestissimo) general availability at Meta across a diverse set of workload. We will provide insights into performance optimizations, correctness, and numerous stability improvements done over the last year. This work has enabled us to run Presto ~2-4X faster with 50% less overall capacity costs and further established a strong path for consolidation across various systems

Amit Dutta
Software Engineer, Meta

PyVelox: Python bindings for Velox

In this talk, we’ll introduce PyVelox, a recently published native Python package that brings the power of Velox to the Python ecosystem.

PyVelox provides seamless bindings to Velox’s core APIs, including PlanBuilder, I/O, LocalRunner, and more. We’ll explore these bindings in-depth, showcasing how they enable Python developers to tap into Velox’s lightning-fast performance and scalability. We’ll also demonstrate just how easy it is to get started with PyVelox on your local machine. With a simple installation process and intuitive API, you’ll be creating and executing plans for running TPCH queries in no time.

Krishna Pai
Software Engineer, Meta

Keynote: Accelerating the future of Open Data Engines – Velox at IBM

IBM has been a key contributor to Velox, leveraging its high-performance execution engine to optimize data processing at scale. In this keynote, Remus will share how IBM is integrating Velox across its data platforms, key innovations driven by the community, and what’s next for Velox in accelerating modern data workloads.

Remus Lazar
VP Software at IBM

Velox at Uber

Join us as we explore the transformative journey of productionizing Presto C++ (Prestissimo) at Uber. This talk will detail our progress on the migration to the next-generation data query engine, highlighting the technical challenges we encountered along the way, key milestones that were achieved, and the overall impact of Velox on Uber’s data processing capabilities.

Sergey Makagonov
Staff Software Engineer at Uber

Presto C++ Iceberg Support update

Presto’s C++ engine continues to improve its support for Apache Iceberg. This talk will cover recent enhancements in metadata handling, query planning, support of Insert, and read performance, along with future improvements on the roadmap.

Ying Su
Software Engineer at IBM

Presto C++ Feature Deep Dive

Presto continues to evolve with key enhancements in its C++ engine, leveraging Velox for performance and extensibility. This session will cover recent developments in 1) Unfenced UDFs, a new approach to defining and executing user-defined functions in Presto, improving flexibility and performance, 2) PartitionOutput to optimize data distribution for large-scale query processing, and 3) TopNRank operator which optimizes TPC-DS Q67 by ~2.5x. Q67 is the longest running query in the TPC-DS workload (about 30% of the total runtime).

Ying Su
Software Engineer at IBM

Soumya Duriseti
Software Engineer at IBM

Aditi Pandit
Software Engineer at IBM

Presto C++ Operational Efficiency: Memory Management, Metrics, and Caching

Operational efficiency is crucial for running Presto at scale. This session will explore recent advancements in Presto’s C++ engine that improve observability, resource management, and performance:

• LinuxMemoryChecker – A new mechanism for better memory tracking and enforcement, enhancing stability in resource-intensive workloads.
• Stats and Metrics – Deep dive into runtime stats and metrics, including a live demo showcasing how these metrics can help optimize query performance.
• SSDCache Usage – A deep dive into leveraging SSD-based caching to accelerate data access and reduce latency in Presto queries.

Christian Zentgraf
Software Engineer at IBM

Deepak Majeti
Velox Committer & Software Engineer at IBM

Minhan Cao
Software Engineer at IBM