feat: Add streaming and concurrent file reads to ArrowScan to reduce memory usage and increase throughput#3046

Open

sumedhsakdeo wants to merge 12 commits intoapache:mainfrom

sumedhsakdeo:fix/arrow-scan-benchmark-3036

Commits on Feb 15, 2026

feat: forward batch_size parameter to PyArrow Scanner

sumedhsakdeo
and
claude
committed
style: fix ruff formatting in residual_evaluator lambda

sumedhsakdeo
and
claude
committed
chore: remove unintended vendor directory changes

sumedhsakdeo
and
claude
committed
feat: add streaming flag to ArrowScan.to_record_batches

sumedhsakdeo
and
claude
committed
feat: add concurrent_files flag for bounded concurrent streaming

sumedhsakdeo
and
claude
committed
fix: remove unused imports in test_bounded_concurrent_batches

sumedhsakdeo
and
claude
committed
feat: add read throughput micro-benchmark for ArrowScan configurations
sumedhsakdeo
committed
fix: remove extraneous f-string prefix in benchmark

sumedhsakdeo
and
claude
committed
ruff: format
sumedhsakdeo
committed
fix: properly reset mock call_count in test_hive_wait_for_lock

sumedhsakdeo
and
claude
committed
feat: add default-4threads benchmark and time-to-first-record metric

sumedhsakdeo
and
claude
committed
chore: remove default-4threads benchmark configuration

sumedhsakdeo
and
claude
committed