LLM benchmarking framework with SystemDS & Ollama & VLLM Backends - LDE Project

[kubraaksux]

Benchmarking framework that compares LLM inference across four backends: OpenAI API, Ollama, vLLM, and SystemDS JMLC with the native llmPredict built-in. Evaluated on 5 workloads (math, reasoning, summarization, JSON extraction, embeddings) with n=50 per workload.

Purpose and motivation

This project was developed as part of the LDE (Large-Scale Data Engineering) course. The llmPredict native built-in was added to SystemDS in PR #2430. This PR (#2431) contains the benchmarking framework that evaluates llmPredict against established LLM serving solutions, plus the benchmark results.

Research questions:

1. How does SystemDS's llmPredict built-in compare to dedicated LLM backends (OpenAI, Ollama, vLLM) in terms of accuracy and throughput?
2. How does Java-side concurrent request dispatch scale with the llmPredict instruction?
3. What is the cost-performance tradeoff across cloud APIs, local CPU inference, and GPU-accelerated backends?

Approach:

• Built a Python benchmarking framework that runs standardized workloads against all four backends under identical conditions (same prompts, same evaluation metrics)
• The llmPredict built-in (from PR Add LLM inference support to JMLC API #2430) goes through the full DML compilation pipeline (parser → hops → lops → CP instruction) and makes HTTP calls to any OpenAI-compatible inference server
• Ran evaluation in two phases: (1) sequential baseline across all backends, (2) SystemDS with Java-side concurrency (ExecutorService thread pool in the llmPredict instruction)
• GPU backends (vLLM, SystemDS) executed on NVIDIA H100 PCIe (81GB). Ollama ran on local MacBook (CPU). OpenAI ran on local MacBook calling cloud API. All runs used 50 samples per workload, temperature=0.0 for reproducibility.

Project structure

scripts/staging/llm-bench/
├── runner.py                  # Main benchmark runner (CLI entry point)
├── backends/
│   ├── openai_backend.py      # OpenAI API (gpt-4.1-mini)
│   ├── ollama_backend.py      # Ollama local server (llama3.2)
│   ├── vllm_backend.py        # vLLM serving engine (streaming HTTP)
│   └── systemds_backend.py    # SystemDS JMLC via Py4J + llmPredict DML
├── workloads/
│   ├── math/                  # GSM8K dataset, numerical accuracy
│   ├── reasoning/             # BoolQ dataset, logical accuracy
│   ├── summarization/         # XSum dataset, ROUGE-1 scoring
│   ├── json_extraction/       # Built-in structured extraction
│   └── embeddings/            # STS-Benchmark, similarity scoring
├── evaluation/
│   └── perf.py                # Latency, throughput metrics
├── scripts/
│   ├── report.py              # HTML report generator
│   ├── aggregate.py           # Cross-run aggregation
│   └── run_all_benchmarks.sh  # Batch automation (all backends, all workloads)
├── results/                   # Benchmark outputs (metrics.json per run)
└── tests/                     # Unit tests for accuracy checks + runner

Note: The llmPredict built-in implementation (Java pipeline files) is in PR #2430. This PR includes the benchmark framework and results only. Some llmPredict code appears in this diff because both branches share the same local repository.

Backends

Backend	Type	Model	Hardware	Inference path
OpenAI	Cloud API	gpt-4.1-mini	MacBook (API call)	Python HTTP to OpenAI servers
Ollama	Local server	llama3.2 (3B)	MacBook CPU	Python HTTP to local Ollama
vLLM	GPU server	Qwen2.5-3B-Instruct	NVIDIA H100	Python streaming HTTP to vLLM engine
vLLM	GPU server	Mistral-7B-Instruct	NVIDIA H100	Python streaming HTTP to vLLM engine
SystemDS	JMLC API	Qwen2.5-3B-Instruct	NVIDIA H100	Py4J → JMLC → DML llmPredict → Java HTTP → vLLM

SystemDS and vLLM Qwen 3B use the same model on the same vLLM inference server, making their accuracy directly comparable. Any accuracy difference comes from the serving path, not the model.

Benchmark results

Evaluation methodology

Each workload defines its own accuracy_check(prediction, reference) function that returns true/false per sample. The accuracy percentage is correct_count / n. All accuracy counts were verified against raw samples.jsonl files and reproduced locally.

Workload	Criterion	How it works
math	Exact numerical match	Extracts the final number from the model's chain-of-thought response using regex patterns (explicit markers like ####, \boxed{}, bold **N**, or the last number in the text). Compares against the GSM8K reference answer. Passes if abs(predicted - reference) < 1e-6.
reasoning	Extracted answer match	Extracts yes/no or text answer from the response using CoT markers ("answer is X", "therefore X") or the last short line. Compares against BoolQ reference using exact match, word-boundary substring match, or numeric comparison.
summarization	ROUGE-1 F1 >= 0.2	Computes ROUGE-1 F1 score between the generated summary and the XSum reference using the rouge-score library with stemming. A threshold of 0.2 means the summary shares at least 20% unigram overlap (F1) with the reference. Predictions shorter than 10 characters are rejected.
json_extraction	>= 90% fields match	Parses JSON from the model response (tries direct parse, markdown code fences, regex). Checks that all required fields from the reference are present. Values compared with strict matching: case-insensitive for strings, exact for numbers/booleans. Passes if at least 90% of field values match.
embeddings	Score within 1.0 of reference	The model rates sentence-pair similarity on a 0-5 STS scale. The predicted score is extracted from the response. Passes if abs(predicted - reference) <= 1.0 (20% tolerance). This is standard for STS-B evaluation.

Accuracy (% correct, n=50 per workload)

Workload	Ollama llama3.2 3B	OpenAI gpt-4.1-mini	vLLM Qwen 3B	SystemDS Qwen 3B c=1	SystemDS Qwen 3B c=4	vLLM Mistral 7B
math	58%	94%	68%	68%	68%	38%
json_extraction	74%	84%	52%	52%	52%	50%
reasoning	44%	70%	60%	60%	64%	68%
summarization	80%	88%	50%	50%	62%	68%
embeddings	40%	88%	90%	90%	90%	82%

Key comparisons

SystemDS vs vLLM (same model, same server — Qwen2.5-3B-Instruct on H100):
SystemDS c=1 matches vLLM Qwen 3B accuracy exactly on all 5 workloads (68%, 52%, 60%, 50%, 90%). This confirms that the llmPredict instruction produces identical results to calling vLLM directly. Both use temperature=0.0 (deterministic), same prompts, same inference server. c=4 shows minor variation on reasoning (64% vs 60%) and summarization (62% vs 50%) because concurrent requests cause vLLM to batch them differently, introducing floating-point non-determinism in GPU computation.

OpenAI gpt-4.1-mini vs local models:
OpenAI achieves the highest accuracy on all 5 workloads. The gap is largest on math (94% vs 68% for Qwen 3B) and smallest on embeddings (88% vs 90% for Qwen 3B, where the local model actually wins). OpenAI's advantage comes from model quality (much larger model), not serving infrastructure.

Qwen 3B vs Mistral 7B (different models, same vLLM server):
Despite being smaller (3B vs 7B parameters), Qwen outperforms Mistral on math (68% vs 38%) and embeddings (90% vs 82%). Mistral is better on reasoning (68% vs 60%) and summarization (68% vs 50%). This shows that model architecture and training data matter more than parameter count alone. Mistral's low math score (38%) has two causes: in 20 of 31 incorrect samples the model computed the wrong answer entirely (wrong formulas, negative results, or refusing to solve), and in 10 cases the correct answer appeared in the response but the number extractor grabbed an intermediate value instead due to verbose chain-of-thought formatting.

Ollama llama3.2 3B (MacBook CPU):
Ollama leads on summarization (80%) likely because llama3.2's training emphasized concise outputs that align well with the ROUGE-1 threshold. It scores lowest on embeddings (40%) because the model frequently refuses the similarity-rating task or defaults to high scores regardless of actual similarity.

Per-prompt latency (mean ms/prompt, n=50)

Workload	Ollama (MacBook CPU)	OpenAI (MacBook → Cloud)	vLLM Qwen 3B (H100)	SystemDS Qwen 3B c=1 (H100)
math	5781	3630	4619	2273
json_extraction	1642	1457	1151	610
reasoning	5252	2641	2557	1261
summarization	1079	1036	791	373
embeddings	371	648	75	41

Note on measurement methodology: Latency numbers are not directly comparable across backends because each measures differently. The vLLM backend uses Python requests with streaming (SSE token-by-token parsing adds overhead). SystemDS measures Java-side HttpURLConnection round-trip time (non-streaming, gets full response at once). Ollama measures Python HTTP round-trip on CPU. OpenAI includes network round-trip to cloud servers. The accuracy comparison is the apples-to-apples metric since all backends process the same prompts.

SystemDS concurrency scaling (throughput)

Workload	c=1 (req/s)	c=4 (req/s)	Speedup
math	0.44	1.63	3.71x
json_extraction	1.62	5.65	3.49x
reasoning	0.79	3.11	3.95x
summarization	2.62	7.27	2.78x
embeddings	20.07	46.34	2.31x

Throughput = n / total_wall_clock_seconds (measured Python-side, end-to-end including JMLC overhead). Theoretical maximum speedup is 4x. Math and reasoning (longer generation, ~1-2s per prompt) get closest to 4x because the per-request time dominates. Embeddings (very short responses, ~41ms per prompt) only achieves 2.31x because JMLC pipeline overhead becomes proportionally significant.

Cost comparison

All backends incur compute cost (hardware amortization + electricity) for the machine running them. GPU backends run on the H100 server; Ollama and OpenAI run on a local MacBook. OpenAI additionally incurs API cost per token.

How cost is calculated: compute_cost = wall_clock_time × (hardware_cost / lifetime_hours + power_watts × electricity_rate) / 3600. Assumptions: H100 server: 350W, $30K over 15K hours ($2.00/h + $0.105/h electricity = $2.105/h). MacBook: 50W, $3K over 15K hours ($0.20/h + $0.015/h electricity = $0.215/h). OpenAI API cost recorded by the runner from response headers (x-usage header).

Backend	Hardware	Wall clock (250 queries)	Compute cost	API cost	Total cost	Cost per query
Ollama llama3.2 3B	MacBook CPU	706s	$0.0422	--	$0.0422	$0.000169
OpenAI gpt-4.1-mini	MacBook + API	471s	$0.0281	$0.0573	$0.0855	$0.000342
vLLM Qwen 3B	H100 GPU	460s	$0.2688	--	$0.2688	$0.001076
SystemDS Qwen 3B c=1	H100 GPU	230s	$0.1345	--	$0.1345	$0.000538
SystemDS Qwen 3B c=4	H100 GPU	64s	$0.0372	--	$0.0372	$0.000149

OpenAI API cost breakdown (recorded per run): math $0.0227, reasoning $0.0172, json_extraction $0.0080, summarization $0.0076, embeddings $0.0019.

Conclusions

1. SystemDS llmPredict produces identical results to vLLM: SystemDS c=1 matches vLLM Qwen 3B accuracy exactly on all 5 workloads (68%, 52%, 60%, 50%, 90%). Both use the same model on the same inference server with temperature=0.0, confirming that the llmPredict DML built-in adds no distortion to model outputs.

2. Concurrency scales throughput 2.3-3.9x: The ExecutorService thread pool in the llmPredict instruction dispatches up to 4 requests concurrently. Longer-running workloads (math 3.71x, reasoning 3.95x) get closest to the theoretical 4x speedup. Short workloads (embeddings 2.31x) are limited by JMLC pipeline overhead.

3. OpenAI leads on accuracy but costs more per query: gpt-4.1-mini achieves the highest accuracy on all 5 workloads (94% math, 84% json, 70% reasoning, 88% summarization, 88% embeddings) but at $0.000342/query. SystemDS c=4 achieves $0.000149/query — 56% cheaper — with competitive accuracy on focused tasks like embeddings (90% vs 88%).

4. Model quality matters more than parameter count: Qwen 3B outperforms the larger Mistral 7B on math (68% vs 38%) and embeddings (90% vs 82%), while Mistral 7B is stronger on reasoning (68% vs 60%) and summarization (68% vs 50%). The serving framework (vLLM vs SystemDS) has zero impact on accuracy when using the same model.

5. Concurrency reduces compute cost on GPU: SystemDS c=4 at $0.000149/query is the cheapest GPU option — 86% less than vLLM's $0.001076/query — because higher throughput means less wall-clock time per query. Ollama on MacBook CPU is cheapest overall ($0.000169/query) due to low hardware and power costs, but 11x slower.

6. Latency measurements are not comparable across backends: Each backend uses a different HTTP client (Python streaming, Java non-streaming, cloud API) and measures time differently. Per-prompt latency should only be compared within the same backend across workloads, not across backends.