Skip to content

Add 1h pre-aggregated data, improve dashboards, and speed up web queries#27

Merged
cchwala merged 5 commits intomainfrom
feature/performance-and-grafana-improvements
Mar 25, 2026
Merged

Add 1h pre-aggregated data, improve dashboards, and speed up web queries#27
cchwala merged 5 commits intomainfrom
feature/performance-and-grafana-improvements

Conversation

@cchwala
Copy link
Copy Markdown
Member

@cchwala cchwala commented Mar 25, 2026

Changes

  • TimescaleDB: Added cml_data_1h continuous aggregate (min/max/avg per sublink per hour) with an auto-refresh policy, reducing query load for historical data.
  • Grafana real-time dashboard: Rewrote RSL/TSL panels to show a shaded min/max band with mean line. Automatically switches between raw data (≤3 days) and 1h aggregates (>3 days). Added an interval variable (Auto/Raw). Removed Altair dependency.
  • Grafana archive dashboard: New cml-archive dashboard with two bar chart panels (active sublinks/hour and approximate data points/hour), embedded in the archive page via iframe.
  • Webserver performance: Replaced full-table COUNT(*) with approximate_row_count(), replaced MIN/MAX(time) FROM cml_data with MIN/MAX(bucket) FROM cml_data_1h, and removed the per-CML GROUP BY stats query. Fully removed pandas and Altair from the webserver.

cchwala added 3 commits March 25, 2026 17:33
- Add TimescaleDB continuous aggregate cml_data_1h (1h min/max/avg)
- Remove Altair time-series from real-time route; keep for archive only
- Grafana: show 1h min/max band always, avg/raw switching on zoom level
- Grafana: per-sublink colors, interval selector (Auto/Raw)
- Replace COUNT(*) with approximate_row_count() for record totals
- Replace MIN/MAX(time) FROM cml_data with MIN/MAX(bucket) FROM cml_data_1h
- Replace per-CML COUNT GROUP BY with cml_data_1h aggregate estimate
- Remove generate_archive_charts() and pandas/altair imports
- Add cml-archive Grafana dashboard (active sublinks + data points per hour)
- Simplify archive page: remove header, metric cards and top-CML table
- Add compact summary bar with record count, CML count, date range
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 25, 2026

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.18%. Comparing base (d136099) to head (5067a8a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
webserver/main.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
+ Coverage   71.44%   73.18%   +1.73%     
==========================================
  Files          22       22              
  Lines        2021     1965      -56     
==========================================
- Hits         1444     1438       -6     
+ Misses        577      527      -50     
Flag Coverage Δ
mno_simulator 84.18% <ø> (ø)
parser 77.91% <ø> (ø)
webserver 49.08% <0.00%> (+4.34%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cchwala
Copy link
Copy Markdown
Member Author

cchwala commented Mar 25, 2026

Tested also on the machine which now was running the last main for 20 days and contains 125 million data records. The two steps from the migration document worked, but took several minutes to finish.

The real-time site takes approx a second to load, which is not perfect but good enough for now and for the time being. Switching between CMLs by clicking on them on the maps works instantly. It looks like that with 20 days of data:

Bildschirmfoto 2026-03-25 um 21 40 16

@cchwala cchwala merged commit 4dcf26e into main Mar 25, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant