Skip to content

fix: don't count DNS errors against consecutive failure limit#1139

Draft
vringar wants to merge 1 commit intomasterfrom
fix/dns-crash-handling
Draft

fix: don't count DNS errors against consecutive failure limit#1139
vringar wants to merge 1 commit intomasterfrom
fix/dns-crash-handling

Conversation

@vringar
Copy link
Contributor

@vringar vringar commented Feb 23, 2026

Summary

  • DNS resolution errors (dnsNotFound) are expected when crawling large domain lists and don't indicate browser/instrumentation failure
  • Skip failure counter increment and browser restart for DNS errors, preventing premature crawl termination
  • DNS errors are still properly recorded in crawl_history with status neterror

Closes #1116

Changes

  • openwpm/browser_manager.py: Add is_dns_error check before incrementing failure_count and triggering browser restart

Test plan

  • Run test/test_webdriver_utils.py::test_parse_neterror_integration to verify DNS errors are still recorded correctly
  • Verify crawls with unreachable domains don't trigger browser restarts
  • Run full test suite to check for regressions

@vringar vringar force-pushed the fix/dns-crash-handling branch from e3f6f4c to e543f83 Compare February 26, 2026 23:20
@codecov
Copy link

codecov bot commented Feb 27, 2026

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.74%. Comparing base (12fae22) to head (79ce7fe).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
openwpm/browser_manager.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1139      +/-   ##
==========================================
- Coverage   56.75%   56.74%   -0.02%     
==========================================
  Files          40       40              
  Lines        3908     3909       +1     
==========================================
  Hits         2218     2218              
- Misses       1690     1691       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

When crawling large domain lists (e.g. Tranco top 100k), many domains
cannot be resolved via DNS. These dnsNotFound neterrors are expected and
do not indicate a browser or instrumentation failure. Previously they
were counted against MAX_CONSECUTIVE_FAILURES, eventually crashing the
crawl.

Skip the failure counter increment and browser restart for dnsNotFound
neterrors so that DNS resolution failures no longer crash large crawls.

Fixes #1116
@vringar vringar force-pushed the fix/dns-crash-handling branch from e543f83 to 79ce7fe Compare March 2, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consecutive DNS errors crash a Tranco crawl

1 participant