Rewrite tests to python for windows runtime testing#215
Rewrite tests to python for windows runtime testing#215julek-wolfssl wants to merge 11 commits intowolfSSL:mainfrom
Conversation
- Add shared test helper and cross-platform test runner - Extract binary lookup and run_wolfssl() into tests/wolfclu_test.py so all test files share the same logic for finding the wolfssl binary across Linux (./wolfssl) and Windows (x64/Debug/wolfssl.exe etc.). - Add tests/run_tests.py which discovers and runs all *-test.py files, intended for Windows where `make check` is not available. - Enable PKCS7 and CRL in MSVC - Fix three Windows bugs uncovered by the test: - Add StartTCP() (WSAStartup) in client and server setup so Winsock is initialized before gethostbyname/connect calls - Pass SNI hostname (-S flag) to the underlying client_test so modern TLS servers accept the connection - Implement checkStdin() for Windows using WaitForSingleObject so s_client exits promptly when stdin is a closed pipe - Update user_settings.h with defines required for TLS 1.3 and full wolfCLU functionality: WOLFSSL_TLS13, HAVE_HKDF, WC_RSA_PSS, HAVE_SUPPORTED_CURVES, HAVE_FFDHE_2048, HAVE_SNI. - Fix run_wolfssl() to use stdin=DEVNULL when no input is provided, preventing subprocesses from blocking on inherited stdin (e.g. over SSH/network sessions on Windows). - Also add WOLFCLU_SKIP_SLOW_TESTS env var to skip slow tests on Windows, and optimize large file creation to a single write. - Add HAVE_PKCS12 to Windows user_settings.h. Skip binary DER stdin test on Windows where pipe binary mode is unreliable. - Combine ocsp-test.sh and ocsp-interop-test.sh into a single Python test module that tests all client/responder combinations (wolfssl and openssl). - Add StartTCP() in OCSP setup for Winsock initialization on Windows. - Add HAVE_OCSP and HAVE_OCSP_RESPONDER to Windows user_settings.h. - Rewrite base64 test from bash to Python unittest - Rewrite bench test from bash to Python unittest - Rewrite client test to Python and fix Windows networking bugs - Replace tests/client/client-test.sh with a cross-platform Python unittest. - Rewrite dgst test from bash to Python unittest - Rewrite dh test from bash to Python unittest - Rewrite dsa test from bash to Python unittest - Rewrite enc test from bash to Python unittest - Rewrite genkey sign/verify test from bash to Python unittest - Rewrite hash test from bash to Python unittest - Rewrite pkcs7/pkcs8/pkcs12 tests from bash to Python unittest - Rewrite pkey/rsa/ecparam tests from bash to Python unittest - Rewrite rand test from bash to Python unittest - Rewrite server test from bash to Python unittest - Rewrite encdec test from bash to Python unittest - Rewrite x509/CRL tests from bash to Python unittest - Rewrite OCSP tests from bash to Python unittest - Rewrite OCSP SCGI test from bash to Python, drop nginx dependency - Replace nginx + bash with a pure-Python HTTP-to-SCGI proxy using stdlib http.server and raw sockets for the SCGI netstring protocol. No external dependencies needed. - Remove nginx from CI apt-get installs since it is no longer required for testing.
Replace fsanitize-check.yml, macos-check.yml, and ubuntu-check.yml with a single ci.yml that uses a matrix of OS, config, and sanitizer dimensions. Update all checkout actions to v4 and setup-msbuild to v2.
Simplify windows-check.yml: remove ${{env.GITHUB_WORKSPACE}} indirection,
drop PlatformToolset/WindowsTargetPlatformVersion overrides, add timeout,
pin wolfSSL ref, consolidate env vars. Update VS project to v145 toolset,
add missing source files to filters, enable multi-processor compilation.
There was a problem hiding this comment.
Pull request overview
This PR migrates the wolfCLU test suite from shell scripts to Python unittest modules to enable reliable runtime testing on Windows, while also updating Windows build configuration and CI workflows.
Changes:
- Replaced many
tests/**/*.shscripts with equivalenttests/**/*-test.pyPython unittest modules and added a Windows-friendlytests/run_tests.pyrunner. - Updated Automake integration to recognize
.pytests and added Python detection inconfigure.ac. - Updated Windows Visual Studio project/solution and CI workflows; added Windows networking initialization and SNI handling in the client setup.
Reviewed changes
Copilot reviewed 93 out of 94 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| wolfclu/clu_header_main.h | Undefines CRL reason macros before wolfSSL headers (Windows build hygiene). |
| wolfCLU.vcxproj.filters | Adds new source files to the Visual Studio filters list. |
| wolfCLU.vcxproj | Updates toolset and enables multiprocessor compilation. |
| wolfclu.sln | Updates VS version metadata and fixes x64 config mapping. |
| tests/x509/x509-verify-test.sh | Removed (replaced by Python unittest). |
| tests/x509/x509-verify-test.py | New Python unittest for wolfssl verify. |
| tests/x509/x509-req-test.sh | Removed (replaced by Python unittest). |
| tests/x509/x509-process-test.sh | Removed (replaced by Python unittest). |
| tests/x509/x509-ca-test.sh | Removed (replaced by Python unittest). |
| tests/x509/include.am | Registers Python x509 tests in Automake. |
| tests/x509/CRL-verify-test.sh | Removed (replaced by Python unittest). |
| tests/x509/CRL-verify-test.py | New Python unittest for wolfssl crl. |
| tests/wolfclu_test.py | Shared helper to locate wolfssl binary and run commands. |
| tests/testEncDec/test_aesctr.sh | Removed (replaced by Python unittest). |
| tests/testEncDec/test_aescbc_3des_cam.sh | Removed (replaced by Python unittest). |
| tests/testEncDec/README.md | Removed (test approach changed to self-contained Python). |
| tests/testEncDec/include.am | Registers new enc/dec Python test in Automake. |
| tests/testEncDec/encdec-test.py | New encrypt/decrypt round-trip tests for multiple ciphers. |
| tests/server/server-test.sh | Removed (replaced by Python unittest). |
| tests/server/server-test.py | New Python unittest for s_server/s_client handshake. |
| tests/server/include.am | Registers Python server test in Automake. |
| tests/run_tests.py | New Windows-oriented Python test runner discovering *-test.py. |
| tests/rand/rand-test.sh | Removed (replaced by Python unittest). |
| tests/rand/rand-test.py | New Python unittest for wolfssl rand. |
| tests/rand/include.am | Registers Python rand test in Automake. |
| tests/pkey/rsa-test.sh | Removed (replaced by Python unittest). |
| tests/pkey/rsa-test.py | New Python unittest for RSA key conversion/validation. |
| tests/pkey/pkey-test.sh | Removed (replaced by Python unittest). |
| tests/pkey/pkey-test.py | New Python unittest for pkey public/private conversions. |
| tests/pkey/include.am | Registers Python pkey tests in Automake. |
| tests/pkey/ecparam-test.sh | Removed (replaced by Python unittest). |
| tests/pkey/ecparam-test.py | New Python unittest for curve enumeration and key generation. |
| tests/pkcs/pkcs8-test.sh | Removed (replaced by Python unittest). |
| tests/pkcs/pkcs8-test.py | New Python unittest for PKCS8 conversion and error cases. |
| tests/pkcs/pkcs7-test.sh | Removed (replaced by Python unittest). |
| tests/pkcs/pkcs7-test.py | New Python unittest for PKCS7 conversions and printing certs. |
| tests/pkcs/pkcs12-test.sh | Removed (replaced by Python unittest). |
| tests/pkcs/pkcs12-test.py | New Python unittest for PKCS12 extraction options. |
| tests/pkcs/include.am | Registers Python pkcs tests in Automake. |
| tests/ocsp/ocsp-test.sh | Removed (replaced by Python unittest). |
| tests/ocsp/ocsp-test.py | New combined OCSP interop test suite in Python. |
| tests/ocsp/include.am | Registers Python OCSP test in Automake. |
| tests/ocsp-scgi/scgi_params | Removed (nginx dependency eliminated). |
| tests/ocsp-scgi/ocsp-scgi-test.py | New Python HTTP-to-SCGI proxy test replacing nginx. |
| tests/ocsp-scgi/include.am | Registers Python OCSP-SCGI test in Automake. |
| tests/hash/include.am | Switches hash tests from .sh to .py. |
| tests/hash/hash-test.sh | Removed (replaced by Python unittest). |
| tests/hash/hash-test.py | New Python unittest for -hash and shortcut hash commands. |
| tests/genkey_sign_ver/include.am | Switches genkey/sign/verify test registration to Python. |
| tests/genkey_sign_ver/genkey-sign-ver-test.sh | Removed (replaced by Python unittest). |
| tests/genkey_sign_ver/genkey-sign-ver-test.py | New Python unittest for keygen + sign/verify workflows incl. PQC. |
| tests/encrypt/include.am | Switches encrypt tests from .sh to .py. |
| tests/encrypt/enc-test.sh | Removed (replaced by Python unittest). |
| tests/encrypt/enc-test.py | New Python unittest for enc/dec, interop, and legacy names. |
| tests/dsa/include.am | Switches DSA tests from .sh to .py. |
| tests/dsa/dsa-test.sh | Removed (replaced by Python unittest). |
| tests/dsa/dsa-test.py | New Python unittest for dsaparam behaviors. |
| tests/dh/include.am | Switches DH tests from .sh to .py. |
| tests/dh/dh-test.sh | Removed (replaced by Python unittest). |
| tests/dh/dh-test.py | New Python unittest for dhparam behaviors. |
| tests/dgst/include.am | Switches dgst tests from .sh to .py. |
| tests/dgst/dgst-test.sh | Removed (replaced by Python unittest). |
| tests/dgst/dgst-test.py | New Python unittest for signature verification + large-file cases. |
| tests/client/include.am | Switches client test from .sh to .py. |
| tests/client/client-test.sh | Removed (replaced by Python unittest). |
| tests/client/client-test.py | New Python unittest for external TLS connection + x509 extraction. |
| tests/bench/include.am | Switches bench test from .sh to .py. |
| tests/bench/bench-test.sh | Removed (replaced by Python unittest). |
| tests/bench/bench-test.py | New Python unittest for -bench smoke tests. |
| tests/base64/include.am | Switches base64 test from .sh to .py. |
| tests/base64/base64-test.sh | Removed (replaced by Python unittest). |
| tests/base64/base64-test.py | New Python unittest for base64 encode/decode + stdin coverage. |
| src/x509/clu_x509_sign.c | Adjusts unsupported-hash handling in cert signing code path. |
| src/tools/clu_funcs.c | Gates AES-CTR help output on wolfSSL version macro. |
| src/server/clu_server_setup.c | Calls StartTCP() before running server test (Windows networking). |
| src/ocsp/clu_ocsp.c | Calls StartTCP() before OCSP client/responder execution. |
| src/crypto/clu_decrypt.c | Adjusts file-read loop behavior for decrypt path. |
| src/client/clu_client_setup.c | Adds SNI args and calls StartTCP() before running client test. |
| src/client/client.c | Implements Windows checkStdin() logic and enables it on Windows. |
| Makefile.am | Adds .py test extensions and includes new testEncDec include.am. |
| ide/winvs/user_settings.h | Enables SNI/TLS13/PKCS/CRL/OCSP features for Windows builds. |
| configure.ac | Adds Python discovery/substitution for Automake test harness. |
| .gitignore | Ignores Visual Studio build outputs and Python cache dirs. |
| .github/workflows/windows-check.yml | Modernizes Windows build + runs new Python test runner. |
| .github/workflows/ubuntu-check.yml | Removed (replaced by consolidated CI workflow). |
| .github/workflows/macos-check.yml | Removed (replaced by consolidated CI workflow). |
| .github/workflows/fsanitize-check.yml | Removed (replaced by consolidated CI workflow matrix). |
| .github/workflows/ci.yml | New consolidated Ubuntu/macOS matrix build + make check. |
| .gitattributes | Enforces LF for PEM/config files; marks DER as binary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def _ocsp_supported(binary): | ||
| """Check if the given binary supports OCSP.""" | ||
| try: | ||
| r = subprocess.run([binary, "ocsp", "-help"], | ||
| capture_output=True, timeout=5) | ||
| return True | ||
| except (FileNotFoundError, subprocess.TimeoutExpired): | ||
| return False |
There was a problem hiding this comment.
_ocsp_supported() currently returns True whenever the subprocess starts, regardless of whether binary ocsp -help succeeds. This will treat binaries without OCSP support as supported and can make tests fail later for the wrong reason. Return r.returncode == 0 (and possibly also check output for a known help/usage marker) instead of unconditional True.
tests/run_tests.py
Outdated
| runner = unittest.TextTestRunner(verbosity=2, durations=5) | ||
| result = runner.run(suite) | ||
| sys.exit(0 if result.wasSuccessful() else 1) |
There was a problem hiding this comment.
unittest.TextTestRunner doesn’t accept a durations parameter on many Python 3 versions (it was added only in relatively recent releases). This will raise a TypeError and prevent the Windows test runner from running on older-but-still-common Python installs. Consider dropping durations or gating it by checking the runner/testresult capabilities (or using the CLI python -m unittest --durations path).
| @classmethod | ||
| def setUpClass(cls): | ||
| if not os.path.isdir(CERTS_DIR): | ||
| raise unittest.SkipTest("certs directory not found") | ||
|
|
There was a problem hiding this comment.
HashShortcutTest.setUpClass() doesn’t include the filesystem-disabled skip check used elsewhere in the test suite (and present in the previous shell version). If wolfSSL is built with filesystem support disabled, these shortcut tests will still run and are likely to fail when trying to read CERT_FILE. Add the same config.log disable-filesystem detection and SkipTest here for consistent behavior.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 93 out of 94 changed files in this pull request and generated 11 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration"> | ||
| <ConfigurationType>Application</ConfigurationType> | ||
| <UseDebugLibraries>true</UseDebugLibraries> | ||
| <PlatformToolset>v143</PlatformToolset> | ||
| <PlatformToolset>v145</PlatformToolset> | ||
| </PropertyGroup> |
There was a problem hiding this comment.
PlatformToolset is now hard-coded to v145. The Windows CI workflow builds with an auto-detected toolset (mapping VS18 to v144), so local builds of this .vcxproj may fail unless v145 is installed. Consider using $(DefaultPlatformToolset) / removing the hard-coded toolset, or aligning the value with the workflow mapping.
| id: toolset | ||
| run: | | ||
| $major = (vswhere -latest -property installationVersion).Split('.')[0] | ||
| $ts = @{ "16" = "v142"; "17" = "v143"; "18" = "v144" }[$major] |
There was a problem hiding this comment.
Toolset detection maps VS major version 18 -> v144, but the checked-in wolfCLU.vcxproj hard-codes v145. If the toolset mapping changes or $ts resolves to $null (unexpected VS version), MSBuild will fail or build with a different toolset than the project file. Consider making the project file use the default toolset and/or adding a fallback/error if $ts is empty.
| $ts = @{ "16" = "v142"; "17" = "v143"; "18" = "v144" }[$major] | |
| $toolsets = @{ "16" = "v142"; "17" = "v143"; "18" = "v145" } | |
| $ts = $toolsets[$major] | |
| if (-not $ts) { | |
| Write-Error "Unsupported Visual Studio major version '$major'. Unable to determine PlatformToolset." | |
| exit 1 | |
| } |
| AC_PROG_LN_S | ||
| AC_PROG_MAKE_SET | ||
| AM_PROG_CC_C_O | ||
| AM_PATH_PYTHON([3.0],, [:]) |
There was a problem hiding this comment.
AM_PATH_PYTHON is set to silently succeed when Python ≥3.0 isn’t found (ACTION-IF-NOT-FOUND is ':'). Since the test suite now includes many .py tests, this can lead to confusing 'make check' failures later. Consider failing configure when Python 3 isn’t available, or conditionally disabling/registering the Python tests when PYTHON is empty.
| AM_PATH_PYTHON([3.0],, [:]) | |
| AM_PATH_PYTHON([3.0],, | |
| [AC_MSG_ERROR([Python 3.x is required to run the test suite but was not found. Please install Python >= 3.0 and re-run configure.])]) |
| @@ -19,6 +19,9 @@ dist_doc_DATA= | |||
| check_SCRIPTS= | |||
| dist_noinst_SCRIPTS= | |||
|
|
|||
There was a problem hiding this comment.
With TEST_EXTENSIONS including .py and PY_LOG_COMPILER relying on $(PYTHON), any environment where configure didn’t find Python (or where PYTHON is unset) may fail to run tests. Also, unittest-based skips typically exit 0, so automake will count them as PASS rather than SKIP (exit 77), potentially masking missing optional features. Consider requiring Python at configure time and/or providing a wrapper that maps 'all skipped' to exit 77 when appropriate.
| PYTHON ?= python3 |
| for path in candidates: | ||
| if os.path.isfile(path): | ||
| return path | ||
|
|
||
| # Fall back to first candidate; tests will get a clear FileNotFoundError | ||
| return candidates[0] |
There was a problem hiding this comment.
_find_wolfssl_bin() falls back to returning a non-existent candidate path (line 28) when the binary isn’t found, so run_wolfssl() will raise FileNotFoundError. Several test modules call run_wolfssl at import time, which can break test discovery with an unhandled exception. Consider raising unittest.SkipTest (or a clearer exception) when the binary is missing, and/or deferring any run_wolfssl calls until setUpClass.
| def _available_algos(): | ||
| """Parse available algorithms from -encrypt -help output.""" | ||
| r = run_wolfssl("-encrypt", "-help") | ||
| combined = r.stdout + r.stderr | ||
| algos = set() | ||
| for line in combined.splitlines(): | ||
| for token in line.split(): | ||
| if "-" in token and any(c.isdigit() for c in token): | ||
| algos.add(token) | ||
| return algos | ||
|
|
||
|
|
||
| _ALGOS = _available_algos() | ||
|
|
There was a problem hiding this comment.
_ALGOS is computed at import time by spawning wolfssl -encrypt -help. If the wolfssl binary isn’t present (or is not runnable in the current environment), this will raise and prevent the entire test suite from being discovered. Consider deferring algorithm detection to setUpClass (and skipping cleanly on failure) so import-time side effects don’t break discovery.
| @unittest.skipIf(_is_fips(), "skipped in FIPS builds") | ||
| def test_encrypted_key(self): | ||
| r = run_wolfssl("rsa", "-in", | ||
| os.path.join(CERTS_DIR, "server-keyEnc.pem"), | ||
| "-passin", "pass:yassl123") | ||
| self.assertEqual(r.returncode, 0, r.stderr) | ||
|
|
||
| @unittest.skipIf(_is_fips(), "skipped in FIPS builds") | ||
| def test_fail_wrong_password(self): | ||
| r = run_wolfssl("rsa", "-in", | ||
| os.path.join(CERTS_DIR, "server-keyEnc.pem"), | ||
| "-passin", "pass:yassl12") | ||
| self.assertNotEqual(r.returncode, 0) | ||
|
|
||
| @unittest.skipIf(_is_fips(), "skipped in FIPS builds") | ||
| def test_modulus_noout(self): |
There was a problem hiding this comment.
The @skipIf decorators call _is_fips() at import time, which runs wolfssl -v during test discovery (before setUpClass checks/skip logic). This can slow discovery and can hard-fail if the wolfssl binary isn’t available yet. Prefer computing the FIPS flag once in setUpClass (as you already do with cls.is_fips) and using self.skipTest(...) inside the relevant tests.
| def test_nocerts(self): | ||
| r = subprocess.run( | ||
| [WOLFSSL_BIN, "pkcs12", "-nodes", "-nocerts", | ||
| "-passin", "stdin", "-passout", "pass:", "-in", P12_FILE], | ||
| input=b"wolfSSL test", capture_output=True, text=False, | ||
| timeout=60, | ||
| ) |
There was a problem hiding this comment.
For the -passin stdin cases, the subprocess input is b"wolfSSL test" without a trailing newline. If the pkcs12 implementation reads the password as a line (common for CLI prompts), this can cause hangs or failed reads on some platforms. Consider sending b"wolfSSL test\n" (and apply consistently to all stdin password tests in this file).
| # Start server in background | ||
| server = subprocess.Popen( | ||
| [WOLFSSL_BIN, "s_server", "-port", "11111", | ||
| "-key", os.path.join(CERTS_DIR, "server-key.pem"), | ||
| "-cert", os.path.join(CERTS_DIR, "server-cert.pem"), | ||
| "-noVerify", "-readyFile", readyfile], | ||
| stdout=subprocess.PIPE, stderr=subprocess.PIPE, | ||
| stdin=subprocess.DEVNULL, | ||
| ) |
There was a problem hiding this comment.
The s_server process is started with stdout/stderr set to PIPE but its output is never read. If s_server is verbose (or emits repeated warnings), the pipe buffers can fill and block the server process, making this test hang/flaky. Consider redirecting stdout/stderr to DEVNULL (or a temp file) unless the output is actively consumed for assertions.
| /* Set SNI hostname so modern servers accept the connection */ | ||
| if (ret == WOLFCLU_SUCCESS && host != NULL) { | ||
| ret = _addClientArg(clientArgv, sniFlag, &clientArgc); | ||
| if (ret == WOLFCLU_SUCCESS) { | ||
| ret = _addClientArg(clientArgv, host, &clientArgc); | ||
| } | ||
| } |
There was a problem hiding this comment.
The new automatic SNI insertion uses host directly, but host can be an IP literal (IPv4/IPv6) or include an IPv6 scope id (e.g. %en0). Sending those values as SNI is invalid and can cause handshake failures on strict servers. Consider only adding -S <hostname> when the connect target is a DNS name (and stripping any scope id), or make SNI opt-in via a CLI flag.
No description provided.