Tester support for Javascript programming language.#698
Tester support for Javascript programming language.#698Karl-Michaud wants to merge 49 commits intoMarkUsProject:masterfrom
Conversation
Co-Authored-By: lizzie-liu <lizzie-liu@users.noreply.github.com>
for more information, see https://pre-commit.ci
Completed `js_tester` logic
for more information, see https://pre-commit.ci
Co-Authored-By: freyazjiner <freyazjiner@users.noreply.github.com>
Add requirements.system to install Node.js for JS tester
Js tester skeleton
for more information, see https://pre-commit.ci
…s-autotesting into freya-js-autotester
for more information, see https://pre-commit.ci
Use pnpm for JS tester; install pnpm and Jest globally
Test Group Timeout
for more information, see https://pre-commit.ci
Add config and package.json to fix the issue
for more information, see https://pre-commit.ci
README.md
Outdated
| - `java` | ||
| - openjdk-8-jdk | ||
| - `js` (JavaScript) | ||
| - Node.js |
There was a problem hiding this comment.
this isn't exactly right, as "package" here means "system package installed through apt-get"; you can just say "none"
| class JsTestData(BaseTestData, kw_only=True): | ||
| """The `test_data` specification for the JavaScript tester.""" | ||
|
|
||
| timeout: Annotated[int, Meta(title="Test group timeout (seconds)")] = 30 |
There was a problem hiding this comment.
I don't think this is necessary as it should already be inherited from BaseTestData
| if ! command -v node &> /dev/null; then | ||
| apt-get -y update | ||
| DEBIAN_FRONTEND=noninteractive apt-get install -y -o 'Dpkg::Options::=--force-confdef' -o 'Dpkg::Options::=--force-confold' curl ca-certificates | ||
| curl -fsSL https://deb.nodesource.com/setup_22.x | bash - |
There was a problem hiding this comment.
The latest LTS is 24 (not 22), so let's install that
| fi | ||
|
|
||
| if ! command -v pnpm &> /dev/null; then | ||
| npm install -g pnpm |
There was a problem hiding this comment.
Use corepack to install (https://pnpm.io/installation#using-corepack).
| fi | ||
|
|
||
| if ! command -v jest &> /dev/null; then | ||
| npm install -g jest |
There was a problem hiding this comment.
We really should be able to use pnpm to install packages globally. Modify this to use pnpm, and if you still see an error after resolving my other comments, you should do some more debugging. Remember that you can run commands in a test docker container directly (docker compose run --rm server bash); you can also experiment doing so as different users. This script is being run as root.
| --runInBand: run all tests serially in the current process | ||
| """ | ||
| _js_dir = os.path.dirname(os.path.realpath(__file__)) | ||
| config_path = os.path.join(_js_dir, "jest.config.json") |
There was a problem hiding this comment.
We don't need to provide a default jest.config.json, this is up to the instructor (or student) to provide.
In the MarkUs repo PR, please include an example jest.config.json file for the autotest scripts and verify that it is being used when running tests.
| test_data = self.specs.get("test_data", default={}) or {} | ||
|
|
||
| self._ensure_package_json(dir_path) | ||
| timeout = test_data.get("timeout", 60) |
There was a problem hiding this comment.
the default here should be the same as the actual default in the schema (30)
| ) | ||
| return result | ||
|
|
||
| def _ensure_package_json(self, dir_path): |
There was a problem hiding this comment.
This function isn't necessary. If a package.json file exists, you can run the pnpm install command, but otherwise just skip it altogether.
You can extend the MarkUs example to actually have two different assignments, one without a package.json file, and one with a package.json file (that actually contains dependencies that are required for the project).
…s-autotesting into freya-js-autotester
for more information, see https://pre-commit.ci
4 participants
Summary
This PR adds a new JavaScript tester to the autotesting framework, following the structure of the existing testers.
How it works
When tests are run, the tester:
pnpm installin the current working directory to install dependencies frompackage.json--jsonto get machine-readable output,--forceExitto prevent hanging, and--runInBandto run all tests serially in the current process (important for the shared-resource production environment)pending(Jest's term for skipped tests)Design decisions
pnpm over npm: As indicated in feedback, pnpm is faster and more space-efficient. Node.js, pnpm, and Jest are all installed globally as part of tester installation (e.g.;
requirements.system) since they do not need to change on a per-environment basis.Per-test timeout: Jest's per-test timeout can be configured by instructors via a
jest.config.jsfile. It is not part of the schema.Test group timeout: A
timeoutfield (default 30s) is included in the schema to set a hard wall-clock limit on the entire Jest run for a test group. This matches how other testers handle group-level timeouts.Node.js installation: The Debian/Ubuntu package repositories ship outdated versions of Node.js. We use the NodeSource setup script to install Node.js 22, which ensures compatibility with current versions of Jest and JS ecosystem.
Using npm to install Jest and pnpm: In
requirements.system, npm is used to install both Jest and pnpm globally. pnpm lacks root privileges during system setup, sopnpm add -g jestfails with a permission error. Two options were considered: (1)npm install -gsince npm runs with elevated privileges (chosen); (2) install viacurland manually update the bin path. pnpm is kept for all tester logic where it runs as the user.create_environment: There is nothing to set up per environment for the JS tester, sincepnpm installruns at test time (each student submission may have its ownpackage.json). The function just returns the standardPYTHONenv var.