Skip to content

[SYNPY-1764] Add Trivy container vulnerability scanning #1346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BryanFauble wants to merge 1 commit into
base: add-claude-md
Choose a base branch
from
+404 −26
Open

[SYNPY-1764] Add Trivy container vulnerability scanning #1346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 121 additions & 26 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,19 +552,81 @@ jobs:
exit 1

# containerize the package and upload to the GHCR upon new release (whether pre-release or not)
ghcr-build-and-push-on-release:
# Step 1: Build the Docker image and save as tar for scanning
ghcr-build-on-release:
Copy link
Copy Markdown
Member Author

@BryanFauble BryanFauble Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the core architectural change — the old single ghcr-build-and-push-on-release job has been split into a 3-job pipeline that gates image publication on a Trivy vulnerability scan.

graph LR
    A[ghcr-build-on-release] -->|tar artifact| B[trivy-scan-release]
    B -->|pass| C[ghcr-push-on-release]
    B -->|fail: CRITICAL/HIGH found| D[Build stops — image NOT pushed]
Loading

One thing worth noting: the push job (ghcr-push-on-release) rebuilds the image from source rather than loading the tar artifact. This is because docker/build-push-action with load: true (used in the build job) is incompatible with cache-to: type=registry — they require different buildx drivers. The rebuild should be near-instant thanks to cache-from, and this lets us keep populating the registry build cache.

The tag computation moved from inline if: conditionals on two separate build steps into a single set-tags step that outputs the tag string, which the push job reads via needs.ghcr-build-on-release.outputs.image-tags.

Note: This comment was drafted with AI assistance and reviewed by me for accuracy.

needs: deploy
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
image-tags: ${{ steps.set-tags.outputs.tags }}
image-name: synapsepythonclient-release
env:
TARFILE_NAME: synapsepythonclient-release.tar

steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Extract Release Version
run: echo "RELEASE_VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_ENV
shell: bash
- name: Set image tags
id: set-tags
shell: bash
run: |
if [[ "${{ github.event.release.prerelease }}" == "true" ]]; then
echo "tags=ghcr.io/sage-bionetworks/synapsepythonclient:${{ env.RELEASE_VERSION }}-prerelease" >> $GITHUB_OUTPUT
else
echo "tags=ghcr.io/sage-bionetworks/synapsepythonclient:latest,ghcr.io/sage-bionetworks/synapsepythonclient:${{ env.RELEASE_VERSION }}" >> $GITHUB_OUTPUT
fi
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build Docker image
uses: docker/build-push-action@v5
with:
context: .
push: false
load: true
provenance: false
tags: synapsepythonclient-release:local
file: ./Dockerfile
platforms: linux/amd64
cache-from: type=registry,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache
- name: Save Docker image to tar
run: docker save synapsepythonclient-release:local -o ${{ env.TARFILE_NAME }}
- name: Upload tar artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.TARFILE_NAME }}
path: ${{ env.TARFILE_NAME }}
retention-days: 1

# Step 2: Scan the built image with Trivy before pushing
trivy-scan-release:
needs: [ghcr-build-on-release]
uses: ./.github/workflows/trivy.yml
with:
SOURCE_TYPE: tar
TARFILE_NAME: synapsepythonclient-release.tar
IMAGE_NAME: synapsepythonclient-release:local
EXIT_CODE: 1
permissions:
contents: read
security-events: write
actions: read

# Step 3: Push the image to GHCR only if Trivy scan passes
ghcr-push-on-release:
needs: [ghcr-build-on-release, trivy-scan-release]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Log in to GitHub Container Registry
Expand All @@ -573,39 +635,74 @@ jobs:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push Docker image (official release)
id: docker_build
if: '!github.event.release.prerelease'
uses: docker/build-push-action@v3
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
provenance: false
tags: ghcr.io/sage-bionetworks/synapsepythonclient:latest,ghcr.io/sage-bionetworks/synapsepythonclient:${{ env.RELEASE_VERSION }}
tags: ${{ needs.ghcr-build-on-release.outputs.image-tags }}
file: ./Dockerfile
platforms: linux/amd64
cache-from: type=registry,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache
cache-to: type=registry,mode=max,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache
- name: Build and push Docker image (pre-release)
id: docker_build_prerelease
if: 'github.event.release.prerelease'
uses: docker/build-push-action@v3

# containerize the package and upload to the GHCR upon commit in develop
# Step 1: Build the Docker image and save as tar for scanning
ghcr-build-on-develop:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/develop'
permissions:
contents: read
packages: write
outputs:
image-tag: ghcr.io/sage-bionetworks/synapsepythonclient:develop-${{ github.sha }}
image-name: synapsepythonclient-develop
env:
TARFILE_NAME: synapsepythonclient-develop.tar

steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build Docker image
uses: docker/build-push-action@v5
with:
push: true
context: .
push: false
load: true
provenance: false
tags: ghcr.io/sage-bionetworks/synapsepythonclient:${{ env.RELEASE_VERSION }}-prerelease
tags: synapsepythonclient-develop:local
file: ./Dockerfile
platforms: linux/amd64
cache-from: type=registry,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache-prerelease
cache-to: type=registry,mode=max,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache-prerelease
- name: Output image digest (official release)
if: '!github.event.release.prerelease'
run: echo "The image digest for official release is ${{ steps.docker_build.outputs.digest }}"
- name: Output image digest (pre-release)
if: 'github.event.release.prerelease'
run: echo "The image digest for pre-release is ${{ steps.docker_build_prerelease.outputs.digest }}"
cache-from: type=registry,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache
- name: Save Docker image to tar
run: docker save synapsepythonclient-develop:local -o ${{ env.TARFILE_NAME }}
- name: Upload tar artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.TARFILE_NAME }}
path: ${{ env.TARFILE_NAME }}
retention-days: 1

# Step 2: Scan the built image with Trivy before pushing
trivy-scan-develop:
needs: [ghcr-build-on-develop]
uses: ./.github/workflows/trivy.yml
with:
SOURCE_TYPE: tar
TARFILE_NAME: synapsepythonclient-develop.tar
IMAGE_NAME: synapsepythonclient-develop:local
EXIT_CODE: 1
permissions:
contents: read
security-events: write
actions: read

# containerize the package and upload to the GHCR upon commit in develop
ghcr-build-and-push-on-develop:
# Step 3: Push the image to GHCR only if Trivy scan passes
ghcr-push-on-develop:
needs: [ghcr-build-on-develop, trivy-scan-develop]
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/develop'
permissions:
Expand All @@ -623,16 +720,14 @@ jobs:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push Docker image for develop
id: docker_build
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
provenance: false
tags: ghcr.io/sage-bionetworks/synapsepythonclient:develop-${{ github.sha }}
file: ./Dockerfile
platforms: linux/amd64
cache-from: type=registry,ref=ghcr.io/sage-bionetworks/synapsepythonclient:build-cache
cache-to: type=inline
- name: Output image digest
run: echo "The image digest is ${{ steps.docker_build.outputs.digest }}"
103 changes: 103 additions & 0 deletions .github/workflows/docker_build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
Copy link
Copy Markdown
Member Author

@BryanFauble BryanFauble Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reusable workflow exists specifically for the periodic scan's auto-remediation path. It's a self-contained build-scan-push pipeline that the trivy_periodic_scan.yml workflow calls when it needs to rebuild an image after finding new vulnerabilities.

Unlike the build.yml push jobs (which rebuild from cache to maintain cache-to capability), this workflow uses the tar artifact approach end-to-end: the push-image job downloads the tar, loads it via docker load, then tags and pushes. This is fine here because there's no need to update the registry build cache during periodic remediation rebuilds.

The IMAGE_REFERENCES input is comma-separated, and the push step iterates over each tag — this lets the periodic scan push both a specific version tag and a major.minor tag in one go (e.g., 1.2.3 and 1.2).

Note: This comment was drafted with AI assistance and reviewed by me for accuracy.

#
# Reusable workflow to build, scan, and push a Docker image.
# Called by the periodic scan workflow to rebuild images
# when new vulnerabilities are found.
#
name: Build and publish a Docker image

on:
workflow_call:
inputs:
REF_TO_CHECKOUT:
required: false
type: string
description: "Reference to checkout, e.g. a tag like v1.0.1. Defaults to the branch/tag of the current event."
IMAGE_REFERENCES:
required: true
type: string
description: "Comma-separated image references, e.g., ghcr.io/sage-bionetworks/synapsepythonclient:1.0.1"

env:
TARFILE_NAME: image.tar
LOCAL_IMAGE_TAG: rebuild-image:local

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.REF_TO_CHECKOUT }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Build Docker image
uses: docker/build-push-action@v5
with:
context: .
push: false
load: true
tags: rebuild-image:local
file: ./Dockerfile
platforms: linux/amd64

- name: Save Docker image to tar
run: docker save rebuild-image:local -o ${{ env.TARFILE_NAME }}

- name: Upload tarball for use by Trivy job
uses: actions/upload-artifact@v4
with:
name: ${{ env.TARFILE_NAME }}
path: ${{ env.TARFILE_NAME }}
retention-days: 1

outputs:
tarfile_artifact: ${{ env.TARFILE_NAME }}

trivy-scan:
needs: build
uses: "./.github/workflows/trivy.yml"
with:
SOURCE_TYPE: tar
IMAGE_NAME: rebuild-image:local
TARFILE_NAME: ${{ needs.build.outputs.tarfile_artifact }}
EXIT_CODE: 1
permissions:
contents: read
security-events: write
actions: read

push-image:
needs: [build, trivy-scan]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Download tar artifact
uses: actions/download-artifact@v4
with:
name: ${{ needs.build.outputs.tarfile_artifact }}
path: /tmp

- name: Load Docker image from tar
run: docker load -i /tmp/${{ needs.build.outputs.tarfile_artifact }}

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Tag and push Docker image
shell: bash
run: |
IFS=',' read -ra TAGS <<< "${{ inputs.IMAGE_REFERENCES }}"
for TAG in "${TAGS[@]}"; do
docker tag rebuild-image:local "$TAG"
docker push "$TAG"
done
91 changes: 91 additions & 0 deletions .github/workflows/trivy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
Copy link
Copy Markdown
Member Author

@BryanFauble BryanFauble Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the central reusable scanning workflow — called from both build.yml (pre-push scan) and trivy_periodic_scan.yml (post-publish rescan). It supports two modes:

Mode SOURCE_TYPE How it gets the image Used by
Pre-push tar Downloads artifact from calling workflow, loads via docker load build.yml, docker_build.yml
Post-publish image Trivy pulls directly from GHCR trivy_periodic_scan.yml

The EXIT_CODE input controls whether findings fail the workflow (1) or just report (0). Both build.yml and the periodic scan use 1 so vulnerabilities are blocking.

The alternate Trivy DB repos (public.ecr.aws/aquasecurity/trivy-db:2) are important — the default GitHub-hosted DB gets rate-limited due to high download volume across the ecosystem.

SARIF results are uploaded even when Trivy finds vulnerabilities (the success() || steps.trivy.conclusion == 'failure' condition), so findings always land in the Security tab for triage regardless of whether the build passes.

Note: This comment was drafted with AI assistance and reviewed by me for accuracy.

#
# This workflow runs Trivy on a Docker image
# It can pull the image from a container registry
# or download a tar file. The latter is used
# to check a container image prior to publishing
# to the registry.

name: Run Trivy on a Docker image and push results to GitHub

on:
workflow_call:
inputs:
SOURCE_TYPE: # 'tar' or 'image'
required: true
type: string
TARFILE_NAME: # only used if SOURCE_TYPE=='tar'
required: false
type: string
IMAGE_NAME:
required: true
type: string
EXIT_CODE: # return code for failed scan. 0 means OK. Non-zero will fail the build when there are findings.
required: false
type: number
default: 0
outputs:
trivy_conclusion:
description: "The pass/fail status from Trivy"
value: ${{ jobs.trivy.outputs.trivy_conclusion }}

env:
sarif_file_name: trivy-results.sarif
# downloading the trivy-db from its default GitHub location fails because
# the site experiences too many downloads. The fix is to pull from this
# alternate location.
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db:1

jobs:
trivy:
name: Trivy
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Download tar file
id: tar-download
uses: actions/download-artifact@v4
if: ${{ inputs.SOURCE_TYPE == 'tar' }}
with:
name: ${{ inputs.TARFILE_NAME }}
path: /tmp

- name: Load docker image from tar file
if: ${{ inputs.SOURCE_TYPE == 'tar' }}
run: docker load -i ${{ steps.tar-download.outputs.download-path }}/${{ inputs.TARFILE_NAME }}

- name: Run Trivy vulnerability scanner for any major issues
uses: aquasecurity/trivy-action@0.32.0
Copy link
Copy Markdown
Contributor

@linglp linglp Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to try scan attack: https://sagebionetworks.jira.com/browse/SMR-703, we should also update the trivy version like this PR here: https://github.com/Sage-Bionetworks/sage-monorepo/pull/3951/changes

id: trivy
with:
image-ref: ${{ inputs.IMAGE_NAME }}
ignore-unfixed: true # skip vulnerabilities for which there is no fix
severity: 'CRITICAL,HIGH'
format: 'sarif'
limit-severities-for-sarif: true
output: ${{ env.sarif_file_name }}
exit-code: ${{ inputs.EXIT_CODE }}

- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v3.25.12
if: ${{ success() || steps.trivy.conclusion == 'failure' }}
with:
sarif_file: ${{ env.sarif_file_name }}
wait-for-processing: true

- name: Upload Trivy output
uses: actions/upload-artifact@v4
if: ${{ success() || steps.trivy.conclusion == 'failure' }}
with:
name: ${{ env.sarif_file_name }}
path: ${{ env.sarif_file_name }}

outputs:
trivy_conclusion: ${{ steps.trivy.conclusion }}
Loading
Loading