Skip to content

OCPBUGS-65845: fix stale image digest race condition#504

Open
joepvd wants to merge 1 commit intoopenshift:mainfrom
joepvd:fix/OCPBUGS-65845-stale-image-digest
Open

OCPBUGS-65845: fix stale image digest race condition#504
joepvd wants to merge 1 commit intoopenshift:mainfrom
joepvd:fix/OCPBUGS-65845-stale-image-digest

Conversation

@joepvd
Copy link

@joepvd joepvd commented Mar 9, 2026

Summary

Fixes OCPBUGS-65845 — a race condition where the builder uses an older image digest instead of the latest one.

  • Change default pull policy from PullIfMissing to PullIfNewer in both buildDaemonlessImage and extractSourceFromImage. With PullIfMissing, if an image was already pulled (e.g. during a pre-pull step), a newer version pushed to the registry would not be detected. PullIfNewer adds a lightweight registry manifest check that catches cases where a tag was updated between Build object creation and the actual pull. For digest-pinned references this is a no-op (a specific digest cannot have a "newer" version).
  • Fix DaemonlessClient.PullImage digest reference reconstruction. When the tag field contains a digest (e.g. sha256:...), use @ instead of : as the separator, preventing malformed image names like image:sha256:abc instead of the correct image@sha256:abc.
  • Add tests validating image name reconstruction for both tag and digest references, and verifying DockerBuilder.pullImage correctly passes digest references to the docker client.

Test plan

  • TestPullImageNameReconstruction — validates image name reconstruction logic handles tags, digests-as-tags, digests-in-repository, and no-tag cases
  • TestPullImageDigestReference — validates DockerBuilder.pullImage correctly passes digest and tag references through to the docker client
  • Full test suite passes (only pre-existing TestCheckRemoteGit failure due to network access)

Summary by CodeRabbit

  • Improvements

    • Default image pull behavior updated: images now pull only when newer versions are available, reducing unnecessary operations
    • Image reference handling refined for improved compatibility across different image specification formats
  • Tests

    • Added comprehensive test coverage for image pull option parsing with various reference types and registry configurations

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 9, 2026
@openshift-ci-robot
Copy link
Contributor

@joepvd: This pull request references Jira Issue OCPBUGS-65845, which is invalid:

  • expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Fixes OCPBUGS-65845 — a race condition where the builder uses an older image digest instead of the latest one.

  • Change default pull policy from PullIfMissing to PullIfNewer in both buildDaemonlessImage and extractSourceFromImage. With PullIfMissing, if an image was already pulled (e.g. during a pre-pull step), a newer version pushed to the registry would not be detected. PullIfNewer adds a lightweight registry manifest check that catches cases where a tag was updated between Build object creation and the actual pull. For digest-pinned references this is a no-op (a specific digest cannot have a "newer" version).
  • Fix DaemonlessClient.PullImage digest reference reconstruction. When the tag field contains a digest (e.g. sha256:...), use @ instead of : as the separator, preventing malformed image names like image:sha256:abc instead of the correct image@sha256:abc.
  • Add tests validating image name reconstruction for both tag and digest references, and verifying DockerBuilder.pullImage correctly passes digest references to the docker client.

Test plan

  • TestPullImageNameReconstruction — validates image name reconstruction logic handles tags, digests-as-tags, digests-in-repository, and no-tag cases
  • TestPullImageDigestReference — validates DockerBuilder.pullImage correctly passes digest and tag references through to the docker client
  • Full test suite passes (only pre-existing TestCheckRemoteGit failure due to network access)

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link

coderabbitai bot commented Mar 9, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c58f1531-2944-4edb-82fa-d71b8fe3f83b

📥 Commits

Reviewing files that changed from the base of the PR and between b1e7909 and 254d97c.

📒 Files selected for processing (3)
  • pkg/build/builder/daemonless.go
  • pkg/build/builder/docker_test.go
  • pkg/build/builder/source.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/build/builder/source.go

Walkthrough

These changes modify the default image pull policy from PullIfMissing to PullIfNewer in daemonless and source builders, refine how tagged image references are constructed in daemonless image pulls, and add comprehensive unit test coverage for Docker image pull option parsing.

Changes

Cohort / File(s) Summary
Pull Policy Defaults
pkg/build/builder/daemonless.go, pkg/build/builder/source.go
Default image pull policy changed from PullIfMissing to PullIfNewer. When opts.Pull is true or forcePull is enabled, behavior remains PullAlways.
Image Reference Handling
pkg/build/builder/daemonless.go
Refined PullImage reference construction: when a tag is provided, if the tag contains a colon, the image reference uses @ (digest-style) separator; otherwise, colon separator is used.
Docker Image Pull Tests
pkg/build/builder/docker_test.go
Added table-driven unit test TestPullImageDigestReference covering tag references, digest references, simple tags, and no-tag cases with Docker client mocking.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ❓ Inconclusive The custom check is designed for Ginkgo test code, but this PR uses standard Go testing with testing.T and table-driven patterns instead. Clarify whether test quality criteria apply to all frameworks or only Ginkgo. The existing test code follows good practices but uses a different testing framework than the check expects.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the bug fix (OCPBUGS-65845) and the main issue being addressed (stale image digest race condition), which directly aligns with the changeset's core purpose of fixing the race condition by adjusting pull policies and fixing image-name reconstruction.
Stable And Deterministic Test Names ✅ Passed New test TestPullImageDigestReference uses stable, deterministic Go test naming without dynamic values like pod names, timestamps, UUIDs, or IP addresses.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci bot requested review from prabhapa and sayan-biswas March 9, 2026 15:23
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 9, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: joepvd
Once this PR has been reviewed and has the lgtm label, please assign moebasim for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/build/builder/daemonless_test.go (1)

18-75: Consider calling the actual implementation rather than duplicating the logic in the test.

The test implements the reconstruction logic inline (lines 62-69) instead of invoking DaemonlessClient.PullImage. This validates the expected behavior but doesn't verify the actual implementation matches.

♻️ Suggested approach to test the actual implementation
 func TestPullImageNameReconstruction(t *testing.T) {
+	// Track the image name passed to pullDaemonlessImage
+	var capturedImageName string
+	// Note: This would require either:
+	// 1. Mocking pullDaemonlessImage (which is difficult without interfaces)
+	// 2. Or keeping the current approach as a specification test
+	
 	tests := []struct {
 		name string
 		repo string
 		tag  string
 		want string
 	}{
 		// ... test cases ...
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			opts := docker.PullImageOptions{
 				Repository: tt.repo,
 				Tag:        tt.tag,
 			}
-			imageName := opts.Repository
-			if opts.Tag != "" {
-				if strings.Contains(opts.Tag, ":") {
-					imageName = imageName + "@" + opts.Tag
-				} else {
-					imageName = imageName + ":" + opts.Tag
-				}
-			}
+			// Ideally call DaemonlessClient.PullImage and capture the imageName
+			// For now, the inline logic serves as a specification test

Given the testing constraints (mocking pullDaemonlessImage would require significant refactoring), the current approach is acceptable as a specification test that documents expected behavior.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/build/builder/daemonless_test.go` around lines 18 - 75, The test
TestPullImageNameReconstruction duplicates the image-name reconstruction logic
instead of exercising the real implementation; update the test to call the
actual code path (e.g., invoke DaemonlessClient.PullImage or the internal helper
used by pullDaemonlessImage) so the behavior is validated against the
implementation — either refactor the reconstruction into an exported helper
(e.g., ReconstructImageName or similar) and test that helper, or construct a
DaemonlessClient with minimal/mocked dependencies and call
DaemonlessClient.PullImage for each case to assert the produced image reference
matches the expected value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/build/builder/daemonless_test.go`:
- Around line 18-75: The test TestPullImageNameReconstruction duplicates the
image-name reconstruction logic instead of exercising the real implementation;
update the test to call the actual code path (e.g., invoke
DaemonlessClient.PullImage or the internal helper used by pullDaemonlessImage)
so the behavior is validated against the implementation — either refactor the
reconstruction into an exported helper (e.g., ReconstructImageName or similar)
and test that helper, or construct a DaemonlessClient with minimal/mocked
dependencies and call DaemonlessClient.PullImage for each case to assert the
produced image reference matches the expected value.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cd6c074d-971a-4bd9-8b0a-7a9fd0291356

📥 Commits

Reviewing files that changed from the base of the PR and between 711f2e6 and b1e7909.

📒 Files selected for processing (4)
  • pkg/build/builder/daemonless.go
  • pkg/build/builder/daemonless_test.go
  • pkg/build/builder/docker_test.go
  • pkg/build/builder/source.go

@joepvd
Copy link
Author

joepvd commented Mar 9, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Mar 9, 2026
@openshift-ci-robot
Copy link
Contributor

@joepvd: This pull request references Jira Issue OCPBUGS-65845, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @xiuwang

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 9, 2026
@openshift-ci openshift-ci bot requested a review from xiuwang March 9, 2026 19:51
Change the default pull policy from PullIfMissing to PullIfNewer in
buildDaemonlessImage and extractSourceFromImage. With PullIfMissing, if
an image was already pulled (e.g. during a pre-pull step), a newer
version pushed to the registry would not be detected. PullIfNewer adds a
lightweight registry manifest check that catches cases where a tag was
updated between Build object creation and the actual pull. For
digest-pinned references this is a no-op.

Also fix DaemonlessClient.PullImage to use '@' instead of ':' when the
tag field contains a digest (e.g. "sha256:..."), preventing incorrect
image name reconstruction.
@joepvd joepvd force-pushed the fix/OCPBUGS-65845-stale-image-digest branch from b1e7909 to 254d97c Compare March 10, 2026 07:49
@openshift-ci-robot
Copy link
Contributor

@joepvd: This pull request references Jira Issue OCPBUGS-65845, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @xiuwang

Details

In response to this:

Summary

Fixes OCPBUGS-65845 — a race condition where the builder uses an older image digest instead of the latest one.

  • Change default pull policy from PullIfMissing to PullIfNewer in both buildDaemonlessImage and extractSourceFromImage. With PullIfMissing, if an image was already pulled (e.g. during a pre-pull step), a newer version pushed to the registry would not be detected. PullIfNewer adds a lightweight registry manifest check that catches cases where a tag was updated between Build object creation and the actual pull. For digest-pinned references this is a no-op (a specific digest cannot have a "newer" version).
  • Fix DaemonlessClient.PullImage digest reference reconstruction. When the tag field contains a digest (e.g. sha256:...), use @ instead of : as the separator, preventing malformed image names like image:sha256:abc instead of the correct image@sha256:abc.
  • Add tests validating image name reconstruction for both tag and digest references, and verifying DockerBuilder.pullImage correctly passes digest references to the docker client.

Test plan

  • TestPullImageNameReconstruction — validates image name reconstruction logic handles tags, digests-as-tags, digests-in-repository, and no-tag cases
  • TestPullImageDigestReference — validates DockerBuilder.pullImage correctly passes digest and tag references through to the docker client
  • Full test suite passes (only pre-existing TestCheckRemoteGit failure due to network access)

Summary by CodeRabbit

  • Improvements
  • Default image pull behavior updated: images now pull only when newer versions are available, reducing unnecessary operations
  • Image reference handling refined for improved compatibility across different image specification formats
  • Tests
  • Added comprehensive test coverage for image pull option parsing with various reference types and registry configurations

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 10, 2026

@joepvd: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/security 254d97c link false /test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants