Fix UCP pod CrashLoopBackOff caused by startup race condition

[Copilot]

Description

UCP pod intermittently enters CrashLoopBackOff when the Kubernetes API server isn't fully ready at startup. Three compounding issues:

1. DatabaseProvider permanently caches initialization errors — if the first GetClient() call fails (API server not ready), the error is cached forever. All subsequent calls from every service return the same cached error with no retry possible.

2. Initializer service has no retry logic — a single transient failure during manifest registration kills the service, which triggers RunWithInterrupts to shut down the entire UCP process.

3. RunWithInterrupts swallows the error — the service error is logged but discarded; host.Run() returns nil after cancellation, so the process exits with code 0. Kubernetes sees a clean exit that keeps happening → CrashLoopBackOff.

Changes

• DatabaseProvider.initialize() — transient factory errors are no longer cached in p.result. Only successful initialization and permanent config errors (unsupported provider) are cached. Callers can retry on next call.

• Initializer Run() — wraps manifest registration in exponential backoff retry (up to 2 min) via sethvargo/go-retry, already a project dependency. Core logic extracted to registerManifests().

• RunWithInterrupts() — captures the LifecycleMessage.Err from the failed service and returns it when host.Run() returns nil. Adds defer cancel() and distinguishes service failure from normal shutdown in logs.

• Tests — Test_GetClient_CachedError replaced with Test_GetClient_RetryAfterError (verifies retry-then-succeed). Two new initializer tests cover transient error retry and context cancellation during retry.

Type of change

• This pull request fixes a bug in Radius and has an approved issue (issue link required).

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

• An overview of proposed schema changes is included in a linked GitHub issue.
  • Yes
  • Not applicable
• A design document PR is created in the design-notes repository, if new APIs are being introduced.
  • Yes
  • Not applicable
• The design document has been reviewed and approved by Radius maintainers/approvers.
  • Yes
  • Not applicable
• A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
  • Yes
  • Not applicable
• A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
  • Yes
  • Not applicable
• A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.
  • Yes
  • Not applicable

[github-actions]

Unit Tests

    2 files  ±0    415 suites  ±0   6m 29s ⏱️ -17s
4 874 tests +2  4 872 ✅ +2  2 💤 ±0  0 ❌ ±0 
5 776 runs  +2  5 774 ✅ +2  2 💤 ±0  0 ❌ ±0 

Results for commit de07398. ± Comparison against base commit 6d326a9.

This pull request removes 1 and adds 3 tests. Note that renamed tests count towards both.

github.com/radius-project/radius/pkg/components/database/databaseprovider ‑ Test_GetClient_CachedError

github.com/radius-project/radius/pkg/components/database/databaseprovider ‑ Test_GetClient_RetryAfterError
github.com/radius-project/radius/pkg/ucp/initializer ‑ Test_Run/retries_on_transient_database_error
github.com/radius-project/radius/pkg/ucp/initializer ‑ Test_Run/returns_error_when_context_is_cancelled_during_retry

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.38%. Comparing base (6d326a9) to head (de07398).

Files with missing lines	Patch %	Lines
pkg/components/hosting/run.go	0.00%	11 Missing ⚠️
...nents/database/databaseprovider/storageprovider.go	33.33%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11640      +/-   ##
==========================================
+ Coverage   51.37%   51.38%   +0.01%     
==========================================
  Files         699      699              
  Lines       44111    44132      +21     
==========================================
+ Hits        22661    22677      +16     
- Misses      19280    19287       +7     
+ Partials     2170     2168       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.