Draft
Conversation
Contributor
Author
Progress Update — 2026-03-30YAML Library Migration:
|
| Area | Status |
|---|---|
| Go 1.26 upgrade | Done |
| Dependency updates (x/, consul, oauth2, docker) | Done |
| Cobra CLI decomposition | Done |
| yaml.v2 → yaml.v3 migration | Done |
| yaml.v3 → goccy/go-yaml migration | Done |
| Ginkgo v1 → v2 upgrade | Done |
| Multi-database dialect adapter (PostgreSQL + MySQL) | Done |
| go vet clean build | Done |
…ments to SHIELD auth flow (#794) * Add upfront permission check to Postgres restore Introduces a checkRestorePermissions function to verify that the user has sufficient privileges before starting a restore operation. This helps prevent restore failures due to inadequate permissions by checking for superuser status or database creation rights. * Add option to skip Postgres permission check Introduces a new 'pg_skip_permission_check' boolean field to allow skipping upfront permission validation during restore. This is useful for advanced users who understand the risks and want to bypass permission checks, with appropriate warnings in the help text. The PostgresConnectionInfo struct and related logic are updated to support this option. * Fix permission check and add postgres.exe binary Corrected the method call to BooleanValueDefault for 'pg_skip_permission_check' in plugin.go and made minor formatting improvements. Added the postgres.exe binary to the repository. * Authenticate before establishing WebSocket connection Added a preliminary authentication check via bearings API before attempting to connect to the WebSocket. Refactored code to fetch and process bearings data only once, improving efficiency by rejecting subscription if authentication fails. * Improve WebSocket reconnection logic Refactored WebSocket handling to add a dedicated _reconnect method that validates authentication before reconnecting. Bearings data is now only processed on initial connection, not on reconnection, preventing redundant state updates. * Refactor bearings data handling on WebSocket connect Improves logic for processing bearings data on initial connection versus reconnection. On initial connection, all bearings data is cleared and reloaded; on reconnection, only core authentication data and grants are updated. Also fixes passing of bearings data during reconnection. * Delete postgres.exe * Update data.js * Update data.js * Update data.js * Create .env * Add local compose, update build and revamp CSS Add a local docker-compose stack and developer config, improve build flow, and modernize the UI styles. Files added: docker-compose.local.yml (local stack: vault/core/agent/webdav/demo) and .claude/settings.local.json; .gitignore updated to ignore /.vscode. Dockerfile: bump Go to 1.23 and run go mod tidy + go mod vendor before building. Makefile: add JOBS, more robust plugin build (fallback to module mode), safer demo target, and new demo-local / dev-local targets to run the local stack. Major rewrite of web/htdocs/shield.css to a modern dark/light theme using CSS variables, layout improvements, responsive tweaks and refreshed components. * Add collapsible sidebar with icons Introduce a collapsible story-sidebar: include FontAwesome solid icons, restructure sidebar nav items to show icons + labels, and add a toggle button in index.html. Add JS handlers in events.js to toggle collapse state and persist it to localStorage, and restore persisted state on startup in shield.js. Add CSS in shield.css for collapsed/expanded styles, transitions, toggle button, and adjustments to footer and pane widths to support the compact sidebar. * Update top-bar UI and rename M1 to Apple Silicon Change index.html labels from "MacOS (M1)" to "MacOS (Apple Silicon)". Major CSS updates to top-bar and account UX in shield.css: set title widths and truncation, add styled account button and dropdown/flyout (menu, header, divider, item states, current-tenant styling), adjust top-bar spacing, move sidebar down (top/height calc) and tweak sidebar-toggle position, and remove an unused selector. These changes improve account menu usability and layout spacing. * Fix top-bar dropdown and toggle styles Allow the top-bar dropdown to escape the bar and improve the sidebar toggle positioning and visuals. Removed overflow:hidden on table so dropdowns can escape, converted .top-bar .flyout from absolute to fixed (top:48px, min-width:260px) and adjusted its right positioning. Made the sidebar toggle fixed to the viewport and centered on the sidebar right edge / navbar bottom (computed left/top values), changed background to --bg-secondary, increased z-index to sit between sidebar and top-bar, refined transitions, added box-shadow, and consolidated collapsed-state selectors to use body.sidebar-collapsed. Also added a smooth icon rotation transition. These changes fix clipping issues and provide more stable, consistent positioning and styling for the toggle and flyout. * Add theme toggle, top-nav and lock styling Add a light/dark theme toggle and apply persisted theme before first paint to avoid a flash. Introduce a top-bar inline navigation rendered for authenticated users (remove duplicated nav markup), and add a theme-toggle button with JS to toggle data-theme and store the choice in a cookie. Update CSS to style the new top-nav, compact the LOCKED banner into a compact, responsive inline warning, adjust sidebar-toggle positioning, and refine related layout/spacing for header/account link. Changes touch index.html, js/events.js (theme toggle handler + init script), and shield.css (navigation, theme toggle, locked state and layout tweaks). * Add HUD background image and overlay Add bg.jpg asset and update shield.css to use it as the .hud background (cover, centered, no-repeat). Add a semi-opaque overlay (::before) to preserve card readability with a light-theme variant, and ensure .hud children are positioned above the overlay via relative positioning and z-index.
New internal/log/ package provides a thin slog-based shim preserving go-log's function signatures (Infof, Debugf, Errorf, Warnf, Warningf, Alertf, IsDebug) so 363 call sites only require import path changes. New plugin/s3util/ package extracts shared S3 client construction (NewClient), backup path generation (GenBackupPath), and byte-counting reader (CountingReader) for use by both S3 and Backblaze storage plugins.
Remove jmoiron/sqlx dependency from the database layer. SQLite uses ? placeholders natively so the sqlx.Rebind() wrapper was a no-op. Change connection field type from *sqlx.DB to *sql.DB and Open call from sqlx.Open to sql.Open.
Replace external go-querytron dependency with a local generateQueryString() helper in client/v2/shield/qs.go. Uses reflection over qs struct tags to build url.Values, handling string, int, bool (with true/false value mapping), *bool, and *int field types. Update 10 consumer files to call the local function.
Replace github.com/jhunt/go-log import with github.com/shieldproject/shield/internal/log in all agent, core, db, and route packages. The slog shim preserves identical function signatures so no call site changes are needed.
Replace jhunt/go-s3 with AWS SDK v2 via the shared s3util package. Store uses PutObject with a CountingReader for accurate size tracking. Retrieve uses GetObject with io.Copy. Purge uses DeleteObject. Fix path-style detection to check for PermanentRedirect instead of substring "301" match. Fix default endpoint typo (amazonawd → amazonaws).
Replace kurin/blazer with AWS SDK v2 S3-compatible API via the shared s3util package. Auto-detect bucket region using B2 native API (b2_authorize_account + s3ApiUrl hostname parsing) to avoid config changes. Fix three bugs: hardcoded Store() size of 1024 bytes now uses CountingReader for actual size, unchecked io.Copy error in Retrieve() now propagated, and response status checked before JSON decode in region detection.
Replace go-github v0 with v66 using WithAuthToken pattern instead of oauth2 HTTP client wrapper. Fix silently broken team membership listing by using Teams.ListUserTeams (moved from Organizations service in v28+). Add nil guards for org/team pointer fields. Modernize OAuth2 token exchange in GitHub auth provider to use oauth2.Config.Exchange() instead of manual HTTP POST. Replace deprecated io/ioutil.
Replace go-cli struct-tag parsing in plugin.Run() with Cobra root command and 7 subcommands (info, validate, backup, restore, store, retrieve, purge). Persistent flags handle --debug, --endpoint, --key, and --text. Cobra manages --help automatically. Preserve Plugin interface, PluginInfo, Field, ShieldEndpoint, dispatch(), codeForError(), and all error types unchanged so 24 plugin binaries require no modifications.
Replace go-cli/go-envirotron with Cobra commands in shieldd, shield-agent, shield-schema, shield-crypt, and shield-report. Environment variable overrides handled via os.Getenv in PersistentPreRunE. Log setup calls updated to use new slog shim. All flags and behavior preserved.
Split 3508-line main.go into root.go and 14 command group files (cmd_auth, cmd_admin, cmd_tenants, cmd_targets, cmd_stores, cmd_jobs, cmd_archives, cmd_tasks, cmd_users, cmd_sessions, cmd_agents, cmd_fixups, cmd_misc, cmd_op). Each command registers via init() on the root Cobra command. Delete help.go (2062-line help dispatcher) and help.pl (code generator) — Cobra handles help text via Long field on each command. Remove help.go generation target from Makefile.
Add: spf13/cobra, aws-sdk-go-v2 (config, credentials, s3, feature/s3/manager), google/go-github/v66. Remove: jhunt/go-cli, jhunt/go-log, jhunt/go-s3, jhunt/go-querytron, jhunt/go-snapshot, jmoiron/sqlx, kurin/blazer, google/go-github (v0). Bump Go version from 1.23.0 to 1.24 as required by aws-sdk-go-v2.
Advance go directive from 1.24 to 1.26 and toolchain from go1.24.2 to go1.26.1 to enable latest dependency versions for the v10.0.0 release.
x/crypto v0.36.0 → v0.49.0 x/oauth2 v0.30.0 → v0.36.0 x/net v0.38.0 → v0.52.0 x/sys v0.31.0 → v0.42.0 (indirect) x/term v0.30.0 → v0.41.0 (indirect) x/text v0.23.0 → v0.35.0 (indirect)
google.golang.org/api v0.126.0 → v0.273.0 This eliminates the archived go.opencensus.io dependency which has been replaced internally by OpenTelemetry. Also removes google.golang.org/appengine and golang/groupcache as transitive dependencies.
Also drops archived google/btree and mitchellh/mapstructure as transitive dependencies since newer consul/api uses go-viper/mapstructure/v2 (the maintained fork). Only mitchellh/go-homedir remains as an archived indirect dep, still required by hashicorp/go-rootcerts.
go-dockerclient v1.12.1 → v1.13.0 go-sql-driver v1.5.0 → v1.9.3 gorilla/websocket v1.4.2 → v1.5.3 go-sqlite3 v1.14.15 → v1.14.37 prometheus v1.19.1 → v1.23.2 etcd/client/v3 v3.5.9 → v3.6.9 gomega v1.24.2 → v1.39.1 pborman/uuid v1.2.0 → v1.2.1 Also updates numerous transitive dependencies including docker, klauspost/compress, miekg/dns, and others.
The emicklei/go-restful/v3 replace directive was a leftover with no corresponding require entry. Removed during final dependency cleanup.
yaml.v3 returns map[string]interface{} for YAML maps
instead of map[interface{}]interface{}, eliminating the
need for the StringifyKeys conversion utility. The
UnmarshalYAML interface also changed from a callback
to a *yaml.Node receiver.
Changes:
- Swap gopkg.in/yaml.v2 imports to gopkg.in/yaml.v3
- Rewrite duration.UnmarshalYAML for yaml.Node API
- Update AuthProvider.Configure param type
- Remove StringifyKeys from util and all callers
- Update go.mod and vendor
Add suite bootstrap files for route, core/scheduler, and core/bus packages to enable BDD testing with Ginkgo v1 + Gomega.
SetCookie and ClearCookie were missing security flags, leaving sessions vulnerable to XSS and CSRF attacks.
The retry condition used || causing success (rc=0) to loop forever. Changed to && with rc!=0 so retries stop on success. Also made rc atomic for goroutine safety.
CreateRestoreTask now rejects cross-tenant and cross- plugin restores. Also fix API handler tenant check to use target.TenantUUID instead of archive.TenantUUID.
Worker.available was a plain bool read and written from multiple goroutines without synchronization. Changed to atomic.Bool with sync.Mutex guarding the task field.
Verify client connection drop, metrics tracking, and slot recovery when the message bus backlog is full.
Test target deletion with orphaned archives and agent re-registration with address change impact on lookups.
Migrate test framework from Ginkgo v1.16.5 to v2.28.1 and Gomega from v1.24.2 to v1.39.1 across all 24 test files in 7 packages. - Update import paths to github.com/onsi/ginkgo/v2 - Convert 17 async Done patterns in db/bus_test.go to Eventually/Receive with preserved timeouts - Replace deprecated ioutil with os/io equivalents - Update Makefile race target for v2 CLI syntax - Rebuild vendor with pruned v1-only dependencies
Introduce db/dialect.go with Dialect type, DetectDialect(), Rebind(), and IsNoSuchTable() to handle placeholder translation across PostgreSQL ($1), MySQL (?), and SQLite3 (?) without the sqlx dependency.
Wire Rebind() into statement() so queries using ? placeholders translate to $1/$2 for PostgreSQL. Change Connect(file) to Connect(driver, dsn) with eager Ping(). Fix BEGIN TRANSACTION to BEGIN for MySQL, IFNULL to COALESCE for ANSI SQL, and consolidate table-missing checks via IsNoSuchTable().
Add Database.Driver/DSN config fields with sqlite3 defaults for backward compatibility. Import pgx/v5/stdlib and go-sql-driver/mysql in shieldd and shield-schema binaries. Add --driver flag to shield-schema CLI.
Correct mismatched format verbs and missing arguments in log.Errorf/Infof/Warnf calls across core package to pass go vet cleanly.
Add Ginkgo test suite for core package with 12 tests covering duration unmarshaling (string formats, bare integers, fractional values, nested structs). Add 3 stdlib tests for CLI config round-trips, legacy config parsing, and import manifest with mixed-type maps.
Replace gopkg.in/yaml.v3 with github.com/goccy/go-yaml across all 5 source files. Rewrite duration.UnmarshalYAML from *yaml.Node signature to callback signature. Update go.mod and vendor; yaml.v3 fully removed.
SHIELD_CORE_MASTER, SHIELD_CORE_USERNAME, SHIELD_CORE_PASSWORD, and SHIELD_CORE_TOKEN env vars stopped working after the cobra migration because only global flags got os.Getenv fallbacks in PersistentPreRunE. Add inline env var checks for command-local flags in init, unlock, rekey, and login commands, matching the precedence order: CLI flag > env var > interactive prompt.
Update Dockerfile to Go 1.26.1, Ubuntu noble, and embed Vault 1.21.4 in the core container matching phalanx's proven architecture. Update nginx images from bullseye to stable-bookworm. Add healthchecks, proper depends_on conditions, and init scripts that start Vault, wait for readiness, run schema migrations, then launch shieldd with config file.
Add missing event.preventDefault() on the decide template form handlers so clicking Set Up navigates to the password form instead of reloading the page. Add $form.reset() before validation and CSS rule to hide error spans by default in the init form, which uses .ctl containers not covered by the .field rule.
Remove erroneous .Elem() call when reading string field values in the query string generator. String kinds are not pointers and do not need dereferencing.
Delete all existing keys before restoring from backup so that keys created after the backup was taken do not persist through a restore operation.
Buffer stdin to a seekable temp file before upload since PutObject requires a seekable body for payload hash computation. Disable automatic checksum calculation for non-TLS endpoints to avoid trailing checksum failures when streaming.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP - v10.0.0 improvements.