#340 Part 3: Complete Audit Flow integration and create Audit Interface

[ginaxu1]

Summary

This PR centralizes audit logging for POLICY_CHECK and CONSENT_CHECK events in Orchestration Engine, implements PROVIDER_FETCH events, and establishes a clean Auditor interface in shared/audit/ for easy migration when audit-service moves to its own repository.

Key Changes:

• Audit Interface: Created Auditor interface in shared/audit/interface.go for clean abstraction
• Centralized Audit Logging: All audit events (ORCHESTRATION_REQUEST_RECEIVED, POLICY_CHECK, CONSENT_CHECK, PROVIDER_FETCH) are logged from Orchestration Engine for consistent traceID correlation
• POLICY_CHECK Events: Logged by OE after PDP API call (one record per API call, status reflects result)
• CONSENT_CHECK Events: Logged by OE after CE API call (one record per API call, status reflects result)
• Removed REQUEST/RESPONSE Variants: All events are single events (no REQUEST/RESPONSE pairs)
• Complete Traceability: Full audit trail from OE with all events linked by trace ID from the initial request context
• Reusable Audit Service: Moved OpenDIF-specific event types to sample enums.yaml config, keeping DefaultEnums generic

Example Audit Flow

A single OE request now generates a complete sequence of audit events linked by one traceID:

1. ORCHESTRATION_REQUEST_RECEIVED (traceID: abc-123)

   • Logged by Orchestration Engine when GraphQL request is received
   • Includes: application ID, GraphQL query
   • Status: SUCCESS

2. POLICY_CHECK (traceID: abc-123)

   • Logged by Orchestration Engine after PDP API call
   • One audit record per API call
   • Includes: application ID, required fields, authorization status, consent requirements, unauthorized/expired fields
   • Status: SUCCESS if API call succeeds, FAILURE if API call fails or unauthorized/expired
   • Target: policy-decision-point

3. CONSENT_CHECK (traceID: abc-123)

   • Logged by Orchestration Engine after CE API call
   • One audit record per API call
   • Includes: application ID, owner ID/email, consent ID, consent status, consent portal URL, fields count
   • Status: SUCCESS if API call succeeds, FAILURE if API call fails
   • Target: consent-engine

4. PROVIDER_FETCH (traceID: abc-123, target: provider-1, provider-2, etc.)

   • Logged by Orchestration Engine after receiving response from each provider
   • One audit record per API call
   • Includes: application ID, schema ID, service key, requested fields, GraphQL query, response status, errors, data keys
   • Status: SUCCESS or FAILURE based on provider response
   • Target: Provider service key

Example: For a request with 2 providers:

• Total events: 4 (1 orchestration + 1 policy + 1 consent + 2 providers)
• All linked by: traceID: abc-123

All events can be retrieved using: GET /api/audit-logs?traceId=abc-123

Changes

Shared Audit Package (shared/audit/)

• New Audit Interface: Created Auditor interface in interface.go for clean abstraction
  • Provides LogEvent() and IsEnabled() methods
  • Makes it easy to swap implementations when audit-service moves to its own repository
  • AuditClient alias maintained for backward compatibility
• Updated Middleware: Updated middleware.go to use Auditor interface instead of concrete AuditClient
• Updated Initialization: Updated init.go to use Auditor interface
• Client Implementation: Client struct implements Auditor interface (already had required methods)

Event Types

The following event types are now fully implemented:

Event Type	Logged By	Description	Status Values	Target Service
ORCHESTRATION_REQUEST_RECEIVED	Orchestration Engine	GraphQL request received	SUCCESS	-
POLICY_CHECK	Orchestration Engine	Authorization decision result (logged after PDP API call, one record per API call)	SUCCESS, FAILURE	policy-decision-point
CONSENT_CHECK	Orchestration Engine	Consent check result (logged after CE API call, one record per API call)	SUCCESS, FAILURE	consent-engine
PROVIDER_FETCH	Orchestration Engine	Provider fetch result (one record per API call)	SUCCESS, FAILURE	Provider service key

Configuration

Environment Variables

Orchestration Engine supports audit configuration:

• CHOREO_AUDIT_CONNECTION_SERVICEURL: Audit service base URL (e.g., http://localhost:3001)
  • If not set or empty, audit logging is disabled
  • Default: empty (audit disabled)
• ENABLE_AUDIT: Explicitly enable/disable audit logging
  • Values: true, 1, yes (case-insensitive) to enable
  • Default: enabled if CHOREO_AUDIT_CONNECTION_SERVICEURL is set

Upgrade Path

1. Deploy updated audit-service with new event types in enums.yaml:
   • POLICY_CHECK (one record per API call, logged by Orchestration Engine)
   • CONSENT_CHECK (one record per API call, logged by Orchestration Engine)
   • PROVIDER_FETCH (one record per API call, logged by Orchestration Engine)
2. Deploy updated Orchestration Engine with centralized audit logging for all events
3. Deploy updated Policy Decision Point and Consent Engine (audit logging removed, traceID propagation kept)
4. Set CHOREO_AUDIT_CONNECTION_SERVICEURL environment variable in Orchestration Engine
5. Verify audit events are being logged correctly using GET /api/audit-logs?traceId=<trace-id>

Related PRs

• 340 Refactor Audit Service for Generalized Auditing with Trace ID Support #384 - Refactor audit-service to be reusable, general with trace_id
• 317 Add Observability to PDP and OE #408 - Add shared audit client and integrate with OE and Portal Backend
• #340 Part 2: add Audit Service unit tests, db configuration, and delete legacy files #410 - Audit Service clean up and add unit tests

Related Issues

Closes #340