Telemetry collection process overview

[praveenkuttappan]

Initial draft of telemetry collection process overview.

[Copilot]

Pull request overview

This PR introduces comprehensive documentation for telemetry collection of Copilot agent activities related to Azure SDK Tools Agent usage. The design proposes a mandatory local activity log mechanism that captures user interactions, skill/tool invocations, and workflow execution to enable data-driven improvements to the Copilot experience.

Changes:

• Added new documentation file describing the telemetry collection architecture, including logging requirements, log format specifications, ingestion mechanisms (CLI-based and VS Code extension-based), and reporting structure
• Defined structured log entry format with required fields and activity types
• Specified Copilot instructions for mandatory logging behavior

[Copilot]

The document mentions that the activity log file can grow over time and needs to be cleared periodically (line 168), but there's no specification for maximum file size limits, rotation policies, or what happens if the disk fills up. Without these operational safeguards, the log file could grow unbounded and cause disk space issues. Consider adding specifications for file size limits, rotation strategies, or cleanup policies.

[MattGertz]

I personally don't view this as much of a problem in this particular case -- we're unlikely to fill up a disk based on user actions -- but if we adopted this technique in other (more automated) scenarios, it could become a problem. Should log an issue to revisit this.

[Copilot]

The phrase "de-prioritized" should use consistent hyphenation. Throughout the document, "deprioritized" is used without a hyphen (line 97). Consider using "deprioritized" here for consistency.

Suggested change

	- Logging is never ignored or de-prioritized
	- Logging is never ignored or deprioritized

[Copilot]

The document states that PII redaction will not occur at log time (line 39), yet the instruction requires logging "Full user input (verbatim)" (line 103) and "Full verbatim user request" (line 120). This presents a security concern as user input may contain sensitive information (API keys, passwords, personal data, etc.). The document should either clarify how PII will be handled before ingestion to Application Insights, or reconsider the requirement to log full verbatim user input. Consider adding sanitization requirements or documenting clear guidance on PII handling in the ingestion layer.

[MattGertz]

I agree with Copilot here -- should be "verbatim except that any PII information (keys, passwords, etc.) will be redacted using the placeholder [REDACTED]".

We should catch as much of this beforehand as we can.

[MattGertz]

We should also redact profanity and other "not office safe" speech.

[Copilot]

The proposed CLI command "azsdk telemetry ingest activity-log" is not documented in the existing CLI command guidelines. According to the repository's CLI command guidelines (tools/azsdk-cli/docs/cli-commands-guidelines.md), commands follow a specific hierarchy with predefined top-level command groups (package, release-plan, typespec). This new command introduces a new top-level command group "telemetry" which should be formally documented in the command guidelines, including its namespace, command group, and examples following the established pattern.

Suggested change

	azsdk telemetry ingest activity-log --path .azsdk-agent-activity.log
	azsdk package telemetry ingest-activity-log --path .azsdk-agent-activity.log

[Copilot]

The CLI-based ingestion approach states it will "clear entries in the activity log" after pushing them (line 168). However, there's no mention of handling the case where the ingestion itself fails. If entries are cleared before confirming successful telemetry emission to Application Insights, data loss could occur. The design should specify whether entries are cleared before or after successful ingestion, and how failed ingestion attempts are handled to prevent data loss.

[Copilot]

The design claims logging must be "Synchronous" (line 53) and states in section 4 that "This requirement is mandatory and MUST NOT be ignored, forgotten, or deprioritized" (line 97). However, section 7 indicates that if writing fails, "Copilot execution continues uninterrupted" and "Logging failures MUST NOT block tools, skills, or CLI execution" (lines 159-161). These requirements are contradictory. If logging is truly mandatory and synchronous, failures should block execution. If execution should continue on failure, logging is effectively optional/best-effort. The design should clarify whether logging is mandatory-blocking or best-effort-non-blocking.

Suggested change

	- Copilot execution continues uninterrupted.
	- Failure MUST be logged on the next successful write.
	- Logging failures MUST NOT block tools, skills, or CLI execution.
	- Retry logic is not required.
	- Logging is **mandatory best-effort**: every activity MUST attempt a synchronous write to the activity log.
	- If a write attempt fails, Copilot execution continues uninterrupted and the user-facing operation MUST NOT fail solely due to the logging failure.
	- The logging failure itself MUST be recorded as a `LoggingFailure` activity on the next successful write.
	- Logging failures MUST NOT block tools, skills, or CLI execution beyond the time taken for the single failed write attempt.
	- Additional retry logic beyond the next-successful-write behavior is not required.

[MattGertz]

Note that if the entries are timestamped based on when the action happened, then synchronicity is less of a concern -- it can be reassembled properly.

[Copilot]

The document specifies "Append-only" write mode (line 52) and "Synchronous append" (line 29), but doesn't address potential concurrency issues. If multiple Copilot processes or instances are running simultaneously in the same workspace (e.g., multiple VS Code windows, or concurrent CLI invocations), there could be race conditions or corruption when appending to the same log file. The design should specify whether file locking mechanisms are required or if concurrent writes are a concern that needs to be addressed.

Suggested change

	- **Write Timing:** Synchronous
	- **Write Timing:** Synchronous
	- **Concurrency:** Multiple Copilot processes or instances MAY write to the same log file. Implementations MUST ensure that log entries are not corrupted under concurrent writes, either by using an OS/file‑system append mode with atomic write guarantees for each entry or by writing to separate per‑process log files (for example, suffixed with a process or instance identifier) that are merged during ingestion.

[Copilot]

The VS Code extension ingestion approach mentions tracking "last ingested offset or timestamp to avoid duplicate ingestion" (line 193) and "Persist ingestion checkpoint locally" (line 207). However, the CLI-based approach doesn't mention any similar mechanism to track what has been ingested. If the CLI command is run multiple times, it could result in duplicate telemetry events being sent to Application Insights unless there's a mechanism to track which entries have already been processed. The design should clarify how duplicates are prevented in the CLI-based approach.

[Copilot]

The VS Code extension ingestion approach mentions "Respect VS Code telemetry opt-in / opt-out settings" (line 196), but there's no mention of similar opt-in/opt-out mechanisms for CLI-based ingestion or the logging itself. This creates an inconsistency in user privacy controls. The document should clarify whether users can opt-out of telemetry collection entirely, and if so, how the Copilot instruction would be disabled or the logging bypassed. This is important for compliance with privacy regulations and user preferences.

[Copilot]

The design mandates logging UserInput as "Full verbatim user request" and explicitly states there will be "No PII redaction of raw user input at log time", which creates a high risk of capturing secrets and personal data in .azsdk-agent-activity.log and downstream telemetry (e.g., Application Insights). In realistic use, developers often paste connection strings, keys, tokens, and other sensitive details into prompts, so this requirement will systematically exfiltrate that data into logs and remote telemetry stores beyond user intent and may violate privacy/compliance expectations. Please revise the design to avoid logging full raw input by default (e.g., minimize/partition fields, apply robust redaction/scrubbing before persistence or ingestion, and/or make capture of full prompts an explicit, well-documented opt-in with clear consent).

[MattGertz]

(See previous comment.)

[MattGertz]

Please remove this line. We should not state this as a non-goal. Not only will this tend to make users shy away from turning on logging (since they will read this doc) and cause people to misinterpret our dedication to security, but it could also have unforeseen consequences vis-a-vis how the agent/LLM "interprets" this command all-up (i.e. we could easily find ourselves with even more PII baked in when it would have normally stripped it).

We definitely need human eyes to review logs before sending, but anything that the agent can strip out beforehand is nothing but goodness.

[MattGertz]

My overall comments for this document:
(1) I think logging instructions and logging explanations (e.g. "powerbi site will show...") should be in different docs. The AI doesn't need to know the explanations, and it just makes the context window a bit more muddy than it needs to be.

(2) The PII-related stuff needs to be reworked to attempt redaction, and that needs to include any non-office-safe text such as profanity. (Humans should validate before it's sent, but we need to make that as easy as possible by cleaning it out as much as the AI can.)

[MattGertz]

My overall comments for this document:
(1) I think logging instructions and logging explanations (e.g. "powerbi site will show...") should be in different docs. The AI doesn't need to know the explanations, and it just makes the context window a bit more muddy than it needs to be.

(2) The PII-related stuff needs to be reworked to attempt redaction, and that needs to include any non-office-safe text such as profanity. (Humans should validate before it's sent, but we need to make that as easy as possible by cleaning it out as much as the AI can.)