feat: add OTel events and metrics for agentic edit quality signals (#4794)

* docs: add OTel backfill plan for agentic change metrics

* docs: add Claude Code OTel parity analysis with feasibility + line estimates

* docs: expand plan to cover all agentic surfaces (inline chat, CLI, cloud, NES)

* docs: remove NES and Claude Code comparison, keep plan lean

* docs: add 3-pillar signal type mapping (metrics/events/traces)

* docs: re-audit signals — counters when easy, events only with useful attrs

* feat: add OTel event emitters for agentic edit quality metrics

* feat: add OTel counters and histograms for agentic edit quality metrics

* feat: wire OTel events/metrics into userActions.ts for all agentic user actions

* feat: wire OTel survival events into apply_patch, replace_string, and code_mapper tools

* feat: wire OTel counters for agent summarization and edit response metrics

* fix: resolve TypeScript errors — thread IOTelService through intent class hierarchy

* docs: update sprint plan with completion notes

* style: fix import ordering from editor auto-sort

* feat: wire OTel counters for cloud session invoke, PR ready, and CLI PR creation

* docs: consolidate OTel edit quality metrics into agent_monitoring.md

* docs: rename Edit Quality to Agent Activity & Outcome

* docs: align Edit Quality references to Agent Activity naming

* refactor: adopt Harald's type-safe metrics API (EditSource/EditOutcome, 2 survival histograms)

Author: zhichli

docs/monitoring/agent_monitoring.md

@@ -224,6 +224,43 @@ invoke_agent copilot                           [~15s]
 
 **`copilot_chat.time_to_first_token` attributes:** `gen_ai.request.model`
 
+#### Agent Activity & Outcome Metrics
+
+These metrics track the activity and outcomes of agentic code changes across all surfaces (agent mode, inline chat, background CLI, cloud sessions).
+
+| Metric | Type | Unit | Description |
+|---|---|---|---|
+| `copilot_chat.edit.accept.count` | Counter | edits | File-level and inline edit accept/reject decisions |
+| `copilot_chat.edit.hunk.count` | Counter | hunks | Hunk-level accept/reject decisions |
+| `copilot_chat.lines_of_code.count` | Counter | lines | Lines of code added/removed by accepted agent edits |
+| `copilot_chat.edit.survival_rate` | Histogram | ratio (0-1) | How much AI-edited code survives over time (4-gram similarity) |
+| `copilot_chat.user.action.count` | Counter | actions | User engagement: copy, insert, apply, followup |
+| `copilot_chat.user.feedback.count` | Counter | votes | Thumbs up/down on chat responses |
+| `copilot_chat.agent.edit_response.count` | Counter | responses | Agent edit responses by success/error |
+| `copilot_chat.agent.summarization.count` | Counter | events | Context summarization outcomes (applied/failed) |
+| `copilot_chat.pull_request.count` | Counter | PRs | Pull requests created via CLI agent |
+| `copilot_chat.commit.count` | Counter | commits | Git commits created via agent |
+| `copilot_chat.cloud.session.count` | Counter | sessions | Cloud/remote agent sessions by partner |
+| `copilot_chat.cloud.pr_ready.count` | Counter | events | Remote agent job PR ready notifications |
+
+**`copilot_chat.edit.accept.count` attributes:** `outcome` (`accepted`/`rejected`), `edit_surface` (`agent`/`inline_chat`)
+
+**`copilot_chat.edit.hunk.count` attributes:** `outcome` (`accepted`/`rejected`)
+
+**`copilot_chat.lines_of_code.count` attributes:** `type` (`added`/`removed`), `language_id`
+
+**`copilot_chat.edit.survival_rate` attributes:** `edit_source` (`apply_patch`/`replace_string`/`code_mapper`/`inline_chat`), `time_delay_ms`
+
+**`copilot_chat.user.action.count` attributes:** `action` (`copy`/`insert`/`apply`/`followup`)
+
+**`copilot_chat.user.feedback.count` attributes:** `rating` (`positive`/`negative`)
+
+**`copilot_chat.agent.edit_response.count` attributes:** `outcome` (`success`/`error`)
+
+**`copilot_chat.agent.summarization.count` attributes:** `outcome` (`applied`/`failed`)
+
+**`copilot_chat.cloud.session.count` attributes:** `partner_agent` (`copilot`/`claude`/`codex`)
+
 ### Events
 
 #### `gen_ai.client.inference.operation.details`
@@ -278,6 +315,84 @@ Emitted for each LLM round-trip within an agent invocation.
 | `gen_ai.usage.output_tokens` | Output tokens this turn |
 | `tool_call_count` | Number of tool calls this turn |
 
+#### Agent Activity & Outcome Events
+
+These events provide drill-down detail for the agent activity metrics above. They are emitted as OTel log records.
+
+##### `copilot_chat.edit.feedback`
+
+Emitted when a user accepts or rejects a file-level edit from the agent.
+
+| Attribute | Description |
+|---|---|
+| `outcome` | `accepted` or `rejected` |
+| `language_id` | Language of the edited file |
+| `participant` | Chat participant that proposed the edit |
+| `request_id` | Chat request identifier |
+| `edit_surface` | `agent` or `inline_chat` |
+| `has_remaining_edits` | Whether unreviewed edits remain |
+| `is_notebook` | Whether the file is a notebook |
+
+##### `copilot_chat.edit.hunk.action`
+
+Emitted when a user accepts or rejects an individual hunk.
+
+| Attribute | Description |
+|---|---|
+| `outcome` | `accepted` or `rejected` |
+| `language_id` | Language of the edited file |
+| `request_id` | Chat request identifier |
+| `line_count` | Total lines in the hunk |
+| `lines_added` | Lines added |
+| `lines_removed` | Lines removed |
+
+##### `copilot_chat.inline.done`
+
+Emitted when an inline chat edit is accepted or rejected.
+
+| Attribute | Description |
+|---|---|
+| `accepted` | `true` or `false` |
+| `language_id` | Language of the edited file |
+| `edit_count` | Number of edits suggested |
+| `edit_line_count` | Total lines across all edits |
+| `reply_type` | How the response was shown |
+| `is_notebook` | Whether the document is a notebook |
+
+##### `copilot_chat.edit.survival`
+
+Emitted at intervals (5s, 30s, 2min, 5min, 10min, 15min) after an edit is accepted, measuring how much of the AI-generated code survives.
+
+| Attribute | Description |
+|---|---|
+| `edit_source` | `apply_patch`, `replace_string`, `code_mapper`, or `inline_chat` |
+| `survival_rate_four_gram` | 0-1 ratio of AI edit still present (4-gram similarity) |
+| `survival_rate_no_revert` | 0-1 ratio of edit ranges not reverted |
+| `time_delay_ms` | Milliseconds since edit acceptance |
+| `did_branch_change` | Whether git branch changed (ignore if `true`) |
+| `request_id` | Chat request identifier |
+
+##### `copilot_chat.user.feedback`
+
+Emitted when a user votes on a chat response (thumbs up/down).
+
+| Attribute | Description |
+|---|---|
+| `rating` | `positive` or `negative` |
+| `participant` | Chat participant name |
+| `conversation_id` | Conversation session ID |
+| `request_id` | Chat request identifier |
+
+##### `copilot_chat.cloud.session.invoke`
+
+Emitted when a cloud/remote agent session is started.
+
+| Attribute | Description |
+|---|---|
+| `partner_agent` | `copilot`, `claude`, or `codex` |
+| `model` | Model identifier |
+| `request_id` | Chat request identifier |
+
 ### Resource Attributes
 
 All signals carry:
@@ -449,9 +564,9 @@ In your trace viewer, filter by `service.name` to see traces from specific agent
 
 **Traces** — Visualize the full agent execution in Jaeger or Grafana Tempo. Each `invoke_agent` span contains child `chat` and `execute_tool` spans, making it easy to identify bottlenecks and debug failures. Subagent invocations appear as nested `invoke_agent` spans under `execute_tool runSubagent`.
 
-**Metrics** — Track token usage trends by model and provider, monitor tool success rates via `copilot_chat.tool.call.count`, and watch perceived latency with `copilot_chat.time_to_first_token`. All metrics carry the same resource attributes (`service.name`, `service.version`, `session.id`) for consistent filtering.
+**Metrics** — Track token usage trends by model and provider, monitor tool success rates via `copilot_chat.tool.call.count`, and watch perceived latency with `copilot_chat.time_to_first_token`. Agent activity metrics (`copilot_chat.edit.accept.count`, `copilot_chat.edit.survival_rate`, `copilot_chat.lines_of_code.count`) power accept rate and commit survival dashboards. All metrics carry the same resource attributes (`service.name`, `service.version`, `session.id`) for consistent filtering.
 
-**Events** — `copilot_chat.session.start` tracks session creation. `copilot_chat.tool.call` events provide per-invocation timing and error details. `gen_ai.client.inference.operation.details` gives the full LLM call record including token usage and, when content capture is enabled, the complete prompt/response messages. Use `gen_ai.conversation.id` to correlate all signals belonging to the same session.
+**Events** — `copilot_chat.session.start` tracks session creation. `copilot_chat.tool.call` events provide per-invocation timing and error details. `copilot_chat.edit.feedback` and `copilot_chat.edit.survival` events enable drill-down into which edits were accepted/rejected and how code survival varies by edit source. `copilot_chat.user.feedback` links thumbs-up/down votes to specific conversations for quality investigation. `gen_ai.client.inference.operation.details` gives the full LLM call record including token usage and, when content capture is enabled, the complete prompt/response messages. Use `gen_ai.conversation.id` to correlate all signals belonging to the same session.
 
 ---
 

src/extension/chatSessions/copilotcli/node/copilotcliSession.ts

@@ -9,6 +9,7 @@ import type * as vscode from 'vscode';
 import type { ChatParticipantToolToken } from 'vscode';
 import { ConfigKey, IConfigurationService } from '../../../../platform/configuration/common/configurationService';
 import { ILogService } from '../../../../platform/log/common/logService';
+import { GenAiMetrics } from '../../../../platform/otel/common/genAiMetrics';
 import { CopilotChatAttr, GenAiAttr, GenAiOperationName, IOTelService, ISpanHandle, SpanKind, SpanStatusCode, truncateForOTel } from '../../../../platform/otel/common/index';
 import type { ParsedPromptFile } from '../../../../platform/promptFiles/common/promptsService';
 import { CapturingToken } from '../../../../platform/requestLogger/common/capturingToken';
@@ -580,6 +581,7 @@ export class CopilotCLISession extends DisposableStore implements ICopilotCLISes
 					if (pullRequestUrl) {
 						this._createdPullRequestUrl = pullRequestUrl;
 						this.logService.trace(`[CopilotCLISession] Captured pull request URL: ${pullRequestUrl}`);
+						GenAiMetrics.incrementPullRequestCount(this._otelService);
 					}
 				}
 				// Log tool call to request logger

src/extension/chatSessions/vscode-node/copilotCloudSessionsProvider.ts

@@ -15,6 +15,8 @@ import { GithubRepoId, IGitService } from '../../../platform/git/common/gitServi
 import { derivePullRequestState, PullRequestSearchItem, SessionInfo } from '../../../platform/github/common/githubAPI';
 import { AuthOptions, CCAEnabledResult, IGithubRepositoryService, IOctoKitService, JobInfo, RemoteAgentJobResponse } from '../../../platform/github/common/githubService';
 import { ILogService } from '../../../platform/log/common/logService';
+import { GenAiMetrics } from '../../../platform/otel/common/genAiMetrics';
+import { IOTelService } from '../../../platform/otel/common/otelService';
 import { IExperimentationService } from '../../../platform/telemetry/common/nullExperimentationService';
 import { ITelemetryService } from '../../../platform/telemetry/common/telemetry';
 import { DeferredPromise, retry, RunOnceScheduler } from '../../../util/vs/base/common/async';
@@ -307,6 +309,7 @@ export class CopilotCloudSessionsProvider extends Disposable implements vscode.C
 		@IChatDelegationSummaryService private readonly _chatDelegationSummaryService: IChatDelegationSummaryService,
 		@IExperimentationService private readonly _experimentationService: IExperimentationService,
 		@IDomainService private readonly _domainService: IDomainService,
+		@IOTelService private readonly _otelService: IOTelService,
 	) {
 		super();
 		this.registerCommands();
@@ -1962,6 +1965,7 @@ export class CopilotCloudSessionsProvider extends Disposable implements vscode.C
 			partnerAgent: partnerAgent?.name ?? 'unknown',
 			model: modelId ?? 'unknown'
 		});
+		GenAiMetrics.incrementCloudSessionCount(this._otelService, partnerAgent?.name ?? 'unknown');
 
 		// Follow up
 		if (context.chatSessionContext && !context.chatSessionContext.isUntitled && request.sessionResource.scheme === CopilotCloudSessionsProvider.TYPE) {
@@ -2480,6 +2484,7 @@ export class CopilotCloudSessionsProvider extends Disposable implements vscode.C
 					}
 				*/
 				this.telemetry.sendMSFTTelemetryEvent('copilotcloud.chat.remoteAgentJobPullRequestReady');
+				GenAiMetrics.incrementCloudPrReadyCount(this._otelService);
 				this.logService.trace(`Job ${jobId} now has pull request #${jobInfo.pull_request.number}`);
 				this.refresh();
 				return jobInfo;

src/extension/conversation/vscode-node/userActions.ts

@@ -10,8 +10,9 @@ import { EditSurvivalResult } from '../../../platform/editSurvivalTracking/commo
 import { ILanguageDiagnosticsService } from '../../../platform/languages/common/languageDiagnosticsService';
 import { IMultiFileEditInternalTelemetryService } from '../../../platform/multiFileEdit/common/multiFileEditQualityTelemetry';
 import { INotebookService } from '../../../platform/notebook/common/notebookService';
-import { GenAiMetrics } from '../../../platform/otel/common/genAiMetrics';
 import type { EditOutcome } from '../../../platform/otel/common/genAiAttributes';
+import { emitEditFeedbackEvent, emitEditHunkActionEvent, emitEditSurvivalEvent, emitInlineDoneEvent, emitUserFeedbackEvent } from '../../../platform/otel/common/genAiEvents';
+import { GenAiMetrics } from '../../../platform/otel/common/genAiMetrics';
 import { IOTelService } from '../../../platform/otel/common/otelService';
 import { ISurveyService } from '../../../platform/survey/common/surveyService';
 import { ITelemetryService, TelemetryEventMeasurements, TelemetryEventProperties } from '../../../platform/telemetry/common/telemetry';
@@ -91,6 +92,7 @@ export class UserFeedbackService implements IUserFeedbackService {
 					characterCount: e.action.copiedCharacters,
 					lineCount: e.action.copiedText.split('\n').length,
 				});
+				GenAiMetrics.incrementUserActionCount(this.otelService, 'copy');
 				break;
 			case 'insert':
 				/* __GDPR__
@@ -116,6 +118,7 @@ export class UserFeedbackService implements IUserFeedbackService {
 					characterCount: e.action.totalCharacters,
 					newFile: e.action.newFile ? 1 : 0
 				});
+				GenAiMetrics.incrementUserActionCount(this.otelService, 'insert');
 				break;
 			case 'followUp':
 				/* __GDPR__
@@ -134,6 +137,7 @@ export class UserFeedbackService implements IUserFeedbackService {
 					participant: agentId,
 					command: result.metadata?.command,
 				});
+				GenAiMetrics.incrementUserActionCount(this.otelService, 'followup');
 				break;
 			case 'bug':
 				if (conversation) {
@@ -204,7 +208,11 @@ export class UserFeedbackService implements IUserFeedbackService {
 						isNotebookCell: e.action.uri.scheme === Schemas.vscodeNotebookCell ? 1 : 0
 					});
 
-					GenAiMetrics.recordChatEditOutcome(this.otelService, 'chat_editing', outcomes.get(e.action.outcome) ?? 'unknown', document?.languageId, e.action.hasRemainingEdits);
+					{
+						const otelOutcome = outcomes.get(e.action.outcome) ?? 'unknown';
+						emitEditFeedbackEvent(this.otelService, otelOutcome, document?.languageId ?? '', agentId, result.metadata?.responseId ?? '', 'agent', e.action.hasRemainingEdits, this.notebookService.hasSupportedNotebooks(e.action.uri));
+						GenAiMetrics.recordEditAcceptance(this.otelService, 'chat_editing', otelOutcome, document?.languageId);
+					}
 
 					if (result.metadata?.responseId
 						&& (e.action.outcome === vscode.ChatEditingSessionActionOutcome.Accepted
@@ -240,7 +248,13 @@ export class UserFeedbackService implements IUserFeedbackService {
 						measurements,
 						'edit.hunk.action'
 					);
-					GenAiMetrics.recordEditAcceptance(this.otelService, 'chat_editing_hunk', outcome, document?.languageId);
+
+					emitEditHunkActionEvent(this.otelService, outcome, document?.languageId ?? '', result.metadata?.responseId ?? '', e.action.lineCount, e.action.linesAdded, e.action.linesRemoved);
+					GenAiMetrics.recordEditAcceptance(this.otelService, 'chat_editing_hunk', outcome, document?.languageId ?? '');
+					if (outcome === 'accepted') {
+						GenAiMetrics.incrementLinesOfCode(this.otelService, 'added', document?.languageId ?? '', e.action.linesAdded);
+						GenAiMetrics.incrementLinesOfCode(this.otelService, 'removed', document?.languageId ?? '', e.action.linesRemoved);
+					}
 				}
 				break;
 			}
@@ -319,6 +333,7 @@ export class UserFeedbackService implements IUserFeedbackService {
 			},
 			'conversation.appliedCodeblock'
 		);
+		GenAiMetrics.incrementUserActionCount(this.otelService, 'apply');
 	}
 
 	handleFeedback(e: vscode.ChatResultFeedback, agentId: string): void {
@@ -359,6 +374,10 @@ export class UserFeedbackService implements IUserFeedbackService {
 			{},
 			'conversation.messageRating'
 		);
+
+		const otelRating = e.kind === vscode.ChatResultFeedbackKind.Helpful ? 'positive' : 'negative';
+		emitUserFeedbackEvent(this.otelService, otelRating, agentId, result.metadata?.sessionId ?? '', result.metadata?.responseId ?? '');
+		GenAiMetrics.incrementUserFeedbackCount(this.otelService, otelRating);
 	}
 
 	// --- inline
@@ -474,10 +493,9 @@ export class UserFeedbackService implements IUserFeedbackService {
 		this.telemetryService.sendMSFTTelemetryEvent('inline.done', sharedProps, {
 			...sharedMeasures, accepted
 		});
-		this.telemetryService.sendGHTelemetryEvent('inline.done', sharedProps, {
-			...sharedMeasures, accepted
-		});
-		GenAiMetrics.recordEditAcceptance(this.otelService, 'inline_chat', accepted ? 'accepted' : 'rejected', languageId);
+
+		emitInlineDoneEvent(this.otelService, accepted === 1, languageId, editCount, editLineCount, interactionOutcome.kind, isNotebookDocument === 1);
+		GenAiMetrics.recordEditAcceptance(this.otelService, 'inline_chat', accepted === 1 ? 'accepted' : 'rejected', languageId);
 
 		this.telemetryService.sendInternalMSFTTelemetryEvent('interactiveSessionDone', {
 			language: languageId,
@@ -513,13 +531,6 @@ export class UserFeedbackService implements IUserFeedbackService {
 }
 
 function reportInlineEditSurvivalEvent(res: EditSurvivalResult, sharedProps: TelemetryEventProperties | undefined, sharedMeasures: TelemetryEventMeasurements | undefined, otelService: IOTelService) {
-	const survivalMeasures = {
-		...sharedMeasures,
-		survivalRateFourGram: res.fourGram,
-		survivalRateNoRevert: res.noRevert,
-		timeDelayMs: res.timeDelayMs,
-		didBranchChange: res.didBranchChange ? 1 : 0,
-	};
 	/* __GDPR__
 		"inline.trackEditSurvival" : {
 			"owner": "hediet",
@@ -544,8 +555,15 @@ function reportInlineEditSurvivalEvent(res: EditSurvivalResult, sharedProps: Tel
 			"isNotebook": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "Whether the document is a notebook" }
 		}
 	*/
-	res.telemetryService.sendMSFTTelemetryEvent('inline.trackEditSurvival', sharedProps, survivalMeasures);
-	res.telemetryService.sendGHTelemetryEvent('inline.trackEditSurvival', sharedProps, survivalMeasures);
+	res.telemetryService.sendMSFTTelemetryEvent('inline.trackEditSurvival', sharedProps, {
+		...sharedMeasures,
+		survivalRateFourGram: res.fourGram,
+		survivalRateNoRevert: res.noRevert,
+		timeDelayMs: res.timeDelayMs,
+		didBranchChange: res.didBranchChange ? 1 : 0,
+	});
+
+	emitEditSurvivalEvent(otelService, 'inline_chat', res.fourGram, res.noRevert, res.timeDelayMs, res.didBranchChange, String(sharedProps?.requestId ?? ''));
 	GenAiMetrics.recordEditSurvivalFourGram(otelService, 'inline_chat', res.fourGram, res.timeDelayMs);
 	GenAiMetrics.recordEditSurvivalNoRevert(otelService, 'inline_chat', res.noRevert, res.timeDelayMs);
 }