Skip to content

Fix HTML-escaped tool-call args and preserve upstream token usage (stream & non-stream)#239

Merged
CJackHwang merged 2 commits intodevfrom
codex/fix-escaping-issues-and-token-counting
Apr 7, 2026
Merged

Fix HTML-escaped tool-call args and preserve upstream token usage (stream & non-stream)#239
CJackHwang merged 2 commits intodevfrom
codex/fix-escaping-issues-and-token-counting

Conversation

@CJackHwang
Copy link
Copy Markdown
Owner

Motivation

  • Tool-call XML/markup arguments were containing HTML-escaped entities (e.g. >) which caused command execution deviations when parsed as tool arguments.
  • Upstream responses sometimes report token usage using input_tokens/output_tokens or string-valued numbers, causing third-party clients (e.g. Vercel deployments) to show inaccurate token counts when the translator fell back to estimated values.

Description

  • Unescape HTML entities at parse completion for tool calls by applying html.UnescapeString to Name and recursively to Input values in internal/toolcall/toolcalls_parse.go so arguments like > become > again.
  • Extend streaming usage extraction in internal/translatorcliproxy/stream_writer.go so extractOpenAIUsage also considers input_tokens/output_tokens and accepts string-encoded numeric fields by updating toInt to parse string via strconv.Atoi.
  • Preserve upstream usage in non-stream translation by adding extractOpenAIUsageFromJSON and injectNonStreamUsageMetadata in internal/translatorcliproxy/bridge.go and applying usage metadata into converted responses for Claude (usage.input_tokens/output_tokens) and Gemini (usageMetadata) so translated non-stream responses carry the original token counts.
  • Add tests to cover the fixes: TestParseToolCallsUnescapesHTMLEntityArguments in internal/toolcall/toolcalls_test.go, TestExtractOpenAIUsageSupportsResponsesUsageFields in internal/translatorcliproxy/stream_writer_test.go, and TestFromOpenAINonStreamPreservesResponsesUsageShape in internal/translatorcliproxy/bridge_test.go.

Testing

  • Ran go test ./internal/translatorcliproxy ./internal/toolcall and both packages passed (ok ds2api/internal/translatorcliproxy and ok ds2api/internal/toolcall).
  • Added unit tests mentioned above and they succeeded in the test run.

Codex Task

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 7, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
ds2api Ready Ready Preview, Comment Apr 7, 2026 4:55am

@CJackHwang CJackHwang marked this pull request as ready for review April 7, 2026 04:38
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77a401fb19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@CJackHwang CJackHwang merged commit 1c95942 into dev Apr 7, 2026
3 checks passed
@CJackHwang CJackHwang deleted the codex/fix-escaping-issues-and-token-counting branch April 7, 2026 04:56
CJackHwang added a commit that referenced this pull request Apr 7, 2026
Merge pull request #239 from CJackHwang/codex/fix-escaping-issues-and-token-counting

Fix HTML-escaped tool-call args and preserve upstream token usage (stream & non-stream)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant