Terms in languages without word delimiters (CJK, Thai, etc.) are not detected

[aromarious]

Problem

Contextive currently relies on delimiter-based tokenisation (spaces, parentheses, etc.) and regex-based splitting (camelCase, snake_case) to identify candidate terms. This works well for languages that use spaces between words, but languages that don't use spaces to delimit words — such as Japanese, Chinese, Korean, Thai, Lao, and Khmer — cannot match glossary terms at all.

Reproduction

1. Create a glossary file:

contexts:
  - terms:
    - name: 注文
      definition: An order placed by a customer

2. Open a file containing the text 注文が届く
3. Hover over 注文 — no hover result is shown

Why this happens

The Tokeniser extracts the entire 注文が届く as a single token (no delimiters within it). The CandidateTerms regex then attempts to split it by camelCase/snake_case patterns, but since CJK characters are not in the \p{Lu} / \p{Ll} ranges used by the regex, no splitting occurs. The full string 注文が届く is looked up as-is in the index, which only contains 注文, so no match is found.

Affected languages

Any language that does not use spaces between words:

Language	Script	Example
Japanese	Kanji / Kana	注文が届く
Chinese	Hanzi	购物车管理
Korean	Hangul	주문을처리
Thai	Thai script	สวัสดีครับ
Lao	Lao script	ພາສາລາວ
Khmer	Khmer script	ភាសាខ្មែរ
Myanmar	Myanmar script	မြန်မာဘာသာ

[aromarious]

One approach would be to add a substring matching path for CJK tokens in the hover lookup.

The idea:

• Add a containsCJK function that checks whether a token contains CJK characters (by Unicode code point range)
• When the token contains CJK, bypass the candidateTermsFromToken regex (which only handles Latin case patterns) and keep the token whole
• Instead of looking up exact keys in the index, iterate over all index keys and check if any key is a substring of the normalised token
• When the token does not contain CJK, the existing hash-based O(1) lookup is used unchanged

This means:

• No impact on existing English / Latin / Cyrillic behaviour — the CJK branch is only entered when the token contains CJK characters
• The substring search is O(K) where K is the number of index keys, but this only applies to CJK tokens
• This does not cover Thai, Lao, Khmer, or Myanmar — but it addresses the three most widely used non-delimited languages first. The same approach could be extended to other scripts later.

I have a working implementation on my fork (branch feat/cjk-substring-matching): https://github.com/aromarious/contextive/tree/feat/cjk-substring-matching

All existing tests pass (Core: 71, LanguageServer: 204), and I've added tests for CJK detection and index lookup. Happy to discuss the approach before opening a PR.

[chrissimon-au]

Hi @aromarious, thanks so much for your contribution with this!

My only concern is that the substring search is O(n) because I have had reports of users with 25,000+ terms and the hover function is called frequently by most LSP implementations - on every mouse movement - so there could be a performance penalty there.

However, since you've built this to only activate the slower search when the CJK characters are detected, those users should be protected and this only becomes an issue when a CJK user has a large number of terms. In that case, another optimisation could be to pre-index the file when the file is loaded - the O(n) search could be done once, all instances of found terms could be indexed by location, for fast response when the hover request is issued - but I'm happy for that to be left for a later change once the need is confirmed.

In the meantime, I've taken a look at your branch, and it looks very close to being complete - the only thing I'm not sure if I see is a test to assert on the outcome of the findMatchingTermsBySubstring function?

I think it would be great to add an E2e test around the hover function which would cover the branch to findMatchingTermsBySubstring and that function. The appropriate place would be a new case in the test found here: https://github.com/aromarious/contextive/blob/main/src/language-server/Contextive.LanguageServer.Tests/E2e/HoverTests.fs#L66

You could add the cjk terms to three.yml in this folder - https://github.com/aromarious/contextive/tree/main/src/language-server/Contextive.LanguageServer.Tests/fixtures/completion_tests - or add a new glossary file in that folder (e.g. cjk.yml) - that becomes the last argument in the case tuple. Let me know if any questions.

If you are able to add that test, feel free to setup a PR and I'll get it merged.

Thanks again!

[aromarious]

Hi! I've opened a PR to tackle this: #119

Since CJK text doesn't have word delimiters, the approach uses substring matching — when a token contains CJK characters, glossary term keys are matched as substrings within the token rather than relying on the existing delimiter-based splitting.

Tested with Japanese (Kanji, Katakana), Chinese, and Korean.

Would love any feedback!

[chrissimon-au]

An update on this - the github action is locking up during the vscode extension test job on linux and macos (proceeded successfully on windows). Can I confirm which OS you were running/testing on? I'll pull down locally and confirm on linux and macos as well.

Separately, taking another look at the implementation, since it just confirms if the indexed normalized term is found within the 'token', does that mean that it would show the definition even if hovering in a different part of the token?

e.g. given your example above:

contexts:
  - terms:
    - name: 注文
      definition: An order placed by a customer

If you open a file containing the text 注文が届く and then hover over 届 then I think, based on the implementation, it would still show the definition for 注文 - is that what you observe? Is that what you expect?

Do you think we should identify the index of the found substring and then confirm that the hover position is over that particular substring?

[aromarious]

Hi @chrissimon-au, thanks for the feedback on both points.

CI lockup: I tested on macOS (darwin-arm64). I also ran CI on my fork and the VsCode extension test jobs locked up on Linux and macOS in the same way, while Language Server tests passed on all three OSes. I haven't been able to identify the root cause.

• Fork CI run: https://github.com/aromarious/contextive/actions/runs/22651696807

Local VsCode extension test steps and results (macOS darwin-arm64)

$ cd src
$ dotnet publish language-server/Contextive.LanguageServer/Contextive.LanguageServer.fsproj \
  -o language-server/Contextive.LanguageServer/publish
$ mkdir -p vscode/contextive/dist
$ cp language-server/Contextive.LanguageServer/publish/Contextive.LanguageServer \
  vscode/contextive/dist/
$ cd vscode/contextive
$ npm install
$ dotnet fable src && npx webpack
$ cd test && dotnet fable && cd ..
$ npx webpack -c webpack.test.config.js
$ CONTEXTIVE_DEBUG=1 MOCHA_FGREP="Single-Root" node ./out/runTest.js

  ✔ Single-Root Workspace Contextive Activation Tests Contextive is Active: 1ms
  ✔ Single-Root Workspace Contextive Activation Tests CSharp Extension is Active: 351ms
  ✔ Single-Root Workspace Contextive Activation Tests Extension has no default path config: 0ms
  ✔ Single-Root Workspace Contextive Activation Tests Language Client becomes Ready: 0ms
  ✔ Single-Root Workspace Initialize Tests Extension has Initialize Project Command: 2ms
  ✔ Single-Root Workspace Initialize Tests When config is undefined, and default glossary file doesn't exist, initialize Command should create and open in default location: 1190ms
  ✔ Single-Root Workspace Initialize Tests When glossary file exists, Initialize Command should open existing file: 324ms
  ✔ Single-Root Workspace Initialize Tests When config is defined, and glossary file doesn't exist, Initialize Command should create and open in configured location: 406ms
  ✔ Single-Root Workspace Initialize Tests New glossary file should show new term in completion: 894ms
  ✔ Single-Root Workspace Contextive Completion Tests Completion returns expected list: 132ms
  ✔ Single-Root Workspace Invalid Schema Tests Should recover from an invalid schema: 388ms
  ✔ Single-Root Workspace Hover Tests Hover returns expected content: 309ms
  ✔ Single-Root Workspace Hover Tests Contextive Hover results appear last: 714ms
  13 passing (5s)

$ CONTEXTIVE_DEBUG=1 MOCHA_FGREP="Multi-Root" node ./out/runTest.js

  1) Multi-Root Workspace Completion in a root finds terms for rootA
  ✔ Multi-Root Workspace Completion in a root finds terms for rootB: 192ms
  1 passing (723ms)
  1 failing

Note: The Multi-Root rootA failure also reproduces on the main branch, so it is a pre-existing issue unrelated to this change.

Hover position accuracy: You're absolutely right — the previous implementation would show the definition for 注文 even when hovering over 届. I've now implemented the cursor position check you suggested.

For example, with 注文が届く:

• Hovering over 注 or 文 → shows the definition for 注文
• Hovering over が, 届, or く → no hover result

The approach:

• Added Lexer.getWithStart to return the token's start column alongside the token string
• The hover handler computes cursorOffsetInToken = cursor position - token start
• findMatchingTermsBySubstring now checks that the cursor offset falls within the matched key's range (IndexOf result to IndexOf + key.Length)
• The non-CJK path is unaffected

I've also added E2E tests for both positive cases (cursor on matched substring) and negative cases (cursor on non-matching characters).

PR: #120

[aromarious]

One more thing I noticed while testing: when hovering over 注文 in 注文が届く, the definition is correctly shown and the cursor position check works, but the highlight (underline) covers the entire token 注文が届く rather than just 注文.

This happens because the Hover response doesn't include a Range, so VS Code highlights the full token.

A possible improvement: propagate the matched substring's position (from IndexOf) and length (from key.Length) up to the hover response, and set Hover.Range to cover only the matched portion. This would only affect the CJK path — Latin tokens are already highlighted correctly at the token level.

I considered two approaches:

1. Change the return types of findMatchingTermsBySubstring and termFilterForCandidateTermsWithIndex to include match position info alongside the terms. However, this would require changing the Filter type (Context seq -> Context seq) in GlossaryFile, which would have a wider impact across the codebase.

2. Use a ref cell passed as a parameter to findMatchingTermsBySubstring and termFilterForCandidateTermsWithIndex (both private functions in the Filtering module). The ref captures the match range during filtering, and the handler reads it afterward to construct the Hover response with Range set. This keeps the change localised to Hover.fs without touching the Filter type or Core.

I think approach 2 is preferable since it avoids changing the shared Filter type, but I'd appreciate your thoughts.

I'd be happy to submit a PR for this if you think it's worth doing.

---

Separately, I noticed that hovering over account's shows the definition for Account twice. I'll open a separate issue for this.