Searched refs:sub_token_start (Results 1 – 1 of 1) sorted by relevance
41 CodepointIndex sub_token_start = codepoint_span.first; in AppendTokenSymbols() local55 type, CodepointSpan{sub_token_start, sub_token_start + num_codepoints}, in AppendTokenSymbols()59 next_match_offset = sub_token_start + num_codepoints; in AppendTokenSymbols()62 sub_token_start = next_match_offset; in AppendTokenSymbols()