Searched refs:is_token_in_span (Results 1 – 1 of 1) sorted by relevance
343 bool is_token_in_span; in CodepointSpanToTokenSpan() local345 is_token_in_span = codepoint_start < selectable_tokens[i].end && in CodepointSpanToTokenSpan()348 is_token_in_span = codepoint_start <= selectable_tokens[i].start && in CodepointSpanToTokenSpan()351 if (is_token_in_span && !selectable_tokens[i].is_padding) { in CodepointSpanToTokenSpan()