Searched refs:last_token_index (Results 1 – 1 of 1) sorted by relevance
140 int last_token_index; in FindFullTokensSpanInWindow() local141 last_token_index = internal::FindLastFullTokenIndex( in FindFullTokensSpanInWindow()143 wordpiece_window_end = last_token_index == (word_starts.size() - 1) in FindFullTokensSpanInWindow()145 : word_starts[last_token_index + 1]; in FindFullTokensSpanInWindow()147 *num_tokens = last_token_index - *first_token_index + 1; in FindFullTokensSpanInWindow()