Searched refs:total_tokens (Results 1 – 2 of 2) sorted by relevance
139 int total_tokens = 0; in Eval() local144 total_tokens += num_message_tokens; in Eval()145 encoded_offsets.push_back(total_tokens); in Eval()
660 int total_tokens = 0; in EmbedAndFlattenTokens() local665 total_tokens += num_message_tokens; in EmbedAndFlattenTokens()668 if (total_tokens >= max_num_total_tokens) { in EmbedAndFlattenTokens()669 message_token_offset = total_tokens - max_num_total_tokens; in EmbedAndFlattenTokens()