Searched refs:TokenizeSingleToken (Results 1 – 3 of 3) sorted by relevance
62 TokenizerResult BertTokenizer::TokenizeSingleToken(const std::string& token) { in TokenizeSingleToken() function in libtextclassifier3::BertTokenizer
109 TokenizerResult TokenizeSingleToken(const std::string& token);
367 tokenizer_->TokenizeSingleToken(token_text); in PrepareText()