Searched refs:token_texts (Results 1 – 1 of 1) sorted by relevance
108 std::vector<std::string> token_texts; in PreTokenize() local109 std::transform(tokens.begin(), tokens.end(), std::back_inserter(token_texts), in PreTokenize()112 return token_texts; in PreTokenize()