Searched refs:TokenizeWithFilter (Results 1 – 2 of 2) sorted by relevance
38 return TokenizeWithFilter(text, [&](char32 codepoint) { in TokenizeOnDelimiters()48 return TokenizeWithFilter(text, [](char32 codepoint) { in TokenizeOnWhiteSpacePunctuationAndChineseLetter()
67 std::vector<Token> TokenizeWithFilter(const absl::string_view input, in TokenizeWithFilter() function