Skip to main content
Any Isaacus model that analyzes text does so by breaking it down into tokens. This page explains how that works and how you can use our tokenizers in the same way that Isaacus models do. If you’re actually looking to use the Isaacus API, please see our quickstart guide or API reference instead. You can also find the original Kanon tokenizer here and the Kanon 2 tokenizer here.

​
Tokens

Isaacus models work, like all other AI models, by converting their inputs into numbers. Specifically, they convert text into a restricted set of numbers, known as a vocabulary. Each number in the vocabulary represents a token. A token is a unit of text. A token can be thought of as a word, part of a word, number, punctuation mark or space. Any text input sent to an Isaacus model gets broken down into tokens. The process of breaking down text into tokens is called tokenization. A tokenizer is a tool that performs tokenization. Different models have different tokenizers. Different tokenizers have different vocabularies and different tokenization rules. If we were to send the sentence “The client is happy.” to an Isaacus model that uses the Kanon tokenizer, it would first break it down into the tokens “The”, “client”, “is”, “happy”, and ”.”. A vocabulary can only contain so many tokens. Most of the tokens in a vocabulary will be common words but they can also be subwords, as mentioned earlier. For example, there are quite a lot of words that end in the suffix “-ing” so you will often see “-ing” as a token in vocabularies. Having subwords in a vocabulary means that, if a tokenizer encounters a rare word, for example, “discombobulation”, it can break it down into more common subwords, like “disc”, “omb”, “ob” and “ulation”.

​
Isaacus tokenizers

Models based on the original Kanon architecture use the original Kanon tokenizer and those based on the newer Kanon 2 architecture use the Kanon 2 tokenizer. Both tokenizers are freely available on Hugging Face. Consequently, you can use them with the Transformers library to tokenize text in the same way that Isaacus models do. 
from transformers import AutoTokenizer

kanon_2_tokenizer = AutoTokenizer.from_pretrained("isaacus/kanon-2-tokenizer")

text = "The client is happy."

tokens = kanon_2_tokenizer.tokenize(text)

print(tokens)