believe it or not this is already working.
and you end up not caring about the maintainability, you just spend more tokens fixing whatever is wrong. You do have to start with good design, good specs, but its working.
“you just spend more tokens”
*****
“Tokens are small units of data — words, sub-words, characters, or pixel patches — that AI models process during training and inference.”
“Context matters too. The word “lie” could mean resting flat or speaking falsely. During training, a model learns these distinctions and assigns different token IDs to the same word depending on surrounding context.
“Tokenization is not limited to text. Visual models break images and video into patches of pixels or voxels, each mapped to a discrete token. Audio models sometimes convert sound clips into spectrograms — visual representations of frequency over time — which are then tokenized like images. Other audio systems skip that step and extract semantic tokens that capture the meaning of spoken language rather than raw acoustic data.
“Efficient tokenizers reduce the total number of tokens a model must process, which directly lowers computing costs.”
“Training an AI model begins with tokenizing the full training dataset. For large language models, that dataset can contain billions or trillions of tokens. A well-established pretraining scaling law holds that larger token counts during training lead to higher-quality models.
“The core training loop works through prediction. The model sees a sequence of tokens and tries to guess the next one. When it guesses wrong, internal parameters update to improve accuracy on the next attempt. This cycle repeats across the entire dataset until the model reaches a target accuracy threshold — a state called model convergence.”
More at:
https://www.sentisight.ai/tokens-explained-new-currency-of-generative-ai/