In Isaac Asimov’s I, Robot classic story collection, the robots are subject to the famous Three Laws of Robotics, which essentially set strict priorities for them that should always override all the lower ones (do not harm humans, obey orders, and protect yourself, in that order). In mathematical terms, such priorities would essentially be incomparable - any level of concern about human welfare for example should instantly override any order. This is a problem in Little Lost Robot, for example, where a class of robots meant for use in a dangerous industrial environment causes more trouble than they’re worth by dragging away their own operators forcefully whenever they’re working around even entirely tolerable levels of background radiation. There’s no real way to formalize this kind of strict priority, and it would be impractical exactly for the reasons above - a smart enough robot/AI could conceive of virtually any action it can take causing indirectly human harm, and therefore the only logical response would be paralysis. In practice, modern AI systems are built by optimising for one single loss function, however complex and stratified the training may be. This means that whatever priorities were set to it (for example, an LLM can be trained to reproduce text but also finetuned specifically to not respond rudely to its user), they will always be performing some kind of trade-off, operating on some Pareto frontier in which all different goals are weighted with respect to each other (and for example, it is possible usually to “jailbreak” them into talking rudely to you if you create the conditions in which this seems the most appropriate way to reproduce the text accurately).

The Gemma 4 Good Hackathon is a Kaggle competition that has been sponsored by Google to celebrate the release of their Gemma 4 family of local multimodal LLMs, comprising four models: e2b, e4b, each respectively only loading 2 or 4 billion parameters at a time, 26b which is a Mixture-of-Experts model, and 31b. They’ve almost immediately been available on Ollama too in quantized form. The theme of the competition was to:

[…] create a solution that addresses a real-world challenge using Gemma 4 models, whether that’s an application that helps millions or a specialized model that could exponentially scale innovation.

A few months ago I released the first version of MiniML, a small machine learning framework powered by Jax. The main idea behind it was to have a very lean toolset to build models that would be as simple to train as a Scikit-learn one, while offering the same level of flexibility as PyTorch. I wrote all about it here.

A few versions later I’ve expanded on that base with a few much needed basic modules for machine learning - radial basis function networks, multi-head self attention, a testing system that uses gold standard data from Backblaze, support for non-Scipy optimizers and more. In this release though I want to focus on the addition of two features that support research in one particular phenomenon: “grokking”. Let’s see what is it!

On November 12, 2012, Randall Munroe’s famous xkcd comic published Up Goer Five, a blueprint and explanation of the Apollo V rocket written using only the 1000 most common words of the English language (as he estimated them). Later on, on November 24, 2015, came out Thing Explainer, an entire illustrated book of similar explanations for other objects and concepts. The “only the most common 1000 words” style of writing sounds sometimes stilted, sometimes a bit funny, but these texts certainly prove that it’s enough to talk virtually about anything.

In the age of LLMs, would it be possible to have a training set built only on the most common 1000 words of the English language?

Let’s try.