PrismML has released a compressed version of the AI Model Bonsai 27B, which reduces the 27 billion parametric model to 3.9 GB, enabling it to operate locally on smartphones. Using Decrypt to quote company information, the model can reach 11 tokens per second on iPhone 17 Pro Max and become one of the first 27B-grade models to cross the memory threshold for mobile phones.

Such models usually require high levels of memory. According to the text, a 27 billion-billion-parameter model that operates in half-accuracy usually requires approximately 54 GB memory, many of which are difficult to carry. PrismML, this time, compresses the model to the size of a mobile phone that can be operated, using end-side reasoning capabilities instead of cloud deployment.

Keep most of the performance after compression

PrismML uses a program from Caltech related intellectual property. The core approach is to compress model weights from 16-bit floating point accuracy to very low bits expression. The binary version retains only +1 and -1, the three-value version adds 0 to this state and allows a 16-bit scaling factor to be shared for each 128 weights.

By company-disposed data, the two-value version corresponds to about 1.125 bit per weight, with a volume of about one-fourth of the original full accuracy model; the three-value version is about 1.71 bit. PrismML states that the three values Bonsai 27B achieved an average score of 94.6 per cent of the full accuracy model in 15 baseline tests and a version of 76.11.

  • Cellular version approximately 3.9 GB
  • iPhone 17 Pro Max speeds about 11 token/s
  • M5 Pro equipment approximately 26 token/s

Focus on mobile phones and low-quality equipment

Unlike the common low-bit quantification model, Bonsai 27B does not retain a higher "exception" for some sensitive layers. It has end-to-end compression of the embedded layer, the focus layer and the language model output. Typically, such a retention of a high-precision layer would be in exchange for a more stable effect, but would also boost the size of the model.

PrismML also uses a mix of attention structure, and about 75% of the layers use linear attention instead of complete secondary attention. According to the article, this makes the 262,000 token context window more viable on end-side devices, otherwise the calculation costs on cell phone hardware would be excessive.

PrismML launched Bonsai 8B in March this year, when the model was 1.15 GB. This upgrade from 8B to 27B is seen as a step towards a higher-capacity model of its compressed route. According to the article, this size is closer to a stable zone of continuous reasoning, tool calls and multistep proxy missions.

Apples are exposed to relevant compression techniques.

In addition to the model itself, the progress of commercialization has also received attention. The Chief Executive Officer of PrismML, Babak Hassibi, indicated to CNBC that the company was in initial contact with apples, which were assessing the suitability of this compression technique for end-side use.

PrismML also indicated that the next step was to launch a compressed Gemma model, which would then advance a larger scale model. The article also mentions that the company also provides a decoding layer called DSpark, which is used to increase the passivity efficiency in the GPU scene. On NVIDIA H100, this option can result in an increase of about 1.37 times and does not change the distribution of output results.

If such ultra-low-bit compression programmes can be stabilized, the threshold for mobile phones and low-end computers to operate medium-sized models will continue to decline, and the deployment space for end-side AI may be expanded.