Effort Engine

Effort Engine is a novel, real-time adjustable algorithm (bucketMul) for LLM inference, drastically improving speed on Apple Silicon by dynamically reducing matrix multiplication calculations.

This technology is a new algorithm for Large Language Model (LLM) inference, specifically designed to optimize performance on Apple Silicon chips (Swift & Metal implementation). Effort Engine allows operators to smoothly adjust the 'effort' level—the number of calculations performed—in real time during inference. For example, dropping to 25% effort yields twice the speed of regular matrix multiplications while maintaining most model quality. The system, currently implemented for Mistral, also features dynamic weight loading, letting users skip the least important 10-30% of weights for a quick, ad-hoc distillation effect. This approach bypasses full retraining, focusing on immediate, measurable gains in speed and resource efficiency.

https://github.com/kolinko/effort