Syntiant Achieves 50 Percent Acceleration of Large Language Models for Edge Devices

Syntiant Corp., a leader in edge AI deployment, announced results of its first optimizations that reduce the computational footprint of leading large language model (LLM) architectures, enabling these massive neural networks to run on cloud-free, always-on devices at the edge of networks.

“We are focused on making edge AI a reality, bringing low power, highly accurate intelligence to edge devices,” said Kurt Busch, CEO at Syntiant. “Our core model optimizations enable considerable processing time acceleration across a range of compute platforms. Those same optimizations, which Syntiant has deployed across millions of devices worldwide, have been successfully applied to LLMs, bringing conversational speech to the edge, serving as the new interface between humans and machines.”

Recommended AI News: Riding on the Generative AI Hype, CDP Needs a New Definition in 2024

Recommended AI News: McAfee Debuts Deepfake Audio Detection Tech at CES 2024 to Counter AI-Generated and Disinformation

Syntiant used a novel algorithm to determine the sparsity fraction of LLMs to generate significant speedups in output token generation and reduce memory footprint. The company achieved 50% sparsity with minimal accuracy loss when computing with 8- bit quantized weights, significantly improving interpretability and processing power, while reducing cloud costs. These improvements, along with a custom SIMD (single instruction/multiple data) kernel and several other algorithmic innovations, enable Syntiant to achieve a 50% increase in output token generation speed on a LLaMa-7B benchmark.

“Syntiant continues to lead the category of low power NLP with production-ready solutions that directly address voice AI workload efficiency and could enable smarter conversational endpoints with generative AI. At M12, Microsoft’s Venture Fund, we’re proud to back Syntiant who we feel will help establish a ‘first mile’ standard for future speech-to-speech AI use cases,” said Michael Stewart, partner at M12.

Recommended AI News: ALDI’s Digital Supply Chain Leap in Asia with NEO Platform

Demoed live at CES 2024 in Las Vegas, Syntiant’s optimizations demonstrate an adjacent class of LLMs that run entirely at the edge. The ability to enable significant processing time acceleration brings meaningful benefits to end users from both latency and privacy perspectives, across a wide variety of consumer and commercial use cases, from earbuds to automobiles.

[To share your insights with us, please write to sghosh@martechseries.com]