Generative AI Hardware Drive Accessibility and Efficiency


  • Generative AI hardware innovates with memory architecture to cut costs, making large language models more affordable.
  • SambaNova introduces “composition-of-experts” to democratize large models, offering efficiency without sky-high expenses.
  • Tenstorrent seeks efficiency by decoupling model architecture from training hardware, ensuring models are fit for practical use, not just training.

Innovations in Generative AI Hardware are making strides toward affordability, accessibility, and efficiency, challenging the constraints posed by the exponential growth in the size of large language models (LLMs). In a recent panel discussion, industry leaders shared insights into their strategies for addressing these pressing challenges.

Marshall Choy, Senior VP of Products at SambaNova Systems, highlighted the critical role of memory architecture in bringing down the cost of using LLMs. As LLMs boast parameter counts reaching the billions or trillions, the focus has shifted towards memory as the bottleneck. SambaNova Systems has adopted a three-tier memory architecture, addressing latency, bandwidth, and capacity within a single framework. This innovative approach aims to economically scale the usage of LLMs, where memory efficiency is the linchpin.

Democratizing large models

The ballooning size of LLMs presents a significant accessibility challenge. Once a model exceeds a trillion parameters, the associated hardware and operating costs become prohibitive, leaving their utilization confined to a select few. To make large models accessible to a wider audience, SambaNova Systems has introduced a novel concept known as “composition-of-experts.”

This approach diverges from the conventional “mixture-of-experts” paradigm, where complex predictive modeling problems are divided into subtasks. Instead, SambaNova trains domain expert models for precision and task relevance, assembling a trillion-parameter composition-of-experts model. This model can be continually trained on new data without sacrificing prior learning, all while minimizing compute latency and reducing the costs associated with training, fine-tuning, and inferencing.

Efficiency through model techniques

Efficiency in generative AI hardware goes beyond the hardware itself; it extends to the relationship between model architecture and the hardware it runs on. Matt Mattina, VP of AI Hardware and Models at Tenstorrent, emphasized the importance of breaking the feedback loop where model architecture is shaped by the hardware it’s trained on.

Tenstorrent adopts techniques like network architecture search with hardware in the loop, allowing trainers to specify the hardware for inference during training. This paradigm shift ensures that models are tailored not for the training machine but for the ultimate inference machine, leading to more efficient models for practical use.

Specialization at the system level

AI is an ever-evolving field, posing challenges in balancing dedicated chips and custom silicon with system flexibility. Jeff Wittich, Chief Product Officer at Ampere Computing, offers a perspective that favors specialization at the system level. He asserts that this approach provides the flexibility to mix and match components, creating versatile solutions capable of adapting to the rapidly changing AI hardware landscape.

Traditionally, creating and commercializing new hardware takes several years. Ampere’s partnership with companies developing various training and inference accelerators aims to achieve the right balance. Ampere envisions improved performance and efficiency by coupling general-purpose CPUs with accelerators specialized in specific tasks.

Integration benefits and flexibility

Wittich highlights the importance of integration, which should ideally enhance performance and efficiency without sacrificing flexibility. The fusion of general-purpose CPUs with specialized accelerators is seen as a promising avenue. Over time, the tight integration of these accelerators with CPUs is expected to further optimize AI workloads. The key principle remains: integration should enhance capabilities without imposing restrictions.

Disclaimer. The information provided is not trading advice. Cryptopolitan.com holds no liability for any investments made based on the information provided on this page. We strongly recommend independent research and/or consultation with a qualified professional before making any investment decisions.

Share link:

Brenda Kanana

Brenda Kanana is an accomplished and passionate writer specializing in the fascinating world of cryptocurrencies, Blockchain, NFT, and Artificial Intelligence (AI). With a profound understanding of blockchain technology and its implications, she is dedicated to demystifying complex concepts and delivering valuable insights to readers.

Most read

Loading Most Read articles...

Stay on top of crypto news, get daily updates in your inbox

Related News

Peter Thiel
Subscribe to CryptoPolitan