SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

Alberto Marchisio, Davide Dura, Maurizio Capra, Maurizio Martina, Guido Masera, Muhammad Shafique

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Transformers' compute- intensive operations pose enormous challenges for their deployment in resource- constrained EdgeAI / tiny ML devices. As an established neural network compression technique, quantization reduces the hardware computational and memory resources. In particular, fixed-point quantization is desirable to ease the computations using lightweight blocks, like adders and multipliers, of the underlying hardware. However, deploying fully-quantized Transformers on existing general-purpose hardware, generic AI accelerators, or specialized architectures for Transformers with floating-point units might be infeasible and/or inefficient. Towards this, we propose SwiftTron, an efficient specialized hardware accelerator designed for Quantized Transformers. SwiftTron supports the execution of different types of Transformers' operations (like Attention, Softmax, GELU, and Layer Normalization) and accounts for diverse scaling factors to perform correct computations. We synthesize the complete SwiftTron architecture in a 65 nm CMOS technology with the ASIC design flow. Our Accelerator executes the RoBERTa-base model in 1.83 ns, while consuming 33.64 mW power, and occupying an area of 273 mm 2• To ease the reproducibility, the RTL of our SwiftTron architecture is released at https://github.com/albertomarchisio/SwiftTron.

Original languageEnglish (US)
Title of host publicationIJCNN 2023 - International Joint Conference on Neural Networks, Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665488679
StatePublished - 2023
Event2023 International Joint Conference on Neural Networks, IJCNN 2023 - Gold Coast, Australia
Duration: Jun 18 2023Jun 23 2023

Publication series

NameProceedings of the International Joint Conference on Neural Networks


Conference2023 International Joint Conference on Neural Networks, IJCNN 2023
CityGold Coast


  • ASIC
  • Attention
  • GELU
  • Hardware Architecture
  • Layer Normalization
  • Machine Learning
  • Quantization
  • Softmax
  • Transformers

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers'. Together they form a unique fingerprint.

Cite this