AraBit-1.58: Morphology-Aware Embeddings Reduce the Cost of Ternary Quantization in Arabic Text Simplification

Extreme quantization restricts Transformer weights to the ternary set {?1,0,+1}, but whether this transfers to morphologically rich languages is untested. Arabic is demanding, because its clitic-heavy, root-and-pattern morphology yields a vast inventory of surface forms that subword tokenizers segment at linguistically meaningless boundaries. AraBit-1.58 couples a Morphology-Aware Tokenization and Embedding layer (MATE) with a BitNet b1.58 ternary backbone. MATE decomposes each word into proclitics, root, morphosemantic pattern, and enclitics, embeds these separately, and gates to a subword fallback when analysis fails. We evaluate on the SAMER Arabic simplification corpus under a controlled two-by-two design over three seeds that varies only the front end and the weight precision. Ternary quantization compresses weight memory 2.85×, from 104.72 to 36.70?MB, but without a fused ternary kernel it cuts neither latency nor energy. The comparison confirms our pre-specified hypothesis: quantization costs 9.43 SARI with subword input against 7.49 SARI with morphological input, an interaction of +1.93 SARI significant across seeds (paired t(2) =4.55, one-tailed p=0.023). Morphological grounding therefore buys robustness to extreme quantization rather than higher absolute quality, measured on SARI at small scale.

[1]       A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, ?. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 5998–6008, 2017.

[2]       J. Lee-Thorp, J. Ainslie, I. Eckstein, and S. Onta??n. FNet: Mixing tokens with Fourier transforms. Proceedings of NAACL-HLT 2022, 4296–4313, 2022.

[3]       V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.

[4]       J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, 4171–4186, 2019.

[5]       W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23, 1–39, 2022.

[6]       H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei. BitNet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023.

[7]       S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei. The era of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024.

[8]       S. Ma, H. Wang, S. Huang, X. Zhang, Y. Hu, T. Song, Y. Xia, and F. Wei. BitNet b1.58 2B4T technical report. arXiv preprint arXiv:2504.12285, 2025.

[9]       L. Liu, Z. Zheng, C. Wang, T. Su, and Z. Yang. Binary neural networks for large language model: A survey. arXiv preprint arXiv:2502.19008, 2025.

[10]     R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th annual meeting of the association for computational linguistics, 1715–1725, 2016.

[11]     A. Alshanqiti, A. Alkhodre, A. Namoun, S. Albouq, and E. Nabil. A transformer Seq2Seq model with fast Fourier transform layers for rephrasing and simplifying complex Arabic text. International Journal of Advanced Computer Science and Applications, 14, 888–898, 2023.

[12]     G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.

[13]     A. Alshanqiti, A. Namoun, A. Alsughayyir, A.M. Mashraqi, A.R. Gilal, and S.S. Albouq. Leveraging DistilBERT for summarizing Arabic text: An extractive dual-stage approach. IEEE Access, 9, 135594–135607, 2021.

[14]     A. Alsughayyir, and A. Alshanqiti. A light but efficient switch transformer model for Arabic text simplification. International Journal of Advanced and Applied Sciences, 12, 71–78, 2025.

[15]     E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. International conference on learning representations (ICLR), 2023.

[16]     J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of machine learning and systems (MLSys), 2024.

[17]     T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 2022.

[18]     A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. Proceedings of LREC 2014, 1094–1101, 2014.

[19]     O. Obeid, N. Zalmout, S. Khalifa, D. Taji, M. Oudah, B. Alhafni, G. Inoue, F. Eryani, A. Erdmann, and N. Habash. CAMeL tools: An open source python toolkit for Arabic natural language processing. Proceedings of the 12th language resources and evaluation conference (LREC), 7022–7032, 2020.

[20]     A. Abdelali, K. Darwish, N. Durrani, and H. Mubarak. Farasa: A fast and furious segmenter for Arabic. Proceedings of NAACL-HLT 2016: demonstrations, 11–16, 2016.

[21]     A.M. Alshanqiti, S. Albouq, A.B. Alkhodre, A. Namoun, and E. Nabil. Employing a multilingual transformer model for segmenting unpunctuated Arabic text. Applied Sciences, 12, 10559, 2022.

[22]     N. Habash. Introduction to Arabic natural language processing. Morgan & Claypool Publishers, 2010.

[23]     W. Antoun, F. Baly, and H. Hajj. AraBERT: Transformer-based model for Arabic language understanding. Proceedings of the 4th workshop on open-source arabic corpora and processing tools (OSACT4), LREC 2020, 9–15, 2020.

[24]     Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.

[25]     J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 127063, 2024.

[26]     B. Zhang, and R. Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 2019.

[27]     I. Loshchilov, and F. Hutter. Decoupled weight decay regularization. International conference on learning representations (ICLR), 2019.

[28]     B. Alhafni, R. Hazim, J.D. Pineros Liberato, M. Al Khalil, and N. Habash. The SAMER Arabic text simplification corpus. Proceedings of LREC-COLING 2024, 16079–16093, 2024.

[29]     M. Al Khalil, N. Habash, and Z. Jiang. A large-scale leveled readability lexicon for standard Arabic. Proceedings of LREC 2020, 3053–3062, 2020.

[30]     S.S. Al-Thanyyan, and A.M. Azmi. Simplification of Arabic text: A hybrid approach integrating machine translation and transformer-based lexical model. Journal of King Saud University – Computer and Information Sciences, 35, 101662, 2023.

[31]     M.K. Eddine, N. Tomeh, N. Habash, J. Le Roux, and M. Vazirgiannis. AraBART: A pretrained Arabic sequence-to-sequence model for abstractive summarization. Proceedings of the seventh arabic natural language processing workshop (WANLP), 31–42, 2022.

[32]     E.M.B. Nagoudi, A. Elmadany, and M. Abdul-Mageed. AraT5: Text-to-text transformers for Arabic language generation. Proceedings of the 60th annual meeting of the association for computational linguistics, 628–647, 2022.

[33]     T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in neural information processing systems, 36, 10088–10115, 2023.

[34]     W. Xu, C. Napoles, E. Pavlick, Q. Chen, and C. Callison-Burch. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4, 401–415, 2016.

[35]     T. Zhang, V. Kishore, F. Wu, K.Q. Weinberger, and Y. Artzi. BERTScore: Evaluating text generation with BERT. International conference on learning representations (ICLR), 2020.

[36]     M. El-Haj, and P. Rayson. OSMAN: A novel Arabic readability metric. Proceedings of LREC 2016, 250–255, 2016.

[37]     K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the association for computational linguistics, 311–318, 2002.

[38]     E. Sulem, O. Abend, and A. Rappoport. BLEU is not suitable for the evaluation of text simplification. Proceedings of EMNLP 2018, 738–744, 2018.