Advanced Implementation of a Multilevel Model for Text Summarization in Kazakh Using Pretrained Models

Authors

  • Dina Oralbekova Institute of Information and Computational Technologies, Almaty, Kazakhstan | International Engineering and Technological University, Almaty, Kazakhstan
  • Orken Mamyrbayev Institute of Information and Computational Technologies, Almaty, Kazakhstan
  • Mohamed Othman Malaysia Department of Communication Technology and Networks, Universiti Putra Malaysia, Serdang, Malaysia | Laboratory of Computational Science and Mathematical Physics, Institute for Mathematical Research, Universiti Putra Malaysia, Serdang, Malaysia
  • Sholpan Zhumagulova Al-Farabi Kazakh National University, Almaty, Kazakhstan
Volume: 15 | Issue: 5 | Pages: 26711-26721 | October 2025 | https://doi.org/10.48084/etasr.12799

Received: 17 June 2025 | Revised: 11 July 2025 and 17 July 2025 | Accepted: 20 July 2025 | Online: 4 August 2025


Corresponding author: Dina Oralbekova

Abstract

This study investigates transformer models for the task of hybrid text summarization in the Kazakh language. Using mBART, mT5, and XLM-RoBERTa models, a multilevel architecture was developed that processes text at the character, subword, word, and contextual levels. The proposed system performs feature fusion across multiple linguistic layers, enabling the model to capture both fine-grained lexical variation and broader contextual dependencies. The architecture also allows flexible integration with various transformer models, supporting both encoder-decoder and hybrid configurations. This approach significantly improved the quality of generated summaries by effectively accounting for the morphological and semantic features of the Kazakh language. The experimental results showed that mBART achieved the best performance in terms of ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore-F1 metrics, confirming the high effectiveness of the proposed multilevel transformer architecture. This is the first implementation of such an architecture for hybrid summarization in Kazakh, which is a low-resource and morphologically rich language.

Keywords:

multilevel modeling, Kazakh language, hybrid summarization, transformer models, mBART, mT5, XLM-RoBERTa

Downloads

Download data is not yet available.

References

A. Vaswani et al., "Attention is All you Need," in Advances in Neural Information Processing Systems, 2017, vol. 30.

A. Rahali and M. A. Akhloufi, "End-to-End Transformer-Based Models in Textual-Based NLP," AI, vol. 4, no. 1, pp. 54–110, Jan. 2023.

K. Rani Narejo, H. Zan, D. Oralbekova, K. Parkash Dharmani, M. Orken, and K. Mukhsina, "Enhancing Emoji-Based Sentiment Classification in Urdu Tweets: Fusion Strategies With Multilingual BERT and Emoji Embeddings," IEEE Access, vol. 12, pp. 126587–126600, 2024.

D. Oralbekova, O. Mamyrbayev, S. Zhumagulova, and N. Zhumazhan, "A Comparative Analysis of LSTM and BERT Models for Named Entity Recognition in Kazakh Language: A Multi-classification Approach," in Modeling and Simulation of Social-Behavioral Phenomena in Creative Societies, 2024, pp. 116–128.

N. Giarelis, C. Mastrokostas, and N. Karacapilidis, "Abstractive vs. Extractive Summarization: An Experimental Review," Applied Sciences, vol. 13, no. 13, Jan. 2023, Art. no. 7620.

K. S. Kalyan, A. Rajasekharan, and S. Sangeetha, "AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing." arXiv, Aug. 28, 2021.

X. Liu et al., "GPT understands, too," AI Open, vol. 5, pp. 208–215, Jan. 2024.

C. Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.

M. Lewis et al., "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension." arXiv, Oct. 29, 2019.

M. Kirmani, N. M. Hakak, M. Mohd, and M. Mohd, "Hybrid Text Summarization: A Survey," in Soft Computing: Theories and Applications, 2019, pp. 63–73.

L. Qin et al., "A survey of multilingual large language models," Patterns, vol. 6, no. 1, Jan. 2025.

A. Conneau et al., "Unsupervised Cross-lingual Representation Learning at Scale." arXiv, Apr. 08, 2020.

Y. Liu et al., "RoBERTa: A Robustly Optimized BERT Pretraining Approach." arXiv, Jul. 26, 2019.

A. Zulkhazhav et al., "Kazakh Text Summarization using Fuzzy Logic," Computación y Sistemas, vol. 23, no. 3, pp. 851–859, Sep. 2019.

B. Kynabay, A. Aldabergen, and A. Zhamanov, "Automatic Summarizing the News from Inform.kz by Using Natural Language Processing Tools," in 2021 IEEE International Conference on Smart Information Systems and Technologies (SIST), Nur-Sultan, Kazakhstan, Apr. 2021, pp. 1–4.

T. Zhabayev and U. Tukeyev, "Development of Technology for Summarization of Kazakh Text," International Journal of Advanced Computer Science and Applications, vol. 12, no. 9, 2021.

E. Winarko, L. Tanoto, and M. H. Reza, "Indonesian Abstractive Text Summarization Using Stacked Embeddings and Transformer Decoder.," IAENG International Journal of Computer Science, vol. 52, no. 4, 2025.

A. Raza, M. H. Soomro, Salahuddin, I. Shahzad, and S. Batool, "Abstractive Text Summarization for Urdu Language," Journal of Computing & Biomedical Informatics, vol. 7, no. 02, Sep. 2024.

B. C. Challagundla and C. Peddavenkatagari, "Neural Sequence-to-Sequence Modeling with Attention by Leveraging Deep Learning Architectures for Enhanced Contextual Understanding in Abstractive Text Summarization." arXiv, Apr. 08, 2024.

N. Jayatilleke and R. Weerasinghe, "A Hybrid Architecture with Efficient Fine Tuning for Abstractive Patent Document Summarization," in 2025 International Research Conference on Smart Computing and Systems Engineering (SCSE), Colombo, Sri Lanka, Apr. 2025, pp. 1–6.

D. H. Dat, D. D. Anh, A. T. Luu, and W. Buntine, "Discrete Diffusion Language Model for Efficient Text Summarization." arXiv, Mar. 10, 2025.

Y. R. Gogireddy, A. N. Bandaru, and V. Sumanth, "Synergy of Graph-Based Sentence Selection and Transformer Fusion Techniques For Enhanced Text Summarization Performance," Journal of Computer Engineering and Technology, vol. 7, no. 1, pp. 33–41, Jun. 2024.

B. Faizal, S. Abraham, and S. Thomas, "Automated Business Report Summarization Using Transformer Model," in 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, Mar. 2024, pp. 254–258.

R. Chakraborti, R. Banerjee, and S. Das, "Evaluating the Efficacy of Text Summarization Models: A Comparison of NLP Algorithms," in 2025 8th International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech), Kolkata, India, Jan. 2025, pp. 1–5.

A. B. Rao, S. G. Aithal, and S. Singh, "An Evaluation Metric for Assessing Summary-Level Semantic Similarity in Abstractive Text Summarization," in 2025 International Conference on Artificial Intelligence and Data Engineering (AIDE), Nitte, India, Feb. 2025, pp. 602–607.

S. Zangooei, A. Darmani, H. F. Nezhad, and L. Mahmoudi, "ARLED: Leveraging LED-Based ARMAN Model for Abstractive Summarization of Persian Long Documents," in 2025 11th International Conference on Web Research (ICWR), Tehran, Iran, Apr. 2025, pp. 25–32.

O. Langston and B. Ashford, "Automated Summarization of Multiple Document Abstracts and Contents Using Large Language Models." TechRxiv.

B. Barta, D. Lakatos, A. Nagy, M. K. Nyist, and J. Ács, "From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization," in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 7503–7509.

S. Jebbara and P. Cimiano, "Improving Opinion-Target Extraction with Character-Level Word Embeddings," in Proceedings of the First Workshop on Subword and Character Level Models in NLP, 2017, pp. 159–167.

T. Kudo and J. Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018, pp. 66–71.

R. A. Stein, P. A. Jaques, and J. F. Valiati, "An analysis of hierarchical text classification using word embeddings," Information Sciences, vol. 471, pp. 216–232, Jan. 2019.

G. Aguilar, B. McCann, T. Niu, N. Rajani, N. S. Keskar, and T. Solorio, "Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality," in Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 1640–1651.

S. R. Basha, J. K. Rani, and J. J. C. P. Yadav, "A Novel Summarization-based Approach for Feature Reduction Enhancing Text Classification Accuracy," Engineering, Technology & Applied Science Research, vol. 9, no. 6, pp. 5001–5005, Dec. 2019.

Downloads

How to Cite

[1]
D. Oralbekova, O. Mamyrbayev, M. Othman, and S. Zhumagulova, “Advanced Implementation of a Multilevel Model for Text Summarization in Kazakh Using Pretrained Models”, Eng. Technol. Appl. Sci. Res., vol. 15, no. 5, pp. 26711–26721, Oct. 2025.

Metrics

Abstract Views: 86
PDF Downloads: 48

Metrics Information

License

Copyright (c) 2025 Dina Oralbekova, Orken Mamyrbayev, Mohamed Othman, Sholpan Zhumagulova

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.