Information and communication and chemical technologies

Vol. 3 No. 28 (2025) / 2025-09-30 / Number of views: 338

COMPARATIVE ANALYSIS OF EMBEDDING MODELS FOR MATCHING QUESTIONS AND CONTEXTS IN THE KAZAKH LANGUAGE

Authors

Astana Polytechnic College
https://orcid.org/0009-0008-2046-591X
Astana IT University
https://orcid.org/0000-0001-9560-9756
Astana Polytechnic College
https://orcid.org/0009-0001-5346-6184
L.N.Gumilyov ENU
https://orcid.org/0000-0001-8555-7315
Astana Polytechnic College
https://orcid.org/0009-0008-9864-5224
Astana Polytechnic College
https://orcid.org/0009-0009-2690-276X

Keywords

semantic search, sentence embeddings, question–context matching, multilingual transformer models, TF-IDF, Kazakh language, information retrieval

Link to DOI:

https://doi.org/10.58805/kazutb.v.3.28-987

How to quote

Mazhitova, K., et al. “COMPARATIVE ANALYSIS OF EMBEDDING MODELS FOR MATCHING QUESTIONS AND CONTEXTS IN THE KAZAKH LANGUAGE”. Vestnik KazUTB, vol. 3, no. 28, Sept. 2025, doi:10.58805/kazutb.v.3.28-987.

Abstract

This paper presents a comparative analysis of the effectiveness of modern embedding models for matching questions and contexts in the Kazakh language, within the framework of semantic search and question-answering (QA) systems. The study is based on the open dataset Kundyzka/informatics_kaz, which contains approximately 7,700 question–context pairs covering key concepts in computer science. The analysis focuses on comparing the traditional TF-IDF method with state-of-the-art multilingual models, including mMiniLM, LaBSE, Alibaba GTE-multilingual-base, intfloat/multilingual-e5-small, and Snowflake Arctic-Embed v2.0. The evaluation metrics used are Accuracy@1, Mean Reciprocal Rank (MRR), and ROC AUC. The zero-shot experiments demonstrate that embedding-based models significantly outperform TF-IDF, achieving Accuracy@1 in the range of 0.59–0.63, whereas the baseline method yields 0.50. Moreover, visualization results using t-SNE indicate that modern models are able to cluster semantically related questions and contexts more closely, which positively affects the quality of ranking. Among the models considered, Alibaba GTE and Snowflake Arctic-Embed v2.0 achieved the best results, confirming their relevance for developing automatic question-answering systems in educational AI applications. This study highlights that even without fine-tuning on Kazakh data, pretrained multilingual models can substantially enhance semantic matching quality, offering promising prospects for their use in educational systems and other information services.