Paper Information
- Author
- Mohammad Reza Mirbagheri,Mohammad Mahdi Mirkamali,Zahra Motoshaker Arani,Ali Javeri,Amir Mahdi Sadeghzadeh,Rasool Jalili
- Published
- 9-9-2025
- Affiliation
- Department of Computer Engineering, Sharif University of Technology
- Country
- Iran
- Conference
- Computing Research Repository (CoRR)
Abstract
Large Language Models (LLMs), trained on extensive datasets using advanced
deep learning architectures, have demonstrated remarkable performance across a
wide range of language tasks, becoming a cornerstone of modern AI technologies.
However, ensuring their trustworthiness remains a critical challenge, as
reliability is essential not only for accurate performance but also for
upholding ethical, cultural, and social values. Careful alignment of training
data and culturally grounded evaluation criteria are vital for developing
responsible AI systems. In this study, we introduce the EPT (Evaluation of
Persian Trustworthiness) metric, a culturally informed benchmark specifically
designed to assess the trustworthiness of LLMs across six key aspects:
truthfulness, safety, fairness, robustness, privacy, and ethical alignment. We
curated a labeled dataset and evaluated the performance of several leading
models - including ChatGPT, Claude, DeepSeek, Gemini, Grok, LLaMA, Mistral, and
Qwen - using both automated LLM-based and human assessments. Our results reveal
significant deficiencies in the safety dimension, underscoring the urgent need
for focused attention on this critical aspect of model behavior. Furthermore,
our findings offer valuable insights into the alignment of these models with
Persian ethical-cultural values and highlight critical gaps and opportunities
for advancing trustworthy and culturally responsible AI. The dataset is
publicly available at: https://github.com/Rezamirbagheri110/EPT-Benchmark.