Keywords
translation evaluation
ChatGPT
Neural Machine Translation
evaluation criteria
How to Cite
Abstract
Translating legal texts—especially between linguistically and culturally distinct languages such as Arabic and English—poses significant challenges within Translation Studies. This study evaluates the performance of three machine translation tools: ChatGPT (OpenAI GPT-3.5, March 2024), Google Translate (web version, April 2024), and DeepL (free web version, April 2024), focusing on a corpus of Algerian Supreme Court judgments. Drawing on key theoretical frameworks including functionalism, equivalence and legal semiotics, we assess translation quality across four criteria: accuracy, clarity, fluency, and cultural appropriateness. Detailed documentation of tool versions, prompting strategies, and input constraints ensures transparency and supports reproducibility. Our findings indicate that while all tools exhibit limitations inherent to the complexity of legal language, ChatGPT shows particular strength in handling cultural nuances and producing more contextually appropriate translations. Nonetheless, human expertise remains vital for ensuring reliability in professional legal translation. Though the small dataset limits the generalizability of results, this study offers important insights for Translation Studies scholarship and lays groundwork for further research into the evolving role of AI in legal translation.
References
Al-Bahūtī, M. B. Y. (1983). Kashshāf al-qināʽʽan matn al-iqnāʽ (Vol. 2). Dār Al-ʽIlm li-l-Malayīn.
Al-Faḍīlāt, J. M. (n.d.). Binā’ al-usrah al-muslimah ʽalà ḍawʾ al-fiqh wa-l-qānūn. Dār Al-Shihāb.
Al-Jurjānī, A. B. M. A.-S. (1986). Al-taʽrīfāt. Dār Al-Shuʽūn Al-Taqāfiyyah Al-ʽĀmmah.
Al-Kāsānī, A. A. B. M. (1986). Badāʾiʽ al-ṣanāʾiʽ fī tartīb al-sharāʾiʽ (Vol. 2). Dār Al-Kutub Al-ʽIlmiyyah.
Al-Zarkashī, B. A. M. (1993). Sharḥ Mukhtaṣar al-Kharqī (Vol. 4). Maktabat al-ʽUbīkayn.
Baker, M. (1992). In other words: A coursebook on translation. Routledge. DOI: https://doi.org/10.4324/9780203327579
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv. DOI: https://doi.org/10.18653/v1/2023.ijcnlp-main.45
Belhadj, L. (2005). Al-wajīz fī sharḥ qānūn al-usrah al-jazāʾirī (Part 1, 4th ed.). Dīwān Al-Maṭbūʽāt Al-Jāmiʽiyyah.
Benounane, H. M., & Naceur, D. (2020). On the current state of machine translation: Investigating the statistical approach limitations and the neural model implications. Academic Review of Social and Human Studies, 12(2), 38–47. DOI: https://doi.org/10.33858/0500-012-002-064
Biel, Ł. (2014). Lost in the Eurofog: The textual fit of translated law. In Translation and the law (pp. 57–76). Routledge. DOI: https://doi.org/10.3726/978-3-653-03986-3
Boitet, C., Blanchon, H., Seligman, M., & Bellynck, V. (2009). Evolution of MT with the web. In Proceedings of the Conference on Machine Translation (Vol. 25, pp. 1–13). DOI: https://doi.org/10.1109/NLPKE.2010.5587865
Canavese, P. (2023). Translation and clarity in Swiss multilingual legislation: A qualitative study on textuality. Translation & Interpreting, 15(2), 1–20. DOI: https://doi.org/10.12807/ti.115202.2023.a05
Cao, D. (2007). Translating law. Multilingual Matters. DOI: https://doi.org/10.21832/9781853599552
Cheng, Y., & Cheng, Y. (2019). Semi-supervised learning for neural machine translation. In Joint training for neural machine translation: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1965–1974). Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/P16-1185
Colson, J. (1993). Les instruments de la docilité. In Le traducteur et ses instruments. Presses de la Sorbonne Nouvelle. DOI: https://doi.org/10.4000/palimpsestes.734
Cornu, G. (2005). Linguistique juridique. Éditions Montchrestien.
Cotelli Kureth, S., Delorme Benites, A., Haller, M., Noghrechi, H., & Steele, E. (2023). “I looked it up in DeepL”: Machine translation and digital tools in the language classroom. In Human translation and natural language processing: Towards a new consensus? (pp. 81–96). Fondazione Università Ca’ Foscari. DOI: https://doi.org/10.30687/978-88-6969-762-3/006
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., & Sui, Z. (2023). A survey on in-context learning. arXiv. DOI: https://doi.org/10.18653/v1/2024.emnlp-main.64
Durieux, C. (1992). La terminologie en traduction technique: apports et limites. Terminologie et Traduction, 2(3).
Gao, Y., Wang, R., & Hou, F. (2023). How to design translation prompts for ChatGPT: An empirical study. arXiv. DOI: https://doi.org/10.1145/3700410.3702123
Garzone, G. (2000). Legal translation and functionalist approaches: A contradiction in terms? In Translation studies: A reader (pp. 127–136). St. Jerome Publishing.
Haleem, A., Javaid, M., & Singh, R. P. (2022). An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2(4), 100089. DOI: https://doi.org/10.1016/j.tbench.2023.100089
Hidalgo-Ternero, C. (2020). Google Translate vs. DeepL: Analyzing neural machine translation performance under the challenge of phraseological variation. MonTI. Monografías de Traducción e Interpretación, Special Issue 6, 154–177. DOI: https://doi.org/10.6035/MonTI.2020.ne6.5
House, J. (2015). Translation quality assessment: Past and present. Routledge. DOI: https://doi.org/10.4324/9781315752839
Ibn Manẓūr, M. (1999). Lisān al-ʽArab (Vol. 2, 3rd ed.). Dār Ihyāʾ al-Turāth al-ʽArabī & Muʾassasat al-Taʾrīkh al-ʽArabī.
Ibn Qudāmah, A. A. (1984). Al-Mughnī (Vol. 7). Dār Al-Fikr.
James, G. (2020). Introduction to Google Translate. Gilad James Mystery School.
Kamaluddin, M. I., Rasyid, M. W. K., Abqoriyyah, F., & Saehu, A. (2024). Accuracy analysis of DeepL: Breakthroughs in machine translation technology. Journal of English Education Forum (JEEF), 4(2), 122–126. DOI: https://doi.org/10.29303/jeef.v4i2.681
Khoshafah, F. (2023). ChatGPT for Arabic-English translation: Evaluating the accuracy. Research Square. DOI: https://doi.org/10.21203/rs.3.rs-2814154/v2
Liu, L., Utiyama, M., Finch, A., & Sumita, E. (2016). Neural machine translation with supervised attention. arXiv preprint arXiv:1609.04186.
Mohanty, N. (2005). The limits of creativity and the translator’s responsibility. Translation Today, 2(2), 129–148. DOI: https://doi.org/10.46623/tt/2005.2.2.ar10
Moneus, A. M., & Sahari, Y. M. (2024). Artificial intelligence and human translation: A contrastive study based on legal texts. Heliyon, 10(6), e28106. DOI: https://doi.org/10.1016/j.heliyon.2024.e28106
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. DOI: https://doi.org/10.3115/1073083.1073135
Roser Nebot, N. (2019). El papel de la exégesis en la traducción del Corán. REIDOCREA, 8, 231–249. DOI: https://doi.org/10.30827/Digibug.57764
Sacco, R. (1999). Langue et droit. In R. Sacco & L. Castellani (Eds.), Les multiples langues du droit européen uniforme (pp. 126–143). L’Harmattan.
Šarčević, S. (1997). New approach to legal translation. Kluwer Law International.
Snell-Hornby, M. (1988). Translation studies: An integrated approach. John Benjamins. DOI: https://doi.org/10.1075/z.38
Stiti, A., & Benaouda, A. (2023). Translating Algerian laws using neural machine translation. Cahiers de Traduction, 28(1), 421–445.
Turovsky, B. (2016, November 15). Found in translation: More accurate, fluent sentences in Google Translate. Google Blog.
Venuti, L. (1995). The translator’s invisibility: A history of translation. Routledge.
Williams, J., & Chesterman, A. (2002). The map: A beginner’s guide to doing research in translation studies. St. Jerome Publishing.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., & Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Xie, Y., Yu, C., Zhu, T., Bai, J., Gong, Z., & Soh, H. (2023). Translating natural language to planning goals with large-language models. arXiv.
Zhao, Y., Zhang, M., Chen, X., Deng, Y., Geng, A., Liu, L., Liu, X., Li, W., Jiang, Y., Yang, H., Han, Y., Tao, S., Xie, N., Li, X., Ma, M., & Zhang, Z. (2023). Human evaluation for translation quality of ChatGPT: A preliminary study. In Proceedings of the International Conference HiT-IT 2023 (pp. 282–287). DOI: https://doi.org/10.26615/issn.2683-0078.2023_023
License
Copyright (c) 2026 Merabet Mohamed Hamza, Roser Nebot Nicolas
This work is licensed under a Creative Commons Attribution 4.0 International License.
When submitting a paper the author agrees to the following publishing agreement and processing personal data.
PUBLICATION AGREEMENT, COPYRIGHT LICENSE, PERSONAL DATA PROCESSING CONSENT
This is a publication agreement and copyright license (“Agreement”) regarding a written manuscript currently submitted via Pressto platform
(“Article”) to be published in Comparative Legilinguistics International Journal for Legal Communication (“Journal”).
The parties to this Agreement are:
the Author or Authors of the submitted article (individually, or if more than one author, collectively, “Author”) and Comparative Legilinguistics International Journal for Legal Communication (“Publisher”), address al. Niepodległości 4, 61-874 Poznań, represented by its editor in chief Aleksandra Matulewska.
§1. LICENSE OF COPYRIGHT
a) The Author and the Publisher agree that the Author grants a Creative Commons Attribution 4.0 International License, which is incorporated herein by reference and is further specified at Creative Commons — Attribution 4.0 International — CC BY 4.0 copyright license in the Article to the general public.
b) The Author grants to the Publisher a royalty-free, worldwide nonexclusive license to publish, reproduce, display, distribute, translate and use the Article in any form, either separately or as part of a collective work, including but not limited to a nonexclusive license to publish the Article in an issue of the Journal, copy and distribute individual reprints of the Article, authorize reproduction of the entire Article in another publication, and authorize reproduction and distribution of the Article or an abstract thereof by means of computerized retrieval systems (such as Westlaw, Lexis and SSRN). The Author retains ownership of all rights under copyright in the Article, and all rights not expressly granted in this Agreement.
c) The Author grants to the Publisher the power to assign, sublicense or otherwise transfer any and all licenses expressly granted to the Publisher under this Agreement.
d) Republication. The Author agrees to require that the Publisher be given credit as the original publisher in any republication of the Article authorized by the Author. If the Publisher authorizes any other party to republish the Article under the terms of paragraphs 1c and 1 of this Agreement, the Publisher shall require such party to ensure that the Author is credited as the Author.
§2. EDITING OF THE ARTICLE
a) The Author agrees that the Publisher may edit the Article as suitable for publication in the Journal. To the extent that the Publisher’s edits amount to copyrightable works of authorship, the Publisher hereby assigns all right, title, and interest in such edits to the Author.
§3. WARRANTIES
a) The Author represents and warrants that to the best of the Author’s knowledge the Article does not defame any person, does not invade the privacy of any person, and does not in any other manner infringe upon the rights of any person. The Author agrees to indemnify and hold harmless the Publisher against all such claims.
b) The Author represents and warrants that the Author has full power and authority to enter into this Agreement and to grant the licenses granted in this Agreement.
c) The Author represents and warrants that the Article furnished to the Publisher has not been published previously. For purposes of this paragraph, making a copy of the Article accessible over the Internet, including, but not limited to, posting the Article to a database accessible over the Internet, does not constitute prior publication so long as the as such copy indicates that the Article is not in final form, such as by designating such copy to be a “draft,” a “working paper,” or “work-in-progress”. The Author agrees to hold harmless the Publisher, its licensees and distributees, from any claim, action, or proceeding alleging facts that constitute a breach of any warranty enumerated in this paragraph.
§4. TERM
a) The agreement was concluded for an unspecified time.
§5. PAYMENT
a) The Author agrees and acknowledges that the Author will receive no payment from the Publisher for use of the Article or the licenses granted in this Agreement.
b) The Publisher agrees and acknowledges that the Publisher will not receive any payment from the Author for publication by the Publisher.
§6. ENTIRE AGREEMENT
a) This Agreement supersedes any and all other agreements, either oral or in writing, between the Author and the Publisher with respect to the subject of this Agreement. This Agreement contains all of the warranties and agreements between the parties with respect to the Article, and each party acknowledges that no representations, inducements, promises, or agreements have been made by or on behalf of any party except those warranties and agreements embodied in this Agreement.
b) In all cases not regulated by this Agreement, legal provisions of Polish Copyright Act and Polish Civil Code shall apply.
c) Any disputes arising from the enforcement of obligations connected with this Agreement shall be resolved by a court competent for the headquarters of the Publisher.
d) Any amendments or additions to the Agreement must be made in writing and signed by authorised representative of both parties, otherwise being ineffective.
e) This Agreement is signed electronically and the submission of the article via the PRESSto platform is considered as the conclusion of the Agreement by the Author and the Publisher.
f) Clause for consent to the processing of personal data - general
g) The Author shall give his or her consent to the processing of their personal data in accordance with the Act of 10 May 2018 on the protection of personal data and Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of persons physical in connection with the processing of personal data and on the free movement of such data, and repealing Directive 95/46 / EC (General Data Protection Regulation) for the purpose and in connection with making publications available on the PRESSto scientific journals platform and DeGruyter platform, guaranteeing the security of services rendered, and improving them.
I HAVE READ AND AGREE FULLY WITH THE TERMS OF THIS AGREEMENT.
The Author The Publisher