AIセキュリティポータル K Program
Evaluation of the Programming Skills of Large Language Models
Share
Abstract
The advent of Large Language Models (LLM) has revolutionized the efficiency and speed with which tasks are completed, marking a significant leap in productivity through technological innovation. As these chatbots tackle increasingly complex tasks, the challenge of assessing the quality of their outputs has become paramount. This paper critically examines the output quality of two leading LLMs, OpenAI's ChatGPT and Google's Gemini AI, by comparing the quality of programming code generated in both their free versions. Through the lens of a real-world example coupled with a systematic dataset, we investigate the code quality produced by these LLMs. Given their notable proficiency in code generation, this aspect of chatbot capability presents a particularly compelling area for analysis. Furthermore, the complexity of programming code often escalates to levels where its verification becomes a formidable task, underscoring the importance of our study. This research aims to shed light on the efficacy and reliability of LLMs in generating high-quality programming code, an endeavor that has significant implications for the field of software development and beyond.
Agile management in cybersecurity
P.M. Asprion, C. Giovanoli, C. Scherb, S. Bhat
Published: 2023
“so what if chatgpt wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy
Y.K. Dwivedi, N. Kshetri, L. Hughes, E.L. Slade, A. Jeyaraj, A.K. Kar, A.M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja
Published: 2023
What’s ahead for bard: More global, more visual, more integrated
S. Hsiao
Published: 2023
Code quality issues in student programs
H. Keuning, B. Heeren, J. Jeuring
Published: 2017
Revolutionizing radiology with gpt-based models: current applications, future possibilities and limitations of chatgpt
A. Lecler, L. Duron, P. Soyer
Published: 2023
Source code metrics: A systematic mapping study
A.S. Nuñez-Varela, H.G. Pérez-Gonzalez, F.E. Martínez-Perez, C. Soubervielle-Montalvo
Published: 2017
Asleep at the keyboard? assessing the security of github copilot’s code contributions
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri
Published: 2022
A systematic review on the code smell effect
J.A.M. Santos, J.B. Rocha-Junior, L.C.L. Prates, R.S. Do Nascimento, M.F. Freitas, M.G. De Mendonça
Published: 2018
Cymed: A framework for testing cybersecurity of connected medical devices
C. Scherb, A. Hadayah, L.B. Heitz
Published: 2024
Divide, conquer and verify: Improving symbolic execution performance
C. Scherb, L.B. Heitz, H. Grieder, O. Mattmann
Published: 2023
A cyber attack simulation for teaching cybersecurity
C. Scherb, L.B. Heitz, F. Grimberg, H. Grieder, M. Maurer
Published: 2023
Using large language models to generate junit tests: An empirical study
M.L. Siddiq, J.C.S. Santos, R.H. Tanvir, N. Ulfat, F.A. Rifat, V.C. Lopes
Published: 2024
Bug characteristics in open source software
L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, C. Zhai
Published: 2014
Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x
Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue
Published: 2023
Share