The Impact of Using ChatGPT on Enhancing Students' Programming Ability: An Experimental Study on Prompt Engineering Techniques

Muhammad Hifdzi Adini; Harja Santana Purba; Nuruddin Wiranda; Muchammad Ardhabilly; Rabiatul Adawiyah

doi:10.24036/jtip.v19i2.1088

Authors

Muhammad Hifdzi Adini Universitas Lambung Mangkurat
Harja Santana Purba Universitas Lambung Mangkurat
Nuruddin Wiranda Universitas Lambung Mangkurat
Muchammad Ardhabilly Universitas Lambung Mangkurat
Rabiatul Adawiyah Universitas Lambung Mangkurat

DOI:

https://doi.org/10.24036/jtip.v19i2.1088

Keywords:

Prompting Techniques; Programming; Education: Experiment

Abstract

The rapid development of Artificial Intelligence (AI) has created new opportunities for supporting programming learning, particularly through the use of ChatGPT and prompt engineering techniques. This study aims to examine the impact of ChatGPT-assisted learning on students’ programming ability and to compare the effectiveness of two prompting techniques, namely Text-to-Code and Code-to-Code. A quantitative approach with a pretest-posttest two-group experimental design was employed. The participants consisted of 35 Computer Education students, divided into a Text-to-Code group (n = 18) and a Code-to-Code group (n = 17). Data were collected through programming skill tests and a student perception questionnaire. The results showed that both prompting techniques improved students’ programming ability at a medium level. The Text-to-Code group obtained a mean N-Gain score of 0.424, while the Code-to-Code group achieved a higher mean N-Gain score of 0.507. The statistical test indicated a significant difference between the two groups (p = 0.035), suggesting that the Code-to-Code technique was more effective in enhancing programming ability. In addition, students in both groups showed positive perceptions toward the use of ChatGPT, with aggregate mean scores of 4.00 for Text-to-Code and 3.94 for Code-to-Code. These findings indicate that ChatGPT can support programming learning effectively, particularly when students are encouraged to write and analyze code before using AI assistance.

References

R. A. Sukmawati, M. H. Adini, M. Pramita, and A. Rizqan, “Implementasi Gamifikasi Pada Pengembangan Multimedia Pembelajaran Interaktif Dengan Metode Drill and Practice,” EDU-MAT: Jurnal Pendidikan Matematika, vol. 9, no. 2, p. 163, Nov. 2021, doi: 10.20527/edumat.v9i2.11728.

N. M. Mnguni, N. Nkomo, K. Maguraushe, and M. B. Mutanga, “An Experimental Study of The Efficacy of Prompting Strategies In Guiding ChatGPT for A Computer Programming Task,” Journal of Information Systems and Informatics, vol. 6, no. 3, pp. 1346–1359, Sep. 2024, doi: 10.51519/journalisi.v6i3.783.

T. Wang, N. Zhou, and Z. Chen, “Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation,” Jul. 2024, [Online]. Available: http://arxiv.org/abs/2407.05437

R. Yilmaz and F. G. Karaoglan Yilmaz, “Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning,” Computers in Human Behavior: Artificial Humans, vol. 1, no. 2, p. 100005, Aug. 2023, doi: 10.1016/j.chbah.2023.100005.

D. Lee and E. Palmer, “Prompt engineering in higher education: a systematic review to help inform curricula,” International Journal of Educational Technology in Higher Education, vol. 22, no. 1, Dec. 2025, doi: 10.1186/s41239-025-00503-7.

S. Ekin, “Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices,” May 04, 2023. doi: 10.36227/techrxiv.22683919.v2.

M. Kim and L. Adlof, “Adapting to the Future: ChatGPT as a Means for Supporting Constructivist Learning Environments,” TechTrends, vol. 68, no. 1, pp. 37–46, 2024, doi: 10.1007/s11528-023-00899-x.

P. P. Pawar, K. B. Salve, and R. R. Patil, “‘Impact of ChatGPT on Student’s Education: A Comprehensive analysis of positive and negative effects,’” Journal of Advanced Zoology, vol. 44, no. S8, 2023, doi: 10.53555/jaz.v44iS8.3504.

K. Aruleba, I. T. Sanusi, G. Obaido, and B. Ogbuokiri, “Integrating ChatGPT in a Computer Science Course: Students Perceptions and Suggestions,” Dec. 2023, doi: 10.48550/arXiv.2402.01640.

B. Cowan, Y. Watanobe, and A. Shirafuji, “Enhancing Programming Learning with LLMs: Prompt Engineering and Flipped Interaction,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Oct. 2023, pp. 10–16. doi: 10.1145/3634814.3634816.

R. R. Hake, “Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses,” Am. J. Phys., vol. 66, no. 1, pp. 64–74, Jan. 1998, doi: 10.1119/1.18809.

H. B. Mann and D. R. Whitney, “On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other,” The Annals of Mathematical Statistics, vol. 18, no. 1, pp. 50–60, Mar. 1947, doi: 10.1214/AOMS/1177730491.

D. Sun, A. Boudouaia, C. Zhu, and Y. Li, “Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study,” International Journal of Educational Technology in Higher Education, vol. 21, no. 1, Dec. 2024, doi: 10.1186/s41239-024-00446-5.

H. Güner and E. Er, “AI in the classroom: Exploring students’ interaction with ChatGPT in programming learning,” Education and Information Technologies 2025 30:9, vol. 30, no. 9, pp. 12681–12707, Jan. 2025, doi: 10.1007/S10639-025-13337-7.

E. Kasneci et al., “ChatGPT for good? On opportunities and challenges of large language models for education,” Learn. Individ. Differ., vol. 103, p. 102274, Apr. 2023, doi: 10.1016/J.LINDIF.2023.102274.

C. K. Lo, “What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature,” Educ. Sci. (Basel)., vol. 13, no. 4, p. 410, Apr. 2023, doi: 10.3390/EDUCSCI13040410/S1.

S. Li, J. Liu, and Q. Dong, “Generative artificial intelligence-supported programming education: Effects on learning performance, self-efficacy and processes,” Australasian Journal of Educational Technology, vol. 41, no. 3, pp. 1–25, May 2025, doi: 10.14742/AJET.9932.

T. C. Yang, Y. C. Hsu, and J. Y. Wu, “The effectiveness of ChatGPT in assisting high school students in programming learning: evidence from a quasi-experimental research,” Interactive Learning Environments, vol. 33, no. 6, pp. 3726–3743, Jul. 2025, doi: 10.1080/10494820.2025.2450659.

M. Tavakol and R. Dennick, “Making sense of Cronbach’s alpha,” Int. J. Med. Educ., vol. 2, pp. 53–55, Jun. 2011, doi: 10.5116/IJME.4DFB.8DFD/RM.

D. Stoyanova, S. Stoyanova-Petrova, S. Shotarova, S. Lyubomirov, and N. Mileva, “ChatGPT in Programming Education: An Empirical Study on Its Impact on Student Performance, Creativity, and Teamwork,” Education Sciences 2026, Vol. 16, Page 19, vol. 16, no. 1, p. 19, Dec. 2025, doi: 10.3390/EDUCSCI16010019.