AIセキュリティポータル K Program
Detecting Unsuccessful Students in Cybersecurity Exercises in Two Different Learning Environments
Share
Abstract
This full paper in the research track evaluates the usage of data logged from cybersecurity exercises in order to predict students who are potentially at risk of performing poorly. Hands-on exercises are essential for learning since they enable students to practice their skills. In cybersecurity, hands-on exercises are often complex and require knowledge of many topics. Therefore, students may miss solutions due to gaps in their knowledge and become frustrated, which impedes their learning. Targeted aid by the instructor helps, but since the instructor's time is limited, efficient ways to detect struggling students are needed. This paper develops automated tools to predict when a student is having difficulty. We formed a dataset with the actions of 313 students from two countries and two learning environments: KYPO CRP and EDURange. These data are used in machine learning algorithms to predict the success of students in exercises deployed in these environments. After extracting features from the data, we trained and cross-validated eight classifiers for predicting the exercise outcome and evaluated their predictive power. The contribution of this paper is comparing two approaches to feature engineering, modeling, and classification performance on data from two learning environments. Using the features from either learning environment, we were able to detect and distinguish between successful and struggling students. A decision tree classifier achieved the highest balanced accuracy and sensitivity with data from both learning environments. The results show that activity data from cybersecurity exercises are suitable for predicting student success. In a potential application, such models can aid instructors in detecting struggling students and providing targeted help. We publish data and code for building these models so that others can adopt or adapt them.
Predicting student success in cybersecurity exercises with a support vector classifier
Q. Vinlove, J. Mache, R. Weiss
Published: 2020
Learning Obstacles in the Capture The Flag Model
K. Chung, J. Cohen
Published: 2014
Methodological considerations for predicting at-risk students
C. Koutcheme, S. Sarsa, A. Hellas, L. Haaranen, J. Leinonen
Published: 2022
Why we need open data in computer science education research
N. Kiesler, D. Schiffner
Published: 2023
Predicting academic performance: A systematic literature review
A. Hellas
Published: 2018
CS1: how will they do? How can we help? A decade of research and practice
K. Quille, S. Bergin
Published: 2019
The history of computing education research
M. Guzdial, B. du Boulay
Published: 2019
Process mining analysis of puzzle-based cybersecurity training
M. Macak, R. Oslejsek, B. Buhnova
Published: 2022
Exploring How Students Solve Open-Ended Assignments: A Study of SQL Injection Attempts in a Cybersecurity Course
C. Koutcheme, A. Tilanterä, A. Peltonen, A. Hellas, L. Haaranen
Published: 2022
Personalized Learning in a Virtual Hands-on Lab Platform for Computer Science Education
Y. Deng, D. Lu, C.-J. Chung, D. Huang, Z. Zeng
Published: 2018
Factors impacting performance in competitive cyber exercises
A. R. Silva, J. T. McClain, B. R. Anderson, K. S. Nauer, R. Abbott, J. C. Forsythe
Published: 2014
Early identification of student struggles at the topic level using context-agnostic features
K. Arakawa, Q. Hao, W. Deneke, I. Cowan, S. Wolfman, A. Peterson
Published: 2022
Keep It Relevant! Using In-Class Exercises to Predict Weekly Performance in CS1
E. Hicks, A. Cook, K. Malasri, A. Zaman, V. Phan
Published: 2022
PreSS: Predicting Student Success Early in CS1. A Pilot International Replication and Generalization Study
K. Quille, S. Nam Liao, E. Costelloe, K. Nolan, A. Mooney, K. Shah
Published: 2022
Early performance prediction using interpretable patterns in programming process data
G. Gao, S. Marwan, T. W. Price
Published: 2021
Time-on-task metrics for predicting performance
J. Leinonen, F. E. V. Castro, A. Hellas
Published: 2022
Ultra-Lightweight Early Prediction of At-Risk Students in CS1
C. Gordon, S. Zhao, F. Vahid
Published: 2023
A study of keystroke data in two contexts: Written language and programming language influence predictability of learning outcomes
J. Edwards, J. Leinonen, A. Hellas
Published: 2020
Predictive models of academic success: A case study with version control systems
A. M. Guerrero-Higueras, N. DeCastro-García, V. Matellán, M. A. Conde
Published: 2018
A robust machine learning technique to predict low-performing students
S. N. Liao, D. Zingaro, K. Thai, C. Alvarado, W. G. Griswold, L. Porter
Published: 2019
Teaching cybersecurity analysis skills in the cloud
R. Weiss, S. Boesen, J. F. Sullivan, M. E. Locasto, J. Mache, E. Nilsen
Published: 2015
Does time-on-task estimation matter? implications on validity of learning analytics findings
V. Kovanovic, D. Gaševic, S. Dawson, S. Joksimovic, R. Baker
Published: 2016
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead
C. Rudin
Published: 2019
Scikit-learn: Machine learning in Python
F. Pedregosa
Published: 2011
Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data
P. Thölke
Published: 2023
Facing imbalanced data–recommendations for the use of performance metrics
L. A. Jeni, J. F. Cohn, F. De La Torre
Published: 2013
Evaluating neural networks as a method for identifying students in need of assistance
K. Castro-Wunsch, A. Ahadi, A. Petersen
Published: 2017
Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies
P. Ihantola
Published: 2015
Using demographic data as predictor variables: a questionable choice
R. S. Baker, L. Esbenshade, J. Vitale, S. Karumbaiah
Published: 2023
Share