TY - GEN
T1 - Capability detection and evaluation metrics for cyber security lab exercises
AU - Caliskan, Emin
AU - Tatar, Unal
AU - Bahsi, Hayretdin
AU - Ottis, Rain
AU - Vaarandi, Risto
PY - 2017
Y1 - 2017
N2 - This research aims to identify metrics that can be used to evaluate the success of cyber security students, based on the logs, IDS alarms, and system events triggered during the practical lab examination in a cyber range environment. This is achieved by analyzing students of a cyber security master degree class, specially focusing on their lab performance by leveraging educational data mining techniques. After collecting related logs from monitoring systems, sanitized and cleaned data were analyzed with supervised machine learning algorithms. The results reveal there are interesting relationships and common patterns among students from different success levels. Logs and events collected from monitoring systems provide novel findings. Metrics, like the number of IDS alerts, network sessions, or top destination IP addresses, are discovered as indicators of success or failure for final grade. The research has several important contributions. First, it aims to determine the most significant evaluation metrics for capability detection of students. Identifying those metrics would make it much easier to work with in future studies. Growing number of cyber security exercises, as well as practical examinations in academia can benefit from these indicators in order to establish fair, automatically generated and verifiable results. Second, applying machine learning algorithms to the domain of cyber security education will be a more efficient way of evaluating students. Especially for the cyber security exercises with many participants, this technique can significantly reduce the manual workload of exercise organizers. Third, this research discovers common patterns that students from different skill levels possess, such as the total number of IDS alerts they generate during a practical exercise. This information can be used in several beneficial ways, including prevention of cheating and auto-grading systems. The implementation of such standardized scoring engine would be beneficial for evaluating students across different institutions and academic years in a fair way.
AB - This research aims to identify metrics that can be used to evaluate the success of cyber security students, based on the logs, IDS alarms, and system events triggered during the practical lab examination in a cyber range environment. This is achieved by analyzing students of a cyber security master degree class, specially focusing on their lab performance by leveraging educational data mining techniques. After collecting related logs from monitoring systems, sanitized and cleaned data were analyzed with supervised machine learning algorithms. The results reveal there are interesting relationships and common patterns among students from different success levels. Logs and events collected from monitoring systems provide novel findings. Metrics, like the number of IDS alerts, network sessions, or top destination IP addresses, are discovered as indicators of success or failure for final grade. The research has several important contributions. First, it aims to determine the most significant evaluation metrics for capability detection of students. Identifying those metrics would make it much easier to work with in future studies. Growing number of cyber security exercises, as well as practical examinations in academia can benefit from these indicators in order to establish fair, automatically generated and verifiable results. Second, applying machine learning algorithms to the domain of cyber security education will be a more efficient way of evaluating students. Especially for the cyber security exercises with many participants, this technique can significantly reduce the manual workload of exercise organizers. Third, this research discovers common patterns that students from different skill levels possess, such as the total number of IDS alerts they generate during a practical exercise. This information can be used in several beneficial ways, including prevention of cheating and auto-grading systems. The implementation of such standardized scoring engine would be beneficial for evaluating students across different institutions and academic years in a fair way.
KW - Capability metrics
KW - Cyber exercise
KW - Cyber security
KW - Educational data mining
KW - Student evaluation
UR - http://www.scopus.com/inward/record.url?scp=85018944652&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85018944652&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85018944652
T3 - Proceedings of the 12th International Conference on Cyber Warfare and Security, ICCWS 2017
SP - 407
EP - 414
BT - Proceedings of the 12th International Conference on Cyber Warfare and Security, ICCWS 2017
A2 - Lopez, Juan R.
A2 - Bryant, Adam R.
A2 - Mills, Robert F.
PB - Academic Conferences and Publishing International Limited
T2 - 12th International Conference on Cyber Warfare and Security, ICCWS 2017
Y2 - 2 March 2017 through 3 March 2017
ER -