AI and Machine Learning Advances

There is a high rate of cyber threats development and the use of the zero-day attack, which presents a great challenge to the conventional intrusion detection system (IDS). This paper compares the performance and the generalization ability of the AI-based IDS models on two benchmark datasets (NSL-KDD (legacy) and CIC-IDS2017 (modern)) datasets. We present comparisons of Random Forest, XGBoost, and Support Vector Machine in supervised and zero-day simulation scenarios in a leave-one-attack-out set up. Supervised performance has been observed to be almost perfect on both datasets, with recall and ROC-AUC scores being close to 0.999 with the tree-based models. Nevertheless, zero-day analysis demonstrates significant performance reduction, and a drop to about 68 and 58 percent on NSL-KDD and CIC-IDS2017, respectively. These results demonstrate that there is a severe disparity between controlled precision and actual generalization in the real world. The findings show that AI-based IDS models are effective in detecting known attacks but have poor zero-day resiliency, and it is important to note that more generalized and adaptable intrusion detection systems should be designed.