AI-Based Malware Detection Using Large Language Models and Machine Learning

Abstract:

The volume and diversity of new malware continues to outpace what classical signature-based defenses can address: signature matching achieves a very low false-alarm rate but is structurally blind to zero-day malware, polymorphic variants, and repacked samples. Machine learning and deep learning approaches generalize better from learned statistical and behavioral patterns, yet classical models on static features can be evaded by packing, while deep sequence models and large language models applied to API-call or disassembly sequences are more expensive and can themselves be targeted by adversarial perturbation. This paper proposes a hybrid malware-detection pipeline that fuses classical machine learning over static and dynamic features with large-language-model-based semantic analysis of API-call and disassembly sequences through a calibrated ensemble, followed by an explicit adversarial and packing robustness check before deployment. Drawing on established malware-detection and adversarial-robustness literature, the study defines seven evaluation criteria — Detection Accuracy, Precision, Malware Detection Rate, F1 Score, False Alarm Rate, AUC-ROC, and a combined Malware Detection Reliability Score — and applies them to an illustrative case study of an enterprise endpoint-security deployment. A five-configuration ablation study, spanning a signature/heuristic baseline through a deliberately over-engineered ensemble, is used to identify the point of diminishing returns, and the recommended configuration is benchmarked directly against the signature-based baseline, including accuracy on zero-day and previously unseen malware families. The illustrative results indicate substantially higher accuracy and zero-day generalization for the proposed hybrid pipeline relative to the baseline, at a false-alarm rate that remains low in absolute terms. The findings position classical-plus-LLM ensemble detection, evaluated against explicit adversarial criteria, as a foundational deployment practice for organizations operationalizing AI-based malware defense at enterprise scale.