Comparative Evaluation of CNN, Vision Transformer, and Hybrid Architectures with Grad-CAM Explainability for Vision-Based Malware Classification

Image-based malware classification converts malware binaries into grayscale images and classifies them with computer vision models, but the relative merits of convolutional and transformer-based architectures on this task remain unclear. This paper presents a comparative evaluation of three architectures on the Malimg dataset (9,339 images, 25 malware families): a convolutional neural network (ResNet-18), a Vision Transformer (ViT-Base/16), and a hybrid CNN-Transformer model. All three models were trained under an identical protocol with stratified 85/15 splitting, three-fold cross-validation, class-weighted loss, and ImageNet pretraining, and evaluated on a 1,401-sample hold-out test set with Grad-CAM explanations. All architectures exceeded 97% test accuracy, and no model dominated every metric: cross-validation favored the CNN in mean accuracy and stability, whereas the test split placed ViT slightly ahead on accuracy and macro F1. We frame this as an exploratory study: without significance testing, the results identify three viable designs rather than a ranking, because the ViT-Hybrid accuracy gap (0.0008, one test prediction in 1,401) lies within fold-to-fold variation (SD 0.036). All models failed completely on the low-support Autorun.K family and showed reduced precision on Yuner.A, showing that aggregate accuracy conceals family-level weaknesses. Grad-CAM heatmaps offered qualitative insight but cannot be tied to PE structure without image-to-offset mapping. The findings support both pretrained single-backbone and hybrid models for image-based malware family classification and motivate family-level evaluation and PE-aware interpretability in future work. Accepted for presentation at ICoDiT 2026 (Paper ID 218).

Recommended citation: Maulina, A., Widhiarto, E., Charisma, R. A., Tarigan, G. A., & Minor, K. A. (2026). "Comparative Evaluation of CNN, Vision Transformer, and Hybrid Architectures with Grad-CAM Explainability for Vision-Based Malware Classification." The 1st International Conference on Computing and Digital Technology (ICoDiT).