scienceNUS SCALE · Deep Learning capstone

Auditing a Weapons Image Classifier

A 9-class weapons image classifier, audited end to end: a dataset that leaks ~31% duplicates, the accuracy inflation that leakage hides, and two opposite frontier architectures that finish in a statistical tie on genuinely unseen images.

9 classes · 714 images97.2% original · 94.9% clean (best model)Orange + PyTorchAttention vs. convolution

This is the applied finale of the guide: the capstone project from a four-day NUS SCALE course on AI and Machine Learning (Deep Learning). The brief was simple — classify nine types of weapon from images using embeddings in Orange. We did that, then went further. We trained two opposite frontier architectures (an attention-based DINOv3 and a convolutional ConvNeXt-V2), audited the dataset, found it leaks duplicate images between train and test, rebuilt a leakage-proof split, and re-measured every metric on both splits.

The headline numbers are reported two ways throughout this study. The strongest model reaches 97.2% accuracy on the original dataset and 94.9% on the genuinely-unseen clean split, where the two architectures finish in a statistical tie. The original-split number is the one higher, because the leak lets the model recognize images it has effectively already seen — which is exactly the point this study makes visible.

The project received a Letter of Commendationfor the best model across the cohort. This page is the overview; each model's page below carries the architecture, the full training recipe, every metric on both splits, and a live workshop you can run in the browser.

Methodology

How we approached it

1
Baseline in Orange
Image embeddings + neural-network and logistic-regression classifiers — the assignment, reproduced.
2
Two custom models
DINOv3 (attention) and ConvNeXt-V2 (convolution), fine-tuned from large pretrained backbones.
3
Audit the data
Compare every image to every other with a perceptual hash to find near-duplicate train↔val pairs.
4
Rebuild clean
De-duplicate within each class and re-split, so no validation image appears in training.
5
Benchmark on both splits
Every metric, leaky original vs. clean — so the accuracy inflation is visible, not hidden.
6
Test significance
McNemar's exact paired test on the frontier pair, because the validation set is small.
7
Probe out-of-distribution
27 non-weapon images, plus a confidence-threshold defence against forced wrong labels.

The data

What we are working with

The dataset is 714 images across nine weapon classes — six long guns (bazooka, grenade launcher, rifle, shotgun, SMG, sniper), two blades (knife, sword) and the handgun. The classes are roughly balanced, between about 67 and 89 images each, so accuracy is not skewed by a dominant class.

Bar chart of image count per weapon class — Images per class on the original dataset — roughly balanced (≈67–89 per class), so plain accuracy is a fair first summary.

Before trusting any accuracy number we checked the dataset itself. A perceptual-hash audit compared every image to every other and found 82 train↔val duplicate pairs — roughly half of the validation set are near-identical copies of training images. The models had effectively already seen part of the test set, so every number on the official split is optimistic. This is a normal data-quality issue in scraped datasets, not anyone's mistake.

Histogram of perceptual-hash distance from each validation image to its nearest training image — Each validation image vs. its closest training image; the shaded near-duplicate region is the leak — roughly half the validation set.

So we built a leakage-proof split by construction: de-duplicate within each class (union-find over the perceptual hashes, keep the highest-resolution copy), then re-split 80/20 stratified with a fixed seed. 221 of 714 images — nearly 1 in 3 — were duplicates, leaving 493 unique images (395 train / 98 val). The originals are untouched and every change is logged. From here, accuracy means performance on genuinely unseen weapons.