The identification of structural defects in materials through atomic resolution scanning transmission electron microscopy (STEM) images is pivotal for understanding material properties and facilitating advancements in nanotechnology. However, the efficacy of deep neural networks (DNNs), which are commonly employed for such tasks, is significantly hampered by out-of-distribution drift when applied to experimental data. Furthermore, these DNN models often overlook the incorporation of available prior knowledge, such as the chemical composition of samples, into their analytical frameworks. This study proposes a novel approach that challenges the supremacy of DNNs by leveraging an ensemble of simple, yet highly effective, classifiers trained on data simulated using the multislice algorithm for various two-dimensional materials. By categorizing these materials into groups based on differences in atomic number, our methodology not only utilizes prior knowledge regarding chemical composition but also enhances the specificity and accuracy of defect identification.