The intricate anatomy of the eyelid, and the subspecialization trend in pathology made it more difficult for general surgical pathologists to maintain high accuracy across the entire range of diverse eyelid lesions. A lack of access to subspecialty expertise, and misalignment of diagnosis for eye pathology can slow diagnostic speed, resulting in intraorbital and intracranial extension and/or systemic spread which could threaten vision and life. Here, we developed a robust diagnostic deep learning system (DLS) to detect eyelid tumors using digital histopathological sections based on 473,037 pathological patches from 794 haematoxylin-eosin [H&E] stained whole slide images (WSIs) from two hospitals. This 9-class diagnosis task included top five benign and four malignant eyelid tumors. We first proposed a cascade-network instead of single network, to use the features from both histologic pattern and cellular atypia in a holistic pattern. Our model utilizing cascade-network design achieved 1.0 and 0.946 accuracy in the test and independent test set, respectively, for benign and malignant binary classification; without cascade-network design, accuracy was 0.957 and 0.887, respectively. For multiple classification of individual disease, the DLS with cascade-network design achieved 0.989 and 0.931 overall accuracy for WSI diagnosis in the test set (9-class) and independent test set (8-class) respectively, while without cascade design achieved 0.774 and 0.662. In conclusion, this DLS, using cascade-network design, can automatically detect malignancy in histopathologic slides of common eyelid tumors with a high degree of accuracy, which also has potential to augment histopathological diagnosis for a wide range of tumors.