Due to long-term work under complex working conditions, gear wear, cracks, chipped teeth, bearing damage, and other faults easily occur in the gearbox, and different faults easily affect each other. Under the same working conditions, the fault diagnosis of gearbox using deep learning method has been widely studied. However, due to the variable and open working conditions of the gearbox, there is less labeled data available, and the fault category in the target domain is unknown, often including unknown fault categories outside the source domain, which makes the conventional deep learning model unable to achieve high diagnostic accuracy. To solve this problem, we propose an open set fault diagnosis method based on weighted adaptive domain. By evaluating the similarity between the features of samples in each target domain and various features in the source domain, the weights are allocated to the samples in the target domain, to promote the separation of different classes in the target domain in the feature space, and correctly align the corresponding classes in the source domain, and achieving continuous fault diagnosis through continuous confrontation between various modules in the model. The method is verified in two rotating machinery datasets, and the results show that the method is effective and superior in the open set compound fault diagnosis of gearbox.