With the development of Industry 4.0, more and more attention has been paid to system intelligent maintenance by various industries, among which rolling bearing is an indispensable and most important component. Existing methods have such limitations as the need for prior knowledge and manual feature extraction. For this reason, a multi-modal variational autoencoder (MMVAE) is proposed to extract useful features from multiple modalities. Firstly, the fault characteristics of multiple modalities are extracted separately by different variational autoencoders containing complementary information. Secondly, a collaborative training method is proposed to maximize mutual consistency. Specifically, feature extraction and clustering for all modalities are employed for collaborative learning. Fault diagnosis experiments on a benchmark rolling bearing dataset were carried out. Compared with other methods, MMVAE showed remarkable results, with an accuracy of 99.13%.