Unsupervised and self-supervised learning, like RotNet, are getting very closed to supervised methods on many transfer tasks.
Deep Clustering for Unsupervised Learning of Visual Features
https://arxiv.org/abs/1807.05520
Deep Clustering for Unsupervised Learning of Visual Features
https://arxiv.org/abs/1807.05520