Abstract: The expensive inter-machine communication is the bottleneck of distributed optimization. Existing study tackles this problem by shortening the communication rounds, but the reduction of per-round communication cost is not well-studied. This work proposes a progressive manifold identification approach with sound theoretical justifications to greatly reduce both the communication rounds and the bytes communicated per round for partly smooth regularized problems, which include many large-scale machine learning tasks such as the training of $\ell_1$- and group-LASSO-regularized models. Our method uses an inexact proximal quasi-Newton method to iteratively identify a sequence of low-dimensional smooth manifolds in which the final solution lies, and restricts the model update within the current manifold to lower significantly the per-round communication cost. After identifying the final manifold within which the problem is smooth, we take superlinear-convergent truncated semismooth Newton steps obtained through preconditioned conjugate gradient to largely reduce the communication rounds. Experiments show that when compared with the state of the art, the communication cost of our method is significantly lower and the running time is up to $10$ times faster.