This paper is devoted to the question of optimal portfolio construction for equity factor investing. The first part of the paper focusses on how to make sure that a given equity portfolio has the targeted factor exposures, even before imposing any constraints. We show that such portfolios can be derived from mean-variance optimization using stock expected returns as inputs provided these are built in a robust way from information about the factors. We propose a framework to build those robust stock expected returns and show that the targeted factor exposures are retained by the portfolios both before and after applying realistic constraints, e.g. long-only. Other more simplistic approaches fail. In the second part of the paper we illustrate the application of the framework to a practical case where the objectives are, first, to decide about the risk budget allocation to factors in some pragmatic way; and second, to construct a long-only constrained portfolio that retains the targeted exposures to four factors from well-known asset pricing equity models, namely High-minus-Low (HML), Robustminus-Weak (RMW), Conservative-minus-Aggressive (CMA) and Momentum (MOM).