In this paper, we analyze properties of the Continuous Updating Estimator (CUE) proposed by Hansen et al. (1996), which has been suggested as a solution to the finite sample bias problems of the two-step GMM estimator. We show that the estimator should be expected to perform poorly in finite samples under weak identification, in particular, the estimator is not guaranteed to have finite moments of any order. We propose the Regularized CUE (RCUE) as a solution to this problem. The RCUE solves a modification of the first-order conditions for the CUE estimator and is shown to be asymptotically equivalent to CUE under many weak moment asymptotics. Our theoretical findings are confirmed by extensive Monte Carlo studies.