Given the key role of the taxable income elasticity in designing an optimal tax system there are many studies attempting to estimate this elasticity. To account for nonlinear taxes these studies either use instrumental variables approaches that are not fully consistent, or impose strong functional form assumptions. None allow for general heterogeneity in preferences. In this paper we derive the mean and distribution of taxable income, conditional on a nonlinear budget set, allowing general heterogeneity and optimization errors for the mean. We find an important dimension reduction and use that to develop nonparametric estimation methods. We show how to nonparametrically estimate the conditional mean of taxable income imposing all the restrictions of utility maximization and allowing for measurement errors. We apply this method to Swedish data and estimate for prime age males a significant net of tax elasticity of 0.6 and a significant income elasticity of -0.08.