Understanding the dynamics of interest rates and the term structure has important implications for issues as diverse as real economic activity, monetary policy, pricing of interest rate derivative securities and public debt financing. Our paper follows a longstanding tradition of using factor models of interest rates but proposes a semi-parametric procedure to model interest rates. In a semi-parametric approach one typically parameterizes the object of interest while leaving unspecified the rest of the model. We construct factors as linear functionals of key economic time series involving unknown parameters, but treat the response of interest rates to the factors in a nonparametric way. The Average Derivating Estimator, which is a semi-parametric procedure proposed by Hardle and Stoker (1989) and Powell, Stock and Stoker (1989), allows us to proceed in two steps, namely we first identify factors without assuming knowledge of the response function of interest rates to the factors. Once the factors are identified, we proceed with estimating the response function using nonparametric methods. We can view our semi-parametric approach as a prelude to a fullblown parametric formulation for a factor term structure model. Indeed, our empirical results suggest a short term rate specification which deviates from standard parametric models often considered in the literature.