Self-exciting point processes (SEPPs), or Hawkes processes, have found applications in a wide range of fields, such as epidemiology, seismology, neuroscience, engineering, and more recently financial econometrics and social interactions. In the traditional SEPP models, the baseline intensity is assumed to be a constant. This has restricted the application of SEPPs to situations where there is clearly a self-exciting phenomenon, but a constant baseline intensity is inappropriate. In this paper, to model point processes with varying baseline intensity, we introduce SEPP models with time-varying background intensities (SEPPVB, for short). We show that SEPPVB models are competitive with autoregressive conditional SEPP models (Engle and Russell 1998) for modeling ultra-high frequency data. We also develop asymptotic theory for maximum likelihood estimation based inference of parametric SEPP models, including SEPPVB. We illustrate applications to ultra-high frequency financial data analysis, and we compare performance with the autoregressive conditional duration models.