Future extremely large telescopes will be the engines of major progress in the fields of star and planet formation, brown dwarfs, and extrasolar planets. Their throughput will enable spectroscopic studies of the structure of brown dwarf atmospheres; reveal the composition and kinematics of protoplanetary disks; extend radial velocity searches for extrasolar planets to fainter stars and lower masses; and characterize the surfaces of the most distant Kuiper Belt objects. Their resolution will allow us to resolve and track the orbits of close binary young stars and brown dwarfs, establishing their dynamical masses and calibrating their evolutionary tracks; probe the inner region of young stellar object disks, resolving the jet collimation region in accreting systems and disk inner holes in more evolved ones; and move the horizon for stellar companion searches inward to 40 mas, enabling the direct detection of hot young planets orbiting 10 AU from young T Tauri stars. The high contrast imaging capability of future ELTs is uncertain, and depends on future developments in extreme adaptive optics. Wavefront control considerations suggest a fundamental atmospheric contrast limit of 10$^{-8}$ for companion searches to nearby solar-type stars, below which detections are unlikely to be possible. To reach this performance level, continuing investments will be needed in extreme adaptive optics work, in addition to careful attention to the specialized requirements of ultra-high contrast imaging in ELT design. Above the 10$^{-8}$ contrast limit, a 30m telescope has the potential to directly image about a dozen of the currently known radial velocity planets.