Parametrically excited surface wave modes on a fluid layer driven by vertical forcing can interact with each other when more than one spatial mode is excited. We have investigated the dynamics of the interaction of two modes that are degenerate in a square layer, but non-degenerate in a rectangular one. Novel experimental techniques were developed for this purpose, including the real-time measurement of all relevant slowly varying mode amplitudes, investigation of the phase-space structure by means of transient studies starting from a variety of initial conditions, and automated determination of stability boundaries as a function of driving amplitude and frequency. These methods allowed both stable and unstable fixed points (sinks, sources, and saddles) to be determined, and the nature of the bifurcation sequences to be clearly established. In most of the dynamical regimes, multiple attractors and repellers (up to 16) were found, including both pure and mixed modes. We found that the symmetry of the fluid cell has dramatic effects on the dynamics. The fully degenerate case (square cell) yields no time-dependent patterns, and is qualitatively understood in terms of third-order amplitude equations whose basic structure follows from symmetry arguments. In a slightly rectangular cell, where the two modes are separated in frequency by a small amount (about 1%), mode competition produces both periodic and chaotic states organized around unstable pure and mixed-state fixed points.