Bone is a composite material with a mineral hydroxyapatite (HA) phase and an organic collagen-based phase. Each phase represents about half the material by volume. The precise arrangement of these components at the ultrastructural level is unclear but of great interest in understanding the mechanical functionality of bone. Nanoindentation tests show that the elastic modulus of bone is primarily distributed between 10 GPa and 30 GPa. In this study we examine different ultrastructural arrangements of collagen and apatite phases, to test different proposed models for bone composite ultrastructure within the same finite element modeling framework. Different configurations of the composite are considered, including (a) a compliant phase with stiff reinforcing particles, (b) a stiff phase with compliant reinforcing particles, and (c) an interpenetrating two-phase (co-continuous) composite. An elastic modulus of 100 GPa is used for the mineral phase and 100 MPa for the organic phase, with volume fraction of each phase fixed at 0.5. Stiff phase continuity (as the only continuous phase in 2D and 3D or as one of two continuous phases in 3D) gives rise to effective composite elastic modulus values of 25–35 GPa, similar to the experimental results for bone modulus. Isotropic models with compliant phase continuity only give rise to moduli around 300 MPa, far below experimental results. Anisotropy was evaluated by calculating effective moduli in parallel and transverse directions relative to the primary axes of rectangular particles. High aspect ratio, stiff particles embedded in a compliant matrix do result in a substantially stiffened composite in the direction of the particles when compared to symmetric particles. However, this configuration results in a material with an effective elastic modulus of 2 GPa along the particle direction but a transverse modulus of only 250 MPa. Decreased interparticle spacing in the direction of loading was the mechanism for stiffening parallel to the particle long axis, demonstrating an indirect effect of particle aspect ratio. Although many bone models have considered the mineral as a particle reinforcement phase, the current results suggest this arrangement would not give rise to a material with bonelike properties, particularly when transverse modulus is considered, regardless of the particle geometry. Some degree of continuity of the mineral phase is required for bone-like elastic modulus values and thus a partially to fully co-continuous ultrastructural arrangement of phases is supported.