It has already been reported that incorporation of thiol-ene into methacrylate resin could reduce shrinkage stress significantly. However, the mass fraction of thiol-ene in thiol-ene-methacrylate resin system should be limited because of deterioration of mechanical properties. In order to increase mass fraction of thiol-ene in thiol-ene-methacrylate ternary resin system, a new fluorinated ally ether (FUAE) was synthesized and mixed with pentaerythritol tetra (3-mercaptopropionate) (PETMA) at molar ratio of 1:1 to form new thiol-ene resin system, and then the thiol-ene resin was incorporated into Bis-GMA/TEGDMA resin to prepare thiol-ene-methacrylate resin system. The results of neat resin system showed that incorporation of new thiol-ene resin into methacrylate resin could increase degree conversion of methacrylate functional group and decrease volumetric shrinkage. In order to achieve the similar mechanical properties as methacrylate resin, mass fraction of thiol-ene in ternary resin system should not exceed 30 wt%. Thiol-ene-methacrylate resin systems with 10 wt% to 30 wt% were chosen to prepare dental resin composites, and physiochemical properties such as degree of conversion, volumetric shrinkage, shrinkage stress, flexural properties, and water sorption and solubility of these composites were investigated. The results showed that, compared with methacrylate resin-based dental composite, ternary resin-based dental composites had several advantages, such as higher DC% of methacrylate functional group, lower volumetric shrinkage, lower shrinkage stress, lower water sorption and solubility. The flexural properties of ternary resin-based dental composites were comparable or even better than those of methacrylate resin-based dental composite.