[Objectives] The instability and failure of rock mass structures originate from the shear failure of rock joints. Therefore, understanding the shear behavior of rock joints is of great importance for understanding the mechanical properties of rock masses, evaluating the safety and reliability of rock engineering, and exploring the mechanism of geological phenomena. [Methods] To investigate the effect of deformation and failure patterns on the shear strength of rock joints, this study applied the variable cross-section beam theory to analyze stress changes during direct shear process of regular dentate joints. [Results] The possible failure patterns of dentate protrusions included shear tooth-breaking failure, tensile tooth-breaking failure, shear climbing-tooth-breaking failure, and tensile climbing-tooth-breaking failure, with transitions possible between these modes. The failure process of regular dentate rock joints was theoretically analyzed, identifying the failure patterns and corresponding horizontal displacements under different mechanical and geometric conditions. Prediction formulas for shear strength corresponding to different deformation failure patterns were derived, and the conditions for the occurrence and transition of these modes were established. Using parameter sensitivity analysis, the effects of mechanical and geometric factors (e.g., stress level, rock strength, undulation angle i, and width l of dentate protrusions) on failure patterns and shear strength were discussed. To validate the applicability and accuracy of the theoretical predictions, direct shear tests were conducted on regular dentate red sandstone joints with undulation angles of 40° and 60° under different vertical stresses (0.5, 1, 4, 6, and 8 MPa). Comparison between experimental results and theoretical calculations confirmed the correctness of the theoretical predictions. [Conclusions] This study provides theoretical support for further investigation into the generation mechanism of shear strength in natural rock joints. It should be noted that in the analysis of the stress state of the rock joints, the mechanical model of the dentate protrusions was simplified to a variable cross-section cantilever beam, and the failure surface of the protrusions was assumed to be a horizontal plane. Such simplifications may lead to deviations between the theoretical and actual stress distributions of protrusions. Future work will attempt to apply elasticity theory to determine the stress distribution of protrusions, thereby improving the accuracy of theoretical solutions.