The possibilities of using numerical methods for studying the stability of quarry sides taking into account the extreme deformability of rocks, including determining the mechanical characteristics of a rock mass, have been studied. Mathematical models of rock masses, reflecting the full range of strength and deformation properties of rocks, must be implemented in the form of numerical procedures that ensure compliance with the specified laws of state. In this work, this task was accomplished using the finite element method. A method and numerical procedures have been developed based on the deformation theory of plasticity and the finite element method, which implement the linear-elastic nature of the deformation of the rock mass in the pre-limit region and loosening and softening in the beyond-limit stage of deformation. The main goal of the work is to develop procedures that implement a mathematical model for determining the mechanical characteristics of a rock mass based on the numerical finite element method. The resulting new effective numerical procedure for implementing the proposed model based on the finite element method allows one to take into account the technological sequence of field development in a single calculation scheme. The considered procedures for solving nonlinear problems are applicable to media both hardening and softening with associated and non-associated laws of state.