The paper considers the mathematical formulation of geothermy direct three-dimensional problem for stationary and nonstationary forms. The particularity of the considered formulation is the possibility of accounting heterogeneity structure of the modeled object by density and thermophysical parameters. The author introduces the system of typical approximating elements - vertical prisms with arbitrary upper and lower bases and constant values of density, coefficients of thermal conductivity and thermal diffusivity, as well as the density of heat generation caused by the decay of radioactive elements of sedimentary rocks. On the border of the simulated volume the conditions of mixed type: the value of heat flux from the base and the temperature detected by the values ??of the secular variation of the Earth's surface temperature were set. In the problems of estimation of predicted resources for the oil and gas regions the considered boundary values allow taking into account paleoclimatic conditions for generation of petroleum hydrocarbons. The theory of potential and the theory of linear integral equations are used as methods for solving this problem. The accuracy and speed of the algorithms are shown by the calculations of test examples.