I do not know that formula so here is my method.

The radius is r units. The encircle centre is the point (r,r)
To find r, the perpendicular distance of the hypotenuse to (r,r) must equal r.
\(r=\frac{|Ax_1+By_1+C|}{\sqrt{A^2+b^2}}\\ r=\frac{|8r+15r-120|}{\sqrt{8^2+15^2}}\\ r=\frac{|23r-120|}{17}\\ r=\frac{120-23r}{17}\\ 17r+23r=120\\ 40r=120\\ r=3 \)
Radius is 3
Diameter is 6