Using similar triangles (twice)....let AD = x and side of the square= s
AB / AE = BC / DE
sqrt (10) / sqrt (s^2 + x^2) = 3 / s
10 / (s^2 + x^2) = 9 / s^2
10s^2 = 9s^2 + 9x^2
s^2 = 9x^2
s = 3x → x = s/3
And
AC / FG = AB / BG
1 / s = sqrt (10) / sqrt [ (sqrt 10 - 4x)^2 + s^2 ]
1/s^2 = 10 / [ (sqrt 10 - (4/3)s)^2 + s^2 ]
(sqrt 10 - (4/3)s )^2 + s^2 = 10s^2
9s^2 = (sqrt 10 - (4/3)s)^2
9s^2 = 10 - (8sqrt10)s/3 + 16s^2/9
(65/9)s^2 - (8/3)sqrt (10)s - 10 = 0
Solving this for s produces
s = (3/13)sqrt (10) = sqrt (90) / 13
Area of the square = 90 / 169