Let O be the center of the square
Let M be the top right vertex of the square
Let N be the rightmost vertex of the hexagon and P be the top right vertex of the hexagon
Triangle PON is equilateral so PN = ON = 1
OM = (1/2) diagonal of square = S /sqrt 2 where S is the side of the square
Angle OMN = 75° Angle MON = 45° Angle MNO = 60°
Using the Law of Sines
sin (OMN) / 1 = sin MNO / (S/sqrt 2)
sin (75) /1 = sin (60)/ (S/ sqrt 2)
S / sqrt (2) = sin 60 / sin 75
S = sqrt (2) sin (60) / sin (75)
S = sqrt (2) (sqrt (3) / 2) / ( sin (45 + 30)
S = sqrt (6) / (2 (sin 45cos 30 + cos 45 sin 30) )
S = sqrt (6) / ( 2 ( sqrt (2)/2 * sqrt (3)/2 + sqrt(2/2 )(1/2) )
S = sqrt (6) / (2 (sqrt (6) /4 + sqrt (2) / 4))
S = sqrt (6) / [ (sqrt (6) + sqrt (2)) / 2
S = 2 sqrt (6) / (sqrt 6 + sqrt 2)
S = 2 sqrt (6) ( sqrt 6 - sqrt 2) / ( 6-2)
S = (12 - 2sqrt (12)) /4
S = 3 - 2sqrt (4) sqrt (3) / 4
S = 3 - 2*2 * sqrt (3) / 4
S = 3 - 4sqrt (3) / 4
S = 3 - sqrt (3) ≈ 1.27
Here's a pic (AM = 1/2 the side of the square ≈ .63 )