This is what I did:
1) I looked at triangle(BAP) and used the Law of Cosines to find angle(BAP).
I set the length of AB = x.
BP2 = AP2 + AB2 - 2·AP·AB·cos( BAP )
32 = 12 + x2 - 2·1·x·cos( BAP )
9 = 1 + x2 - 2x·cos( BAP )
( 8 - x2 ) / ( -2x ) = cos( BAP )
cos( BAP ) = ( x2 - 8 ) / (2x )
2) Then, I looked at triangle(DAP) and used the Law of Cosines to find angle(DAP).
Since it's a square, AD = x.
DP2 = AP2 + AD2 - 2·AP·AD·cos( DAP )
72 = 12 + x2 - 2·1·x·cos( DAP )
49 = 1 + x2 - 2x·cos( DAP )
( 48 - x2 ) / ( -2x ) = cos( DAP )
cos( DAP ) = ( x2 - 48 ) / (2x )
3) Since angle(DAP) = 90o - angle(BAP) ---> cos( DAP ) = sin( BAP )
---> sin( BAP ) = ( x2 - 48 ) / (2x )
4) Since sin2( BAP ) + cos2( BAP ) = 1
[ ( x2 - 48 ) / (2x ) ]2 + [ ( x2 - 8 ) / (2x ) ]2 = 1
[ ( x4 - 96x2 + 2304 ) / ( 4x2 ) ] + [ ( x4 - 16x2 + 64 ) / ( 4x2 ) ] = 1
multiplying both sides by 4x2:
( x4 - 96x2 + 2304 ) + ( x4 - 16x2 + 64 ) = 4x2
2x4 - 116x2 + 2368 = 0
But this doesn't have any real solutions?
Where did I mess up?