Ammonia is the classic system for $\ce{sp^3}$ hybridisation save methane. The lone pair (and each of the $\unicode[Times]{x3C3}$-bonds) has almost $25~\%$ s-character which corresponds nicely to $\ce{sp^3}$. However, the whole system can also swing around, changing its hybridisation to $\ce{sp^2}$ and back; a process during which the lone pair is temporarily in a p-type orbital and the s-character of the bonding orbitals increases.
Elements outside of the second period show a much smaller tendency to involve the s-orbital in bonding. The bonding orbitals only have an s-character of approximately $16~\%$. This also means that the phosphorous lone pair has a much higher s-character of approximately $50~\%$, while the bonding $\unicode[Times]{x3C3}$-orbitals have a larger p-character. (This also explains the much smaller $\ce{H-P-H}$ bonding angle of approximately $90^\circ$.) An orbital of high s-character has a long way to go to turn into a p-type orbital, and the three largely p-type $\unicode[Times]{x3C3}$-orbitals have an equally long way to go to give $\ce{sp^2}$-type orbitals so the interconversion and thereby the inversion of $\ce{PH3}$ is strongly hindered.