The Hill Sphere is basically where the secondary body's gravity dominates over the primary's. https://en.wikipedia.org/wiki/Hill_sphere
Use the second equation there to calculate it. Note that actually stable prograde orbits only seem to be found within about 1/3-1/2 of the Hill Radius. Retrograde orbits can be more stable at a greater distance.
The Roche Limit is where tidal forces can tear satellites apart - it depends on the densities and materials of the planet and satellite. The full explanation is on the wiki page below but I think the best formula to use is the first formula in the Fluid Satellites section (since satellites are not purely rigid bodies and deform under stress). Don't worry about the second equation there, you don't really need to care about oblateness etc :https://en.wikipedia.org/wiki/Roche_lim ... satellites
This post (or any other post I made here) may not be quoted or copied beyond the SF RPG boards without my explicit permission.evildrganymede.net - visit the The Worldbuilding HubCheck out my Youtube channel for all my streamed gaming videos!