I assume the Hill Sphere formula here was meant for two bodies with a vast disparity in mass?https://en.wikipedia.org/wiki/Hill_sphe ... d_examples
Because if the less massive body has more than 3/8 the mass of the more massive body, the Hill Sphere radius is more than half the semi-major axis - meaning the LESS massive body would dominate the attraction of bodies even in some cases when they were closer to the MORE massive body than to itself. For example if the less massive body has 8/9 the mass of the more massive body, its Hill Sphere radius is 2/3 (cube root of 8/27) the semi-major axis, meaning it includes object twice as far from it than from the more massive body. Which kind of doesn´t make sense, when you think about it. Or at least *I* cannot make sense of it - and unfortunately these are the sorts of mass disparities that we´re going to find among many a binary pair.
It occurred to me, however, that the rules so far already allow for smaller zones of stable orbits around the less massive (or at least less luminous) star of the pair:
The rules use orbital shells whose radius depends on the parent star´s luminosity. So if star A of a binary pair was identical to Sol, and star B had a luminosity 0.25 times that of Sol, the orbital radius of the "sweet spot" orbital shell (the one that gets the same amount of energy from its star that Earth gets from Sol) around star B would be half that of Sol´s sweet spot, i.e. 0.5 AU. Since a difference of 10 orbital shells equal half/twice the orbital radius, the orbital radius of star A´s shell 0 and star B´s shell 10 would be identical - 1.0 AU.
So let´s say these two stars orbit each other at a distance of 4.0 AU, which would be orbital shell 20 for the more massive star, A. That means that, under the base model for stars of indentical mass, planets could orbit each at shell 0 or further in - 1.0 AU for star A.
Of course, given the disparity in mass, an orbit 1.0 AU out from the less massive star, B, wouldn´t be stable. But orbital shell 0 for star B isn´t at 1.0 AU, it´s at 0.5 AU.
So, a vastly simplified and probably mathematically blasphemous idea would be to simply assume that the reduction in zone of stable orbits due to lower mass and the reduction orbital radii of shells due to lower luminousity are roughly equivalent - so that "take the orbital shell of the more massive star at which the binary orbits each other, subtract 20 and use the result as the orbital shell of the less massive star that is its outer limit for stable planetary orbits" produces somewhat reasonable results.
Unfortunately, for all I can tell from looking at real star data, using luminosity as a proxy for mass in that rule appears to vastly overstate differences in mass - our hypothetical star B would probably be midway through spectral type K with 2/3 to 3/4 the mass of Sun. So the outer limit for stable orbit would be a lot smaller under this rule than in reality.
The rule also doesn´t account for the more massive star having a larger outer limit for stable orbits than it would if its companion had the same mass.
Both problems could be adressed, though, by reducing the minimum separation between the binary orbit and planets around each star by one shell for every X points of luminosity difference, up to a certain minimum.