One could experiment using instead of the angular distance from the center of the bounding box the same distance compared to the maximal possible angular distance in the same direction.
This way the deviation for larger objects would not be punished as much as for smaller objects.
getMaximalPointOnBox() of AbstractDistanceMeasure already helps with the implementation.