My father's old engineering handbooks state that the formula for a multilayer coil is only approximate, being most accurate where the cross sectional area is close to being a square (10:9) and the proportional diameter is close to the illustrated drawing. I suppose the formula for a single layer coil is also a close approximation.
{EDIT} Multilayer coil formula used is L=(0.8a^2N^2)/(6a+9b+10c) where L is microhenries, (a) is mean radius, (b) is length and (c) is winding thickness in inches. I entered the dimensions into a spreadsheet years ago and found the optimum dimensions for creating the most inductance from a given length of wire using this formula, but I can't post the results with 100% confidence that I remember them correctly. I suspect the ratio was also 6:10:9.{/EDIT}
The REAL formulas I saw in the book are infinitely long and contain undecipherable math.
I would think that if the coil is flat, use the single layer formula. If the cross sectional area is close to square, use the multilayer formula. If it is neither, use the technique used by design engineers in the 1930s: experiment with prototypes until it works.