Thing is that less that way, way less than 1% of businesses are going to luck out and go big.
Being small is not any sort of defect, failure, floundering, etc. Not at all.
If you set out to expand, and wind up losing your shirt and going under, that's one thing. But if being a community restaurant or tire store or whatever is what you've set out to do, then accomplishing that and running a successful business is not in any way a failure.