Your question is spot on. As I write, there are only 617,000 Bitcoin addresses with more than $10,000 (USD) of value in them – in the whole world – and only 2.5 million with more than $1,000 in them. The whole sector employs a few thousand people at most. These are tiny numbers. A concerted effort by a few big governments to close the on-off ramps and crypto would be swatted like a fly, even if a few coins were technically still alive.
The political cost to the politicians of a crackdown would be far too high to bear if tens of millions of people were affected. But with just a few million hodlers put out of pocket the blowback would be relatively minor, even in the US. In other words, a governmental crackdown is almost certainly the biggest threat to crypto. The genie is out of the bottle, but it is not yet strong enough to be sure of survival.
Confronting government on the issue of the money supply is only ever going to end one way. To forestall the possibility of that conflict we must either get millions of new people involved, and fast, or else demonstrate that crypto not only offers unquestionable benefits, but that it can also be absorbed into the existing financial system without causing too much trouble. Crypto Crusaders who scoff at regulation and think Bitcoin is going to spell the end of fiat are like Don Quixote and his windmills.