They could though. I doubt it happens but cancelling the Tokyo marathon was a huge deal. Only elites being allowed to run it. 30,000+ people two weeks before the event left with travel plans they have to change. It’s a world marathon major too so not a small thing.
What I was getting at is that hosting all the athletes, their coaches, trainers, etc at the Olympic village is a HUGE undertaking and the issue with the virus would be it spreading AMONG THE ATHLETES in close quarters, not necessarily the fans in attendance being an issue. The village is staffed by local residents, who go back to their homes every day after they are done working. It's too risky.