The argument that "men and women are just different, simple as that!" has been used to shut down actual conversation surrounding sex and gender for generations because people are married to these perceived truths about what men and women look like and what they are capable of. You refer to weight classes as if men and women could not conceivably compete in a weight class they both met the requirements for. People have been socialized to view male and female athletes a certain way but there just isn't any research to indicate that there is a meaningful difference at all between transgender and cisgender athletes.
Here is an article from 2016 that refers to two then-recent studies that came to this conclusion. Both the journals themselves are linked in the article, but for ease, the article summarizes the findings quite clearly:
Another study published in 2017,
Sport and Transgender People: A Systematic Review of the Literature Relating to Sport Participation and Competitive Sport Policies, came to the same exact conclusion as these two others:
People think they are being sensible when they say things like "I support trans equality but athleticism is based on physiology of sex rather than gender," but this concern speaks to a lack of understanding of the physiological changes transitioned athletes reach and boils athletic performance down to sex versus all the other factors associated with athletics and fitness. Fallon Fox began her transition in 2006. It's a moot conversation. She has been a woman for her entire athletic career.
This notion that transgender athletes have an innate unfair advantage isn't harmless musing nor is it rooted in factual observation. Attitudes like this create a status quo of scrutiny and doubt. It delegitimizes the accomplishments of marginalized people because they're accused of having an advantage they don't have. Nobody ever cares if transgender athletes compete and lose, but there is this terrible fear that they could win. This is transphobia and this is why Ronda Rousey is wrong.
Any conversation about how to transgender athletes impact the world of sports should be done from the basis of how to incorporate them, not by rationalizing ways to exclude them from competitions. At the end of the day there are actual transgender people who could explain and articulate this better than I can but it must be exhausting having to defend the legitimacy of your existence all the time.