We're a small minority, perhaps one or two million in the United States. (It's impossible to get any reliable figures.) Given that the general population is over 300 million, many people, perhaps most people, in the US don't know any TG people personally. Or if they do know TG people, they may not even realize it.
This is a dangerous situation, because when ignorant and hateful people defame us, most Americans are susceptible to their views because they have no actual TG acquaintances to judge by.
As a parallel example, when most lesbians and gay men were closeted several decades ago, most Americans readily accepted the hateful rhetoric that claimed gay people were evil and sick. As more and more gay people came out, people began to realize that they did know gay people. Gay people were their neighbors, their coworkers, and even their relatives. It's one thing to hate gay people in the abstract when you think you don't know anyone gay. It's a lot harder to hate when you have a nice lesbian cousin that you love.
Right-wing and fundamentalist hate-mongering against TG people does a LOT of damage. Since most people don't have any personal acquaintance with TG people to serve as a reality check, many ordinary people accept the most outrageous lies about us because they assume the people who defame us must know what they are talking about. It's a particular problem because some of the hate-mongers occupy positions that are normally accorded a high level credibility: religious leaders and public officials, for instance.
Make no mistake about it. We are being defamed in an organized way by opportunists and Bible-thumpers for base motives of political and religious advantage. Reactionaries are literally trying to convince Americans that TG people are perverted men wearing dresses who want to sneak into public restrooms for the purpose of molesting little girls. And it's causing a lot of damage. Look no further than the vote in Houston TX, where exactly that pitch successfully moved the electorate to overturn the city's human rights ordinance. We need to take this seriously.