surely it would be more frightening if just one person turned up... and you didn't like them.
I'd have thought founding a group would be easy enough, book a room in a church hall/above a pub - get a little website and some flyers, go on all the places non binary types might hang out in real life and on web and distribute.