Thanks for the replies.
Much later I learned he was previously born a girl, years later, that rumor was being spread. I grew up in a small town so eventually everyone knew. I did not understand any of it by then. Never heard about anyone who transitioned, I was really ignorant and young I guess.
Yes, I now see how it actually was bullying... he grabbed and shook me, telling me to confess. The weird thing is though, I kept saying I was a boy. When I look back, I try to make sense of it... At the one hand, he might saw something about me that I wasn't even sure of myself... maybe he was trying to help me in some weird way... or maybe not. I am not even sure why I am telling this, it's just something from my past, a piece of the puzzle.