I've always been fairly weak in my arms. Since I was small growing up (being rather girlish in grade school) I was picked on quite a bit. That also prompted me to learn some good defensive maneuvers. Izumi, what type of martial arts did / do you practice? I've learned a few Aikedo / Police moves that can get you out of just about any situation. You could probably youtube them if it concerns you so much.
Pebbles: if you want to hurt the guys ribs, punch downward, not into them. Try to catch between bones. The ribs are meant to go in, but not to spread out. When you punch the ribs directly with a straight punch they go in, if you punch at an angle, the ribs spread and causes pain.
Good defense is learning what spots hurt when only using a few pounds of force.
EDIT: Just be careful, don't punch him too hard like that, you might actually break a rib and I know yall are just playing around!