I don't think you are. PERHAPS the Adam's apple makes resonance a bit deeper, but it's not even part of the vocal organs. Essentially, if your voice gets deeper, it's because your vocal cords changed. In the process, it's likely that your larynx was enlarged. And in the process, it pushes out stuff, notably the thyroid cartillage. Mainly the voice deepening is the cause of having a larger Adam's apple, not the contrary. And an indirect cause based on probability at that.
But even so, I'm pretty sure there are other factors than larynx enlargement that make a voice deeper. Meaning that a relatively small larynx can probably somehow produce a deeper voice than expected. Not to mention that even if it's pushed out, the thyroid cartillage might be small enough to still not be visible. Who knows?
What I do know is that I have a pretty visible Adam's apple and no male voice, and that some have a male voice but no visible Adam's apple. And I doubt it's -that- rare that voice and cartillage don't match perfectly.