Sounds more like an open relationship than polyamorous relationship...or I mean, it just seems to me polyamory is more effective when all parties know, acknowledge, and love each other equally, whereas it seems like you dont seem to even know his two other lovers.
If he only agreed to try things out and you dont seem to connect with him well I unfortunately dont see that working well. But...well hey I could be wrong, try to set up more dates with him and find common ground, he might grow to acknowledge and understand you better, especially since he's open to giving you a chance.