My boyfriend is planning on hanging out with someone I know wants to get with him and I'm really upset.?

My boyfriend of two weeks now has told me about this guy thats wanting to hang out with him. He told me that before we started dating this guy wanted to make him dinner but he didn't ever go out with him. But he also wanted to ask me if it was ok if the two of them hung out sometime. Now my boyfriend told me he isn't attracted to him and that I can trust him and I really want to but because it's only been two weeks I'm still a bit upset over the fact that he will be alone with a guy I know wants in his pants. So what should I do? He asked me flat out if it was ok and I told him yes I trust him but that I wasn't happy with it but I'd deal. The more I think about it the sicker I get and I'm thinking of the worst things that could happen and it makes me upset. So either don't fret about it and trust him and try my best to swallow it all and not freak out or I tell him how much it upsets me he would even associate himself with someone who knowingly wants to be with him. Please help me.