I’m sorry you went through that, I’m sure it wasn’t easy to get through. It must have taken a lot to be able to accept that and still want to be in a relationship together, and I applaud you for that strength and patience.
I also think you should communicate these thoughts and doubts to your spouse. Maybe what you need is closure on this particular issue, and to discuss it as openly as you can. I’m not married so I haven’t gone through anything like this, but I have had similar reactions to problems with some of the people I love. This conflict of going back-and-forth between several emotions was driving me crazy, especially when mistrust got in the way, and in the end I decided to just be honest and say what was on my mind. Sometimes it was harder than other times, but by the end of it I was always glad I’ve said something to the other person. Only then was I able to shut off my thoughts from turning into doubts and consuming my mind.
You went through a very hard thing, and you don’t deserve to suffer the consequences like this. Don’t be afraid of asking for the things you want.
I hope this was helpful, and I wish you all the best.