Work on an exciting project with a toxic team or reassign/quit?

I'm working on a project that is very interesting in the fintech space. The team I'm working with is quite toxic and my seniors are not very supportive. Office politics has led to founders assuming I'm bad at my job.

I'm feeling conflicted. On one hand, I would like to continue working on the project as it might be a good learning opportunity and may open up doors to better companies in the future + the current company otherwise isn't bad (just three people I directly with are the problem). On the other hand, constant criticism and negative feedback misdirected towards me is making me want to ask my manager for another project with a team I'm more comfortable with.. or take some time off and quit, find another job.

I've tried my best to get along with the colleagues, but they seem to be playing their own game of politics even though I don't want to get involved. I want to do my job and live my life outside of it. My manager isn't being helpful either - raises his hands and says it's for us to figure it out..

What would you do? I'm happy to hear your suggestions on this. Any help would be appreciated.