Yo let me tell you something.
We (men) go thro this shit from childhood and when men speak with other men, we know instinctively the common pressure the other person is going through.
common pressure means wife, kids, parents, & society
The probable reason for you would be, most women don’t have life or death situation. I believe your manager is looking after you. He is worried about how others will treat you even though you have good performance.
Because you have never experienced what he is saying, it might feel like it is unfair to you.