If you have a fixed set of working hours (instead of flexible) then I think the manager has every right to ask why are you Away on Teams for most of the day.
Picture this: It's like you have a 10 to 7 PM job, you walk in to the office, put your laptop at the desk and not to be found anywhere around for most of the day. You come back and pack up stuff and leave. Any manager will start questioning that. It would be dumb of him/her to not notice that actually.
On the other hand if you're having flexible work hours and all that matters is getting the job done for the day no matter what the time is, then I don't see a point in tracking your Teams status since only getting the job done matters.