You are right. After a point most of your time will go to managing other things in the company. Still work on your own thing.
Being a founder myself, I can tell from personal experience. What you love now might not be the thing that you love in the future or maybe not with the same intensity. Just talk to multiple people in 5th or 6th year of their career, you’ll find a general pattern.
Running your own show will give you freedom and that freedom is really more valuable than most other things you can do for someone else. Also if you are able to make money doing your own thing, that enriches life to another level altogether. I hope it helps.