Movie drama and AI
You know how when you're young and you see those dramatic movie scenes where the lead role is having a drama argument with a loved one. You feel that the argument is so pointless and illogical and hence the scene isn't very realistic. I mean why would the protagonist argue about something which basically translates to "You can't help me with this! I have to do this on my own!".
But when you grow up a bit and you get your first gray hairs - you start to realise that such seemingly irrational arguments are a part and parcel of life and probably one of the few realistic details in movies.
Given that at times we're not even able to align our fellow loved ones on important things, I do not have a lot of hope for us to be able to align any superintelligent AI. People have grown up following very different paths and trajectories with very different experiences so it becomes difficult to attain alignment. I suspect that a superintelligent AI would be so far removed from humanity and human experiences that we'll barely be able to convince it of anything on any topic.
The superintelligent AI would already have thought about (computed?) anything and everything that humanity can think about collectively. I feel that it will be very hard to convince it to change its perspective on something that has been pre-computed.
Super-alignment problem is going to be a great filter for human civilisation.
Great filter is basically a proposal to explain the Fermi Paradox.
Fermi Paradox - if space is so huge, why don't we see any signs of alien civilisations?
Great filter - every civilisation encounters great filter moments that they have to pass. Only after passing N great filters, a civilisation will become easily visible to everyone in the galaxy. No civilisation seems to have passed the required number of great filters yet so we don't see any alien civilisations in our space telescopes.
Nah dude, everyone for some reason thinks that superintelligent AI will have god like thinking powers. "Anything and everything that humanity can think......" might very well be an infinite set, how does anybody compute that? No amount of intelligence can break the laws of mathematics or physics. AI might take ur job or maybe 0.001 % chance it becomes skynet but that still doesn't make it a god capable of everything.