[Thread] Share your experiences with A/B testing – what surprising results have you encountered?

Back in my early days at a startup, we were all about A/B testing to optimize our product's user experience. We took it seriously, perhaps a bit too seriously at times.

One day, we decided to test the colour scheme of the buttons on the app. It seemed like a minor tweak, but we were honestly aiming for anything. The original colour was a soothing blue, and our designer suggested a bold change to bright pink. She thought it would make our app pop and attract more attention.

The A/B test was set up, and I eagerly awaited the results. A week later, the data came in, and it was time for the big reveal at our team meeting. I was excited, thinking that the bold pink buttons would surely lead to a spike in user engagement.

To everyone's surprise, the bright pink buttons had indeed made a significant impact. But it wasn't the kind of impact we had hoped for. User engagement had skyrocketed, but not in the way we expected.

As it turned out, the pink buttons were such an eyesore that users were accidentally tapping them left and right, often resulting in unintended actions. People were complaining about the app being "Aggressively Pink" and "Impossible to use without making a mistake."

The funny part was the passionate feedback we received. Users were not just expressing frustration; they were writing hilarious, borderline poetic rants about the "pink invasion" on the app. 😂

After a good laugh and some consideration, we quickly reverted to the original blue buttons, much to the relief of our users.

What about you all? Share your stories. 😂