What do you like best?
When we started implementing Optimizely on mobile (iOS and Android), the team at Optimizely was particularly helpful in working with the engineers and product team. This really accelerated the process for us to dive in and get started using the product.
The Optimizely team seems very passionate about helping their customers get the most out of their platform. They are always available to answer an e-mail, support request, forum post, jump on a phone call and even come by for a visit. On the flip side, they are always asking for feedback on the product, which provides a sense that we are working with a living, breathing company.
Optimizely has spent a lot of time understanding the intricacies of multivariate testing and statistical analysis and made it dead simple to see the results in their product. Even if your focus isn't around data, you can plainly see how your experiments are performing and focus on what you are best at. If you want to learn more, they've provided a lot of resources to help you understand concepts like statistical significance.
What do you dislike?
I'd like to see more APIs to be able to export data and integrate it a data warehouse so that we could perform our own analytics.
Sometimes it's not obvious how long an experiment will need to run based on a product's existing traffic. Having an estimate integrated directly into the experiment would be very helpful.
Sometimes I am concerned that they've made things too simple to get in and start running experiments, without having proper constructs on what experiments you should be running and what you are actually testing. Just because you've seen results on testing a button in an early stage of a funnel does not mean your end conversion is going to change.
Recommendations to others considering the product
One of Optimizely's biggest pluses is that it works on multiple platforms, which many other products do not. Another major factor was their evangelization not jut of their product, but of conversion rate optimization and statistical relevance in general. Finally, when speaking with them, they didn't so much try to sell their product as they spoke to how it could integrate into our overall workflow.
What business problems are you solving with the product? What benefits have you realized?
Optimizely is used as a critical piece of understanding and testing user behavior by product, design, marketing, and engineering groups. It is used in multiple ways, from quick tests to see how users interact with buttons, hero images, promotional text, to test out operational changes and feature roll-outs, all the way to quantifying and proving hypotheses from in person user tests, surveys, and other qualitative data. From a platform perspective, Optimizely is used for web, mobile web, and mobile apps (iOS and Android). It's helpful to have a single platform for all your testing needs across different platforms as the teams already have enough other tools that are used on a daily basis to manage projects and analyze results.