IBM Learning - User Feedback System
A collection of widgets, forms and reports used across 50+ learning sites to get continual user feedback
What is the learning feedback system?
When brought on as YL user researcher, IBM learning apps did not have a strong system in place for collecting ongoing user feedback to inform design.
I established a no cost SAAS for the learning org, with a collection of forms, widgets, and reports, that product owners could easily turn on to leverage continuous learner feedback.
The system is now used on over 50 learning sites/apps, including some externally sold IBM learning products.
The Project: Develop Culture of User Feedback
I wanted to provide a way for learning products to collect continuous user feedback and enhance the voice of the user in the space
Self Initiated Side Project
As a side project, it needed to scale with minimal time commitment.
No Budget or Support Team
Had to be put in place with no funding for tools like Medalia. Only an existing survey gizmo license to work with and very minimal dev support.
Actionable For Any Team
Many products had no user research or even design support. The system needed to be actionable and useful for all teams.
My Role: Product Owner and User Researcher
I acted as a product owner and user researcher, coordinating directly with learning site owners (clients) and had limited dev support.
In-App NPS Widget
A easy to install configurable NPS widget with reporting for any site
As a learning site with no budget for research or tools, we need a way to know what our users think of our site and to establish a benchmark for comparison with other tools and ourself over time.
NPS allows products to have a benchmark to see what users think about them and to see if they are improving with changes they make. Though there are some shortcomings when using to evaluate a product rather than a brand, as originally intended, it is still useful and this was also built when IBM was pushing hard to standardize NPS as a standard metric in the business.
An in-app NPS form, as opposed to email based, gets significantly more participation and provides better results since the user is providing feedback in context, rather than trying to recall an experience in the past.
Web intercepts are advanced feature of Survey Gizmo, which allows you to embed surveys into a site. Out of box they are quite limited and disruptive to the user flow, but using some custom code, we were able to “hack” this feature to provide an integrated and highly configurable NPS widget, similar to what other high end expensive SAAS solutions offered.
We could easily control when and where it appeared, and combine the form data with metadata from the site itself to offer additional analysis possibilities.
The Survey Gizmo product team was very impressed with this unexpected use of their feature set, having not seen it used in this way before.
The solution was built such that the widget could easily be reused by any site. Simply adding a new intercept in Survey gizmo and associated it with the survey form allows it to appear on a new site, configurable with its own parameters. The site just needed to embed a small line of code to be fully functional.
Display
The NPS widget was designed to display the question immediately on load vs asking them to opt in to see the scale. There were some technical challenges with this initially, but making this changed showed a nearly 3 times greater participation rate.
Settings
When, where, how often and who to show the widget to are all configurable. I customize this by the site in question. For IBM sites with high repeat visit traffic, or where the user can not really make an informed opinion on the tool within a short time of visit, it only shows to repeat visitors after a short delay. For sites that are less likely to see repeat visits or that can be assessed within short time of initial use, it shows after a moderate delay, usually a few minutes. The frequency is scaled based on web traffic. I scale it such that try to ensure we reach a statistically significant number responses. This has proven to mean a much displaying to a much lower % of users on high traffic sites than low traffic sites.
Email capture
While it is best practice to capture email on NPS submission so you can follow up, the learning teams simply do not have the resources to provide this follow up. In which case, capturing email is more harmful than helpful, because it gives the user the perception they might be able to expect correspondence.
The widget allows user to volunteer for feedback sessions
Customizable Feedback Form
A configurable feedback form that any learning site could deploy with nothing more than a url link
As a learning site, we need an easy way for our users to provide self initiated feedback to us, whenever they are inclined to do so. But this shouldn’t replace or hinder a user’s ability to get tech support.
Many sites had no feedback capabilities, but some, like Your Learning, had the basics of a feedback form in place. There were some issues though.
- It wasn’t discoverable
- It was overly lengthy and complex
- Users used it to request assistance, but no one monitored it to respond in reasonable time frame
- Every site needed to create their own survey and own reports
The new form was built by leveraging some advanced survey gizmo features, allowing the form to accept url variables, which control the text on the form along with what questions are displayed.
This allowed us to have a single form, rather than creating a new form for every site that popped up.
It also allowed me to control the survey design, preventing teams with no research support from designing poor survey questions or overloading a feedback survey.
Default question
The default question was designed to be open ended and free form. It always appears first, even if the product has chosen to add other questions. This allows the user to quickly and easily tell us whatever prompted them to click on the feedback form to begin with. Previously the user had to go through some forced questions to just get a similar field.
Question control
A big success of this approach was to help control feedback form design across the org. It also ensured feedback forms would be simple and short. While longer surveys can be tempting, to get lots of specific answers, I am very much against them because they get such low participation rate, and have a very high chance of providing biased data.
Reporting and Analysis
The feedback data is used to drive design changes and make strategic decisions
The system provides automatic reporting for sites so they can closely monitor their user feedback and react
Process and Documentation
I established a semi-automated process and an in-depth self guided website so new products could easily onboard the system.
To help the ever growing audience understand and adopt this new system, I constructed an IBM publisher site that provided a one stop shop for all things related to this system.
To ensure that it didn’t become a drain on my time, I created a semi-automated process for new sites to onboard.
I built a simple survey that product owners could complete which generate a url including the appropriate variables for their custom version of master feedback form. It also sent me the required info to perform the quick steps to set up the NPS.
Impact
The solution was adopted by many sites and used by product and design teams as well as executives
This system allowed many sites to start getting monitoring user feedback for the first time. Over 50 learning sites are using NPS and over a dozen are using the feedback form so far, which rolled out more recently, including the commercial and public versions on IBMs YL LMS.
It also substantially increased the volume of feedback available for sites. Over 250,000 NPS responses have been captured and the feedback form responses doubled on Your Learning when it switched to the new form.
By monitoring user feedback form this system, product teams are able to develop a backlog of requirements and UI changes driven by user input. For Your Learning, I have
Just by starting to collect feedback, it has increased awareness of the user experience and drove a continual improvement. Since starting to collect NPS, there has been a general trend towards higher and higher NPS scores across products.
Driven a large number of UX backlog items that have been crossed off over time
Input to projects like YL Next Gen
