Optimization explainer
🖥️ Product design
🧠 AI explainability
Design team
Product design → Myself and McKenzie Gutshall
Content design → Apurva Dabhade
Design research → Diana Sapanaro
Awards
🏆 Outstanding Technical Achievement Award for Trust in AI for Sterling Fulfillment Optimizer
🏆 Artificial Intelligence Excellence Awards, Nominee Finalist
✦ Challenge
IBM Sterling Fulfillment Optimizer (SFO) uses AI to optimize order management systems by finding the best node (store, warehouse, or vendor) for sourcing online orders. Fulfillment Managers can discover which node was selected to fulfill an order but had no insight into why that node was chosen. It was a complete black box. For example, a store that was hundreds of miles away could be picked over a much closer warehouse, leaving users to wonder if SFO was making the best decision.
“We need a level of comfort to understand why a decision was made... it’s more about comfort and clarity than anything else”
✦ Action
1. User Research
1A. Discovery
Building the optimization explainer tool was the first epic McKenzie Gutshall and I were assigned to for IBM Sterling Fulfillment Optimizer. Our first task was to review existing documentation to understand the product and user needs. We completed research sessions with customers to dig deeper into their roles, how they currently use SFO, and their needs and pain points with the product. These sessions were guided discussions led by our UX researcher, Diana Sapanaro.
1B. Excel walkthrough
Engineering emulated the information and actions that could be available through Optimization Explainer via an Excel spreadsheet. Users gave feedback on the overall flow and data to help us identify what worked well and could be improved.
1C. Card sort
To continue our generative research journey, we created a card sort activity in UserTesting that listed the Key Performance Indicators (KPIs) that had the potential to be valuable to our customers. Our users were tasked with sorting each KPI into must-include, nice-to-have, or not needed buckets and then prioritizing them in order of importance. This gave us a clearer understanding of what information was most valuable to increase the visibility and explainability of our product.
1D. Consolidate feedback
The feedback from the Excel and card sort exercise was synthesized into a checklist for the product team to reference throughout the rest of the design process.
Blurred text is IBM confidential2. Hills
The design team worked with product management to craft three hill statements for Fred, the Omni-Channel Fulfillment Director, to guide our design process.
The fulfillment director can understand why Optimizer fulfilled orders through specific nodes within 5 minutes.
The fulfillment director can access all relevant information needed to explain how a specific order was fulfilled in a single tool.
The fulfillment director can communicate sourcing details to key stakeholders without requiring assistance from IBM.
Fred
Omni-Channel Fulfillment Manager
Maximizes customer satisfaction and minimizes costs for his retail company.
3. User flow and UI design
3A. Sketch
Using our generative research and hills as a guide, we sketched wireframes. McKenzie and I defined user flows and UI layouts separately and then shared them with each other, much like a design critique.
My sketchesMcKenzie Gutshall's sketches3B. Digital lofi
After discussing the best-unified vision of our sketches, digital lofis were created in Sketch. We then shared our work with product managers, frontend engineers, backend engineers, content design, and data science. Collaborative sessions were held at least once a week to iterate on the designs. Once everyone was comfortable with the user experience, we collected evaluative feedback from our customers.
3C. Evaluative feedback
Multiple 60-minute evaluative research sessions were scheduled with Fortune 500 retail customers. Our goal was to discover whether or not the product team’s user experience helped relieve our users' pain points around visibility and understanding why SFO selected a node for order fulfillment.
Feedback
Overall, the designs matched users' expectations.
Users wanted additional node and cost data table columns, including cost breakdowns.
Customers requested additional filtering and drill-down capabilities.
“This is robust and awesome.”
“Out of the gate, this is great.”
“This is the data that will be super helpful.”
3D. Deliverables
Once the team aligned on a design direction based on user feedback, we created high-fidelity deliverables.
McKenzie and I worked closely with engineering to ensure the experience matched the team’s vision and design standards, including IBM’s Carbon Design System. This is what’s available to customers in production as of April 2023.
✦ Outcome
Our users now understand why SFO chose a node for order fulfillment. We successfully increased customer trust in AI. The Optimization Explainer feature even contributed to converting a rocky customer relationship into an $8M contract extension.
FROM THE IBM STERLING ORDER MANAGEMENT EXECUTIVE DIRECTOR OF ENGINEERINGFROM THE IBM STERLING FULFILLMENT OPTIMIZER PRODUCT MANAGERFROM THE IBM ORDER MANAGEMENT PROGRAM DIRECTOR OF PRODUCT MANAGEMENTCustomer quotes
“Explainer was huge in letting us really understand how AI was making decisions allowing us to be confident to use the tool more effectively to help optimize our costs.”
“I’m pretty pleased with it. You’re getting at the right level of fidelity. It’s very intuitive.”
Awards
🏆 Outstanding Technical Achievement Award for Trust in AI for Sterling Fulfillment Optimizer
🏆 Artificial Intelligence Excellence Awards, Nominee Finalist