Benefits report
🖥️ Product design & leadership
📉 Data visualization
🧠 AI explainability
Design team
Product → Myself and McKenzie Gutshall
Content design → Apurva Dabhade
Design research → Diana Sapanaro
Awards
🏆 Outstanding Technical Achievement Award for Trust in AI for Sterling Fulfillment Optimizer
🏆 Artificial Intelligence Excellence Awards, Nominee Finalist
✦ Challenge
IBM Sterling Fulfillment Optimizer (SFO) uses AI to optimize order management systems by finding the best store, warehouse, or vendor for online order fulfillment. This product can save customers up to 5x more monthly than a standard order management system, but how would they know?
Our customers needed a self-sufficient way to report the financial and inventory benefits of using SFO to their stakeholders. At the time, if they wanted to discover these benefits, they would have to contact IBM support to pull the data and then calculate the results to share with their teams. We used two hills to guide our user journey:
The Fulfillment Manager can understand the correlation between the parameter settings and the benefits of Optimizer so that he can optimize his future optimization setting strategically.
The Fulfillment Manager can view optimized orders over a period of time to understand the financial benefits of Optimizer without the assistance of IBM.
✦ Action
1. Understanding the AI
The trickiest part of designing for AI is, well... understanding the AI. We worked closely with the Chief Scientist, Pavel Zelinsky, to understand how SFO selects the best node (store, warehouse, or vendor) for online order fulfillment. He walked the design team through a very detailed data science methodology deck. This is what we heard:
However, we had to understand the formula to design for and explain the monetary and inventory benefits of SFO. I sat in our design lab, watched the recording of our meeting with Pavel numerous times, and drew out my understanding much like an infographic:
First, I ran this by the design team to ensure we were on the same page, then shared it with Pavel. He had a few edits. 😅
The numerous iterative updates and collaboration sessions between design and data science led to a mutual understanding of the SFO AI:
2. Card sort research activity
Now that we could explain how the AI model works, we needed to speak with customers to find out the data they care most about when reporting benefits. Diana Sapanaro, the design researcher, led the effort with help from myself and McKenzie Gutshall, the other product designer.
We aimed to determine which KPIs our customers wanted to include in monthly, quarterly, and annual reports through a 60-minute remote card sort activity. Example:
3. Sketching
Now that we understood the AI model and had an idea of the KPIs our customers care about, it was time to ideate. McKenzie and I worked together for years. We had a tried-and-true design process was utilized for this work.
3A. Don’t talk, sketch
After we have a common understanding of the assignment and user story, we start sketching. However, we don't discuss any ideas with each other. We sketch whatever ideas come to mind on whiteboards, notebooks, windows, basically whatever we can find.
Credit: McKenzie GutshallCredit: Gina ranalli3B. Review
Next, we hang our sketches on a wall but don't explain our ideas to each other. The other person describes how they perceive the design. Only after can we clarify our thought process.
3C. Merge
After discussing the pros and cons of each, we choose what should go in the first draft that will be shared with the cross-functional team. I jokingly call this phase “Frankensteining” our work.
Network levelItem level4. Iterating with the cross-functional team
The first design is rarely the best. You need feedback from customers and stakeholders from other disciplines to get it right. McKenzie and I met with internal stakeholders (content designers, design researchers, product managers, data scientists, frontend engineers, and backend engineers) at least once a week to iterate. We eventually landed on a low-fidelity mockup we shared with IBM Sterling Fulfillment Optimizer Sponsor Users.
5. Lofi feedback with Sponsor Users
Our goal was to understand if our vision matched customer needs and expectations through evaluative feedback sessions with some of our large-scale retail customers.
Feedback
Reviewing the monetary and inventory benefits was useful.
The right data was included, but the visual representation on the "View benefits screen" was not intuitive for our users. The reinvestment portion of the charts was particularly confusing.
Our customers preferred reviewing and reporting on the numbers in the "Compare alternative model" view.
Quotes
Regarding the "View benefits" view:“This is a cool view. For me, I feel like it helps us answer the question of the different thresholds we put on. That’s asked of us a lot, there’s this big algorithm, are we seeing benefit. This helps us put some context into it.”
Regarding the "Compare alternative models" view:“I think that these are all metrics we want to see. I don’t think this is the best way to represent the info.”
6. Delivered designs
Taking everything we learned from the numerous iterations with Sponsor Users and the product team, we delivered the following design that was then coded through collaboration with frontend engineering, backend engineering, and QA.
©️ Copyright IBM International Business Machines Corp.
This is what is in production as of April 2023:
7. Retrospective
Following agile best practices, I led the cross-functional team through a retrospective to understand areas of improvement.
✦ Outcome
The product team delivered an experience that gave our customers a self-sufficient way to understand and report the monetary and inventory benefits that the AI of IBM Sterling Fulfillment Optimizer provides.
From the IBM Sterling Order Management Executive Director of Engineering
From the IBM Sterling Fulfillment Optimizer Product Manager
IBM Order Management Program Director of Product Management
Customer quotes
“This is a lot of what we would like to see and what we asked for.”
“I like it a lot. It’s certainly the KPIs that I would fit in to the budget talk that I have every month.”
Awards
🏆 Outstanding Technical Achievement Award for Trust in AI for Sterling Fulfillment Optimizer
🏆 Artificial Intelligence Excellence Awards, Nominee Finalist