Retriever Playground
Salesforce
June 2023 - Current (Generally Available)
Design Systems, User Research, Experience Design, Information Architecture, Design Strategy
Retriever Playground is a comprehensive testing and optimization environment for retrievers.
I led the design for a testing tool that helps users validate and optimize retrievers before using them in RAG pipelines, delivering a complete 6-month design roadmap. Through this work, I introduced a testing cycle framework (Configure, Test, Analyze) that became valuable enough to share with other testing teams across the organization. By facilitating cross-functional working sessions with product and engineering, I helped clarify what information was truly critical in the testing process—reframing roadmap discussions and simplifying the mental model for users who previously thought testing retrievers required a PhD in patience.
To comply with my non-disclosure agreement, I have omitted and obfuscated confidential information in this case study. All information in this case study is my own and does not necessarily reflect the views of Salesforce.
“My personal anecdotes and thoughts will be in quotes and in blue”
Tools
Figma (Design, FigJam, Make)
Cursor AI
Skills
User Research
Information Architecture
Design Strategy
Experience Design
Design Systems
Communication and Presentation Skills
Team
Som Liengtiraphan (Me, Senior Product Designer)
Barnali Patnaik (Senior Product Manager)
Anubha Dubey (Design Manager)
Clickable prototype of our MVP retriever playground in advanced mode. The password is the same as this page’s.
Impact & Contribution
Lead designer for retriever playground, owning end-to-end UX design from initial concept through GA release and facilitating integrations with various usage points in the Salesforce platform.
Key Deliverables & Outcomes
Shipped to Production
Retriever Playground reached GA- February 2026
Designs
Created 6-month design roadmap for Retriever Playground UI within Einstein Studio
Designed Testing workflows for debugging existing retrievers
Designed Validation and fine-tuning capabilities for custom retrievers
Enabled pre-deployment validation reducing irrelevant results in production
Created a seamless workflow from raw content to refined context and validated retrieval
Established foundation for enterprise-grade RAG success with reliable, explainable, and high-quality retrieval
The design process for this project and the tools used in each step
Project Context
The Retriever Playground was created to address critical gaps in the retrieval testing workflow following feedback from interviews and heuristic analysis.
Findings
Persona Mismatch: Previously, retrieval testing was happening inside Prompt Builder, which didn't align with the data persona who typically sets up Retrievers—not the prompt builder persona who writes prompts
Lack of Necessary Metrics: Prompt Builder lacked the specific metrics needed for retrieval evaluation, such as recall scores and chunk analysis
Broader Use Cases: Retrievers are used in more places than just prompts—including Flow, apps, and ADL—requiring a dedicated testing environment
Pre-Production Validation: Users needed a safe space to fine-tune, optimize, validate, and debug RAG system configurations before deploying to production
Our testing process before retriever playground
Above: Previously, this was the only way to test retrievers. It was roundabout with no retrieval metrics, no ability to view all the documents retrieved, and no way to fine tune configurations to optimize retrieval.
Without a place to test retriever configurations before deployment, costly errors might occur, leading to longer development cycles, retrieval errors, and ultimately lost opportunities and revenue.
The Problem
How might we enable users to efficiently test, debug, and optimize retriever configurations before deploying them in production RAG pipelines?
The Users
There are 3 main categories of users custom retriever builder: Enterprise Customers, Data Scientists, and Product Managers.
-
These are data admins and customer support users who use the retrievers on a daily basis to complete their tasks.
Primary Job to Be Done
Debugging why a retriever is failing to pull the expected document/article
Feature Need
Accuracy testing mechanism
Recall metrics
Retriever configuration adjustments (filters, number of results, etc.)
-
These are AI researchers who’s just it to maintain retriever quality over time and use.
Primary Job to Be Done
Experimenting with models, optimizing retrieval for domain-specific tasks, analyzing intermediate outputs
Feature Need
Advanced retriever configurations
Debugging tools and metrics retrieved information
Performance metrics
-
These are the supervisors that ensures the quality of their developed AI products in their organization.
Primary Job to be Done
Validating retrieval logic and quality before deployment
Feature Need
Save/Activate workflow
Pre-Deployment Validation
Quality Monitoring overtime
Platform-wide process to create and use retrievers in Salesforce
Scope and Constraints
This project's scope covered developing features, identifying integration touchpoints, and delivering the MVP. The primary constraint were technical complexities and platform integration.
Scope
The Retriever Playground MVP delivered essential features across three user personas. Enterprise Customers (Admins/Support) got ground truth selection, recall scoring, and configuration adjustments for debugging retrieval issues. AI Researchers/Data Scientists received advanced mode with rewritten queries, chunk score debugging (vector and keyword), and performance monitoring. Product Managers could save/activate workflows, validate pre-deployment, and reduce irrelevant results. Core P0 functions covered retriever selection, search index display, configurable settings, filtering, query input, ground truth spot checks, recall scoring, intermediate outputs, and save-and-activate workflow. Bottom line: a safe sandbox to test retriever configurations against real data before production—because nobody wants to play "guess and check" with their RAG pipeline!
Constraints
The timeline was aggressive: design kicked off May-June 2025, MVP approval at PM Summit in September 2025, with a targeted launch in the 258-260 release window (October-December 2025). The geographically distributed team across design, PM, and engineering made real-time collaboration tricky, and engineering bandwidth constraints—juggling DC1 and other initiatives—delayed some deliverables. Phase 1 technically couldn't support advanced mode or enriched index features (those needed additional AI gateway logic), limiting the MVP to individual, active, basic configuration retrievers with Data Cloud sources. Despite all that, the team delivered a foundational testing environment that tackled the critical persona mismatch and surfaced the retrieval metrics that mattered.
“My little graveyard of discard designs. We spent around 2 weeks toying with integrations in other products within Salesforce before designing that it’s best if we create a custom UI”
Experience Maps
We started with experience maps to help plan the design and keep everyone aligned on what we were building.
Above: The workflow for our MVP retriever playground
Given our geographical spread and time-zone differences, alignment became our biggest challenge—turns out, herding cats across three continents requires more than just a really good calendar. Journey maps became our North Star, keeping everyone focused on what we were building and which features would ship in each release. Throughout the project, I maintained both an overall experience map and individual release maps to keep us on track.
The designs that didn’t make it
Prompt Builder Integration
We initially explored adding a playground environment within Prompt Builder, where users were already attempting to test their retrievers. However, it quickly became clear that the people writing prompts and those focused on data retrieval quality were different personas — and that a testing environment shouldn't be tied solely to prompt building, since retrievers are used across multiple areas. As a result, this approach was abandoned.
Workbench Integration
We considered placing the retriever playground within workbench, since that's where our batch testing environment already exists. However, when mapping out the user workflow, we discovered that most users prefer to deep dive into individual test cases before running batch tests. This meant a batch-first interface would be overwhelming for their primary use case. Additionally, workbench's separation from retriever builder would limit users' ability to fully configure retrievers. For these reasons, we decided against this approach.
As with any design project, more ideas were trashed than made it into the final designs. Here are some of the designs that I created, were thrown out, and why.
A visual of retriever playground in advanced mode from our MVP design.
Retriever Playground Framework
Retriever Playground framework follows the natural cycle of tweaking and testing. It was designed in panels for configuration, confirmation, and calculation.
Configuration
The Configuration Panel is your retriever's control center—pick which one you want to test, peek at its search index details, and fine-tune filter conditions to result counts and return fields. For more sophisticated retrievals, users can on advanced mode to unlock query expansions and rewrites for when basic just won't cut it.
Confirmation
The Confirmation Panel is where the testing magic happens. Here, users input their query and can generate an answer key using the grounding method to establish a baseline for measuring retriever accuracy. Once the test runs, users get a behind-the-scenes look at the retriever's step-by-step process on its way to the right documents.
Calculation
The Calculation Panel displays all the metrics generated from the test, giving users a clear view of performance and accuracy data. Users can also dive into the individual document chunks the retriever surfaced to answer their query—because sometimes you need to see the receipts.
An integration of retriever playground with Salesforce’s Agentforce Builder
Platform Integrations
Similar to retrievers builder, retriever playround’s strategic value lies in integration with cross-product experiences.
Strategically, Retriever Playground empowers users to test their retrievers before deployment across Salesforce's platform. The more confident users are about their retrievers, the more they'll use them, and the better their retrievers will perform—creating a flywheel effect. Here are some examples of how Retriever Playground is currently integrated into Salesforce's platform today.
Retriever Builder Integration - The Playground launches directly from Retriever Builder via a "Test Retriever" button, allowing users to test and tune no-code retrievers seamlessly
Agent Builder - One-click handoff from Agent Builder to Retriever Playground for debugging retrieval issues
Data Cloud - Integration for search indexes and data management
Agentforce - Integration for agent action support and retriever actions
Prompt Builder - Seamless retriever usage integration
Einstein Studio - Positioned as the primary retriever management hub
An exploration of agentic retriever testing
Future Feature Expansions
Retriever Playground is just getting started, and we've got plenty of exciting ideas brewing—pending user feedback and a quick reality check from engineering, of course.
More explorations of agentic retriever testing. Here users are asking the agent to help pick their test data.
Here are some ideas we have that we are excited to explore in the near future:
Comparison View: Ability to compare multiple retrievers side-by-side to determine the best one, with latency as the primary metric of interest
Dialogue based testing: Giving users the abilities to conduct retriever testing through mini UI actions with an emphasis on dialogue with an agent
Bulk Testing Integration: Seamless integration with Testing Centre for scale testing
On-Canvas AI Assistance: Agentforce within Retriever Playground for less technical users
Content Workbench Integration: Unified workflow combining content processing with retriever testing, including editing retriever and search index configurations directly
Unified Version History: Track changes to retriever configurations and content processing pipelines together
Combined Evaluation Dashboard: Show metrics like recall, tokens sent in one place
Prompt + Retriever Testing: End-to-end test combining prompt template with retriever output
Auto-Suggest Retriever Improvements: Use heuristics or AI to suggest config changes based on test performance
Custom SQL Retriever Support: SQL-based retriever building capabilities with parameterized queries
SentOS/ADL Integration: Binding ADL data sources to agents and supporting ADL uploaded files in ground truth evaluation