Retriever Playground

Salesforce

June 2023 - Current (Generally Available)

Design Systems, User Research, Experience Design, Information Architecture, Design Strategy

Retriever Playground is a comprehensive testing and optimization environment for retrievers.

I led the design for a testing tool that helps users validate and optimize retrievers before using them in RAG pipelines, delivering a complete 6-month design roadmap. Through this work, I introduced a testing cycle framework (Configure, Test, Analyze) that became valuable enough to share with other testing teams across the organization. By facilitating cross-functional working sessions with product and engineering, I helped clarify what information was truly critical in the testing process—reframing roadmap discussions and simplifying the mental model for users who previously thought testing retrievers required a PhD in patience.

To comply with my non-disclosure agreement, I have omitted and obfuscated confidential information in this case study. All information in this case study is my own and does not necessarily reflect the views of Salesforce.

“My personal anecdotes and thoughts will be in quotes and in blue”


Tools

  • Figma (Design, FigJam, Make)

  • Cursor AI

Skills

  • User Research

  • Information Architecture

  • Design Strategy

  • Experience Design

  • Design Systems

  • Communication and Presentation Skills

Team

  • Som Liengtiraphan (Me, Senior Product Designer)

  • Barnali Patnaik (Senior Product Manager)

  • Anubha Dubey (Design Manager)

Clickable prototype of our MVP retriever playground in advanced mode. The password is the same as this page’s.

Impact & Contribution

Lead designer for retriever playground, owning end-to-end UX design from initial concept through GA release and facilitating integrations with various usage points in the Salesforce platform.

Key Deliverables & Outcomes

Shipped to Production

  • Retriever Playground reached GA- February 2026

Designs

  • Created 6-month design roadmap for Retriever Playground UI within Einstein Studio

  • Designed Testing workflows for debugging existing retrievers

  • Designed Validation and fine-tuning capabilities for custom retrievers

  • Enabled pre-deployment validation reducing irrelevant results in production

  • Created a seamless workflow from raw content to refined context and validated retrieval

  • Established foundation for enterprise-grade RAG success with reliable, explainable, and high-quality retrieval

The design process for this project and the tools used in each step

Project Context

The Retriever Playground was created to address critical gaps in the retrieval testing workflow following feedback from interviews and heuristic analysis.

Findings

  • Persona Mismatch: Previously, retrieval testing was happening inside Prompt Builder, which didn't align with the data persona who typically sets up Retrievers—not the prompt builder persona who writes prompts

  • Lack of Necessary Metrics: Prompt Builder lacked the specific metrics needed for retrieval evaluation, such as recall scores and chunk analysis

  • Broader Use Cases: Retrievers are used in more places than just prompts—including Flow, apps, and ADL—requiring a dedicated testing environment

  • Pre-Production Validation: Users needed a safe space to fine-tune, optimize, validate, and debug RAG system configurations before deploying to production

Our testing process before retriever playground

Above: Previously, this was the only way to test retrievers. It was roundabout with no retrieval metrics, no ability to view all the documents retrieved, and no way to fine tune configurations to optimize retrieval.

Without a place to test retriever configurations before deployment, costly errors might occur, leading to longer development cycles, retrieval errors, and ultimately lost opportunities and revenue.

The Problem

How might we enable users to efficiently test, debug, and optimize retriever configurations before deploying them in production RAG pipelines?

The Users

There are 3 main categories of users custom retriever builder: Enterprise Customers, Data Scientists, and Product Managers.

Platform-wide process to create and use retrievers in Salesforce

Scope and Constraints

This project's scope covered developing features, identifying integration touchpoints, and delivering the MVP. The primary constraint were technical complexities and platform integration.

Scope

The Retriever Playground MVP delivered essential features across three user personas. Enterprise Customers (Admins/Support) got ground truth selection, recall scoring, and configuration adjustments for debugging retrieval issues. AI Researchers/Data Scientists received advanced mode with rewritten queries, chunk score debugging (vector and keyword), and performance monitoring. Product Managers could save/activate workflows, validate pre-deployment, and reduce irrelevant results. Core P0 functions covered retriever selection, search index display, configurable settings, filtering, query input, ground truth spot checks, recall scoring, intermediate outputs, and save-and-activate workflow. Bottom line: a safe sandbox to test retriever configurations against real data before production—because nobody wants to play "guess and check" with their RAG pipeline!

Constraints

The timeline was aggressive: design kicked off May-June 2025, MVP approval at PM Summit in September 2025, with a targeted launch in the 258-260 release window (October-December 2025). The geographically distributed team across design, PM, and engineering made real-time collaboration tricky, and engineering bandwidth constraints—juggling DC1 and other initiatives—delayed some deliverables. Phase 1 technically couldn't support advanced mode or enriched index features (those needed additional AI gateway logic), limiting the MVP to individual, active, basic configuration retrievers with Data Cloud sources. Despite all that, the team delivered a foundational testing environment that tackled the critical persona mismatch and surfaced the retrieval metrics that mattered.

“My little graveyard of discard designs. We spent around 2 weeks toying with integrations in other products within Salesforce before designing that it’s best if we create a custom UI”

Experience Maps

We started with experience maps to help plan the design and keep everyone aligned on what we were building.

Above: The workflow for our MVP retriever playground

Given our geographical spread and time-zone differences, alignment became our biggest challenge—turns out, herding cats across three continents requires more than just a really good calendar. Journey maps became our North Star, keeping everyone focused on what we were building and which features would ship in each release. Throughout the project, I maintained both an overall experience map and individual release maps to keep us on track.

The designs that didn’t make it

Prompt Builder Integration

We initially explored adding a playground environment within Prompt Builder, where users were already attempting to test their retrievers. However, it quickly became clear that the people writing prompts and those focused on data retrieval quality were different personas — and that a testing environment shouldn't be tied solely to prompt building, since retrievers are used across multiple areas. As a result, this approach was abandoned.

Workbench Integration

We considered placing the retriever playground within workbench, since that's where our batch testing environment already exists. However, when mapping out the user workflow, we discovered that most users prefer to deep dive into individual test cases before running batch tests. This meant a batch-first interface would be overwhelming for their primary use case. Additionally, workbench's separation from retriever builder would limit users' ability to fully configure retrievers. For these reasons, we decided against this approach.

As with any design project, more ideas were trashed than made it into the final designs. Here are some of the designs that I created, were thrown out, and why.

A visual of retriever playground in advanced mode from our MVP design.

Retriever Playground Framework

Retriever Playground framework follows the natural cycle of tweaking and testing. It was designed in panels for configuration, confirmation, and calculation.

Configuration

The Configuration Panel is your retriever's control center—pick which one you want to test, peek at its search index details, and fine-tune filter conditions to result counts and return fields. For more sophisticated retrievals, users can on advanced mode to unlock query expansions and rewrites for when basic just won't cut it.

Confirmation

The Confirmation Panel is where the testing magic happens. Here, users input their query and can generate an answer key using the grounding method to establish a baseline for measuring retriever accuracy. Once the test runs, users get a behind-the-scenes look at the retriever's step-by-step process on its way to the right documents.

Calculation

The Calculation Panel displays all the metrics generated from the test, giving users a clear view of performance and accuracy data. Users can also dive into the individual document chunks the retriever surfaced to answer their query—because sometimes you need to see the receipts.

An integration of retriever playground with Salesforce’s Agentforce Builder

Platform Integrations

Similar to retrievers builder, retriever playround’s strategic value lies in integration with cross-product experiences.

Strategically, Retriever Playground empowers users to test their retrievers before deployment across Salesforce's platform. The more confident users are about their retrievers, the more they'll use them, and the better their retrievers will perform—creating a flywheel effect. Here are some examples of how Retriever Playground is currently integrated into Salesforce's platform today.

  1. Retriever Builder Integration - The Playground launches directly from Retriever Builder via a "Test Retriever" button, allowing users to test and tune no-code retrievers seamlessly

  2. Agent Builder - One-click handoff from Agent Builder to Retriever Playground for debugging retrieval issues

  3. Data Cloud - Integration for search indexes and data management

  4. Agentforce - Integration for agent action support and retriever actions

  5. Prompt Builder - Seamless retriever usage integration

  6. Einstein Studio - Positioned as the primary retriever management hub

An exploration of agentic retriever testing

Future Feature Expansions

Retriever Playground is just getting started, and we've got plenty of exciting ideas brewing—pending user feedback and a quick reality check from engineering, of course.

More explorations of agentic retriever testing. Here users are asking the agent to help pick their test data.

Here are some ideas we have that we are excited to explore in the near future:

  1. Comparison View: Ability to compare multiple retrievers side-by-side to determine the best one, with latency as the primary metric of interest

  2. Dialogue based testing: Giving users the abilities to conduct retriever testing through mini UI actions with an emphasis on dialogue with an agent

  3. Bulk Testing Integration: Seamless integration with Testing Centre for scale testing

  4. On-Canvas AI Assistance: Agentforce within Retriever Playground for less technical users

  5. Content Workbench Integration: Unified workflow combining content processing with retriever testing, including editing retriever and search index configurations directly

  6. Unified Version History: Track changes to retriever configurations and content processing pipelines together

  7. Combined Evaluation Dashboard: Show metrics like recall, tokens sent in one place

  8. Prompt + Retriever Testing: End-to-end test combining prompt template with retriever output

  9. Auto-Suggest Retriever Improvements: Use heuristics or AI to suggest config changes based on test performance

  10. Custom SQL Retriever Support: SQL-based retriever building capabilities with parameterized queries

  11. SentOS/ADL Integration: Binding ADL data sources to agents and supporting ADL uploaded files in ground truth evaluation