SKT
2023
Making AI Conversations Natural and Reliable
Initiating and designing Adot’s first grounded LLM experience to reduce hallucinations and deliver natural conversations based on verified, real-time data during the Asian Games
Role
Product Designer
Timeline
Jul 2023 - Oct 2023
Team
Product Manager
Engineers
Operations Staff
Tools
Figma
Internal user data platform
Jira
Confluence
Overview
Adot could sound natural or stay factual. I set out to make it do both.
Adot is SKT's AI Assistant app, used by 3M+ users each month. A fully LLM-powered experience could support natural conversation, but frequent hallucinations made it too risky for topics that required factual, real-time information. To ensure accuracy, Adot relied on scenario-based conversations written and maintained by product designers. The information was reliable, but the responses were rigid and repetitive.
When I took on the Asian Games, I saw an opportunity to overcome this tradeoff. I approached a team researching grounding technology and initiated Adot’s first project combining an LLM with verified, real-time data. This allowed Adot to generate responses naturally while keeping the underlying information factual. I designed the grounding framework, user-facing experience, and operational tool in close collaboration with engineering, design, and operations teams.
Opportunity
Every question could be phrased differently. Adot’s answer was always the same.
For major events such as elections and the Olympics, product designers anticipated user intents, mapped out conversation branches, and wrote and maintained every response. Dynamic fields such as dates, scores, and rankings allowed the latest data to be inserted, but the structure and language of each response remained fixed.
This approach delivered reliable information, but Adot could only respond within flows that had already been designed.
When I was assigned the Asian Games as my first major project, I saw an opportunity to create an experience that could understand the many ways users naturally ask questions and respond with language shaped by their intent and context.
Initiative
I took the idea to a team researching grounding, and together we turned it into a live product.
At the time, grounding* was still being explored internally and had not yet been applied to an Adot service. I reached out to the team, shared the limitations of the existing scripted approach, and proposed using the Asian Games as an opportunity to test a new conversational experience.
We worked together to evaluate the idea from both technical and user experience perspectives. I defined the types of questions users might ask and the information each answer would require, while the engineers explored how an LLM could retrieve and use verified data. This collaboration became Adot’s first live service to combine an LLM with grounding technology.
* Grounding: A technique that connects an LLM to verified, up-to-date data so it can generate responses based on facts rather than relying solely on what the model already knows.
Solution
I designed the full system behind each answer, from grounding logic to real-time operations.
A natural conversation required more than connecting an LLM to a database. Adot needed to understand what users were asking, retrieve the right information, present it in a useful format, and keep it accurate as events unfolded in real time.
I designed the solution as three connected parts: 1) a grounding framework, 2) conversational and visual interfaces, 3) an operational tool.
Grounding Framework
Links user intents to verified data
I designed a framework that mapped each user intent to the verified data required for an accurate response, allowing the LLM to generate answers dynamically instead of selecting from prewritten scripts.
Mapping user intents to required data
Each type of question required a different combination of information. A schedule question needed the sport, date, time, and competing teams, while a medal question required the country, medal counts, and ranking.
I categorized the key intents behind Asian Games queries and defined the data fields required to answer each one.
Turning the framework into a grounding system
The engineers used this framework to structure the database and connect each intent to the appropriate documents and data sources. When a user asked a question, the LLM identified the intent, retrieved the relevant data, and generated a response based on that information.
Conversational & Visual Interfaces
Makes answers easy to understand
I paired grounded conversational responses with scannable visual interfaces, allowing users to get a quick answer in chat or explore schedules, results, and rankings in more detail.
A conversational response worked well for answering a specific question, but it was less effective when users wanted to scan multiple events or compare information at a glance. I designed two connected interfaces so users could move naturally between asking and browsing.
Operational Tool
Keeps live match data accurate and up to date
Grounded responses were only as reliable as the data behind them. To provide accurate schedules, scores, and rankings as events unfolded, a team of 15 operators monitored the Games and entered updates directly into the database.
I designed the structure and workflow of the operational tool so the team could enter and update match data quickly and consistently. I then worked with engineers to translate the requirements into a functioning internal tool.
Before launch, I created a detailed data-entry guide and prepared the operations team for live use. During the Games, I oversaw the workflow and coordinated with operators and engineers to resolve tool issues and correct inaccurate data as they occurred.
Outcomes
Over two weeks, Adot answered nearly 90% of 37,000+ queries without falling back.
During the Asian Games, users asked about medal counts, match schedules, national rankings, and individual athletes in a wide range of natural expressions. The service processed more than 37,000 queries during its two-week run.
Of those queries, 89.67% received a grounded response instead of a fallback or “I don’t know” message. As Adot’s first live service combining an LLM with grounding, the result showed that the approach could support natural conversations while providing reliable, real-time information.
Real examples of grounded conversations
Users could ask questions in their own words, while Adot identified the intent, retrieved the relevant data, and generated a response grounded in that information.
Takeaway
Nearly 90% of queries succeeded. The remaining failures mattered just as much.
Most users received a grounded response, but those who encountered a fallback experienced a complete break in the conversation. The overall success rate meant little in that moment. I learned that fallback responses are not secondary error states. They are a core part of the conversational experience.
A better fallback should help users recover by clarifying what the system understood, suggesting related questions, or guiding them toward an available answer instead of simply saying, “I don’t know.”
Grounding was still an emerging technology, so response quality depended on continuous adjustments to the data structure, intent mapping, and prompts. Working closely with engineers taught me how to design around technical uncertainty, respond quickly when issues appeared, and communicate clearly across design, development, and operations.
