The Hangzhou Asian Games
Overview
As a product designer of the conversational AI assistant app Adot
I designed a special feature to deliver accurate, timely answers and dedicated in-app pages in response to user queries about the Games. This project was the company’s first to simultaneously utilize an LLM Large Language Model and grounding* techniques, and it was recognized for its successful outcomes.
* Grounding refers to the process of linking or aligning an AI system's outputs, representations, or actions to real-world data to ensure they are accurate and relevant.
I designed a special feature to deliver accurate, timely answers and dedicated in-app pages in response to user queries about the Games. This project was the company’s first to simultaneously utilize an LLM Large Language Model and grounding* techniques, and it was recognized for its successful outcomes.
* Grounding refers to the process of linking or aligning an AI system's outputs, representations, or actions to real-world data to ensure they are accurate and relevant.
Adot is a conversational AI service developed by SK Telecom
It serves as a smart virtual assistant capable of understanding natural language queries, providing accurate information, and supporting daily tasks such as scheduling, reminders, and recommendations.
Adot is designed to continuously learn and improve through user interactions, offering seamless integration with other SK Telecom services and third-party platforms.
Product Design
Project Management
Collaborated with back-end and front-end developers, UX/UI designers and operations staff
2023. 07 - 2023. 10
Figma, Confluence, Jira, Slack
Goal
To address the hallucination issue of LLMs and provide users with accurate and real-time information about the Hangzhou Asian Games
Process
1 Problem Definition & Solution
Adot has been designed to respond to a wide range of user queries in daily life, particularly by addressing high-interest events like elections or the Olympics. However, the team avoided relying solely on LLMs, as they often produced irrelevant or inaccurate answers to user questions. Instead, they employed a scenario-based approach, manually categorizing user query types and matching them with pre-defined responses, ensuring the same fixed answers were delivered consistently.
I believed that, as an AI service, Adot should leverage LLMs to their fullest potential to expand its ability to handle diverse user inputs. My goal was to move away from a chatbot-like experience with rigid, pre-scripted scenarios and instead provide users with more dynamic and natural conversations.
To achieve this, I explored ways to integrate LLMs into the response generation process while ensuring the use of accurate and reliable data. I reached out to the development team to share this situation and was able to collaborate with a team researching grounding technology. After reviewing the idea from both technical and user experience perspectives, we were able to initiate a project to integrate LLMs into Adot for the first time with a service dedicated to the Hangzhou Asian Games.
2 Dynamic Response Framework Development
The required data for responses varies depending on user utterance types and intents. To ensure accurate response generation, I developed a new type of framework that linked LLMs to appropriate data sources by mapping specific data to each utterance type. Unlike the previous scenario system, which required manually inputting user utterances and corresponding responses, this approach focused on directly connecting utterance types with data.
User utterance intents
- Check match schedules
- Check current national rankings
- Check medal counts
- Check match results
- View highlights of the match
- Read news articles about the match
...
Data types
- Match date
- Sport
- Score
- Country
- Medal
- Ranking
...
The developers built a database structure aligned with this framework, trained the LLM, and ensured it was correctly linked to the intended documents in the database. The overall grounding system architecture of the project was as follows.
3 Visual Interface Planning
Although Adot is a conversational AI, relying solely on dialogue can sometimes result in a less convenient user experience. To address this, we incorporated some visual elements to complement the interaction. For the Asian Games project, we decided to provide a feature screen as well, recognizing that delivering a visual interface would be more suitable for users who want to view multiple pieces of information at once.
I decided to provide two types of interfaces, planned their structure and functionality, and collaborated with the GUI designer to finalize the design. Subsequently, I created a User Interface Design document (UID) that detailed the screen layout, components, and interactions, and collaborated with front-end developers to successfully implement the interface.
01
Within the chat interface, we provided basic schedule information and South Korea's ranking related to the Hangzhou Asian Games alongside responses to user queries. Additionally, an interactive component was included, allowing users to click and navigate to the dedicated feature screen. This ensured that users interested in further exploring Asian Games-related content could do so easily and seamlessly.
02
We aimed to provide a simple and intuitive interface where users could easily access detailed information about the Asian Games, including match schedules and results organized by date and sport, as well as the current medal standings.
4 Operational Tool Design for Real-Time Data Input
To provide accurate and fast real-time match data for the Asian Games, we decided to monitor matches directly and input the data into the database. A team of 15 operators was planned for this task, and it was essential to ensure that they could input data quickly and easily. To achieve this, I designed a structure optimized for the operational tool and collaborated with developers to complete its implementation.
5 Overall Management of Real-Time Data Input Operations
After the project was thoroughly prepared through testing and QA, I created a detailed guide on inputting real-time match data and provided it to the operations staff. Once the project was deployed to the app and the Games began, I continued to oversee the operation to ensure everything was running smoothly. In cases where issues arose with the operational tool or incorrect data was entered, I took the lead in communicating promptly to resolve the problems efficiently.
Outcomes
During the short two-week service period, a total of 37,000+ user utterances were recorded.
Among these, 89.69% were accurate responses rather than fallback or "I don't know" replies, making it a successful outcome for the first attempt at implementing LLM grounding.
Examples of conversations where user utterances were accurately analyzed, searched, and grounded :
우리나라 지금까지 아시안게임 메달 몇 개 땄어?
How many medals has our country won so far in the Asian Games?
아시안게임 메달 순위를 다음과 같이 보여드릴게요.
- 대한민국: 금메달 33개, 은메달 47개, 동메달 77개 (총 157개, 3위)
Here are the current medal standings for the Asian Games:
- South Korea: 33 gold medals, 47 silver medals, 77 bronze medals (157 in total, ranked 3rd)
이번 항저우 아시안게임 8강 축구 언제 해?
When is the quarterfinal soccer match in the Hangzhou Asian Games?
아시안게임 축구 경기일정을 알려드릴게요. - 축구 여자 8강 (9/30 17:30) - 대한민국 대 북한 - 축구 남자 8강 (10/1 21:00) - 대한민국 대 중국
Here is the soccer match schedule for the Asian Games:
- Women's Quarterfinal (Sept 30, 5:30 PM): South Korea vs. North Korea
- Men's Quarterfinal (Oct 1, 9:00 PM): South Korea vs. China
아시안게임 여자골프 임지유 어떻게 됐어?
How did Lim Ji-yoo do in women's golf at the Asian Games?
아시안게임 여자골프 경기에서 임지유 선수는 -2(142)의 기록으로 공동 21위로 컷탈락을 했습니다. 따라서, 3,4라운드에 출전하지 못하였습니다.
In the women's golf event at the Asian Games, Lim Ji-yoo scored -2 (142) and finished tied for 21st, missing the cut. As a result, she did not advance to the 3rd and 4th rounds.
Takeaways
- Providing ambiguous "I don't know" responses can lead to poor user experiences, making it essential to design fallback mechanisms carefully, such as prompting similar utterances or refining intent detection. While only 6% of responses fell into this category, for the users who experienced failures, it could feel like 100%. Interestingly, well-prepared fallback messages often provided more satisfying experiences by addressing user needs effectively. In the future, I plan to further refine and enhance this aspect of the system's design.
-
The grounding performance was not yet highly advanced. However, by working closely with developers during this short-term project, we adapted quickly by restructuring the system and revising prompts to overcome challenges, achieving a response success rate of nearly 90%. The lessons learned from this project were highly valuable for future attempts to expand grounding to a broader scope within the app. With further technical refinements, the system's overall accuracy and reliability can be enhanced. On a personal note, I experienced the importance of responding swiftly to critical situations and maintaining transparent communication with collaborators. This project also allowed me to develop and strengthen my communication skills in a dynamic and high-pressure environment.
- I was able to develop screen planning skills and communication skills essential for their implementation.