U-Tune
Learn to Fine-Tune the easy way
Team
- Hebah Haque
- Rocio Perez
- Prajas Kadepurkar (me)
Duration
9 months (Apr 2024 - Dec 2024)
My Role
- UX Design Lead
- Frontend Development
Problem
Fine-Tuning is inaccessible and comes with risks
Fine-tuning Generative AI models has immense potential to help users leverage these systems for personalized applications. However, it also introduces risks such as data privacy issues, inaccuracies, and ethical concerns. Non-technical users often struggle to understand these risks, making it difficult for them to fine-tune models safely and effectively. Without clear tools to guide users in understanding these challenges and making informed decisions, fine-tuning could lead to misuse or unintended consequences. The challenge is to design a solution that enables users to explore the possibilities of fine-tuning while educating them about the risks and promoting responsible use.
Defining Non-technical Users
Who benifits from fine-tuning and How?
Before going further, it's important to define non-technical users. These are people with little to no technical knowledge of generative AI and the fine-tuning process. They are often unaware of the risks associated with these systems and may also have a general mistrust of AI. However, they could still benefit greatly from using fine-tuning in their work or personal projects.
Rachel Thomas
Owner of an online home decor business
Goals
- Enhance customer support on her e-commerce site by creating a chatbot expert in her specific product line.
- Increase efficiency by reducing the time spent personally answering customer queries.
- Build trust with customers through accurate, reliable chatbot responses.
Pain Points
- General chatbot models lack the specialized knowledge required to answer detailed questions about her product line.
- Feels overwhelmed by the technical complexity of fine-tuning AI models.
- Concerned about customer data privacy and ensuring the chatbot doesn't provide incorrect information.
Alex Martinez
Content Creator for a fitness brand
Goals
- Generate social media content that aligns perfectly with their brand's voice and guidelines.
- Increase engagement on social media through high-quality, consistent posts.
- Save time while maintaining creative control over the content creation process.
Pain Points
- General AI models generate content that doesn't match their brand's tone or aesthetics.
- Struggles with understanding how to tweak AI models to reflect the nuances of their brand voice.
- Needs a way to experiment with AI outputs quickly without diving into overly technical tools.
Defining Success Metrics
Simplify Fine-Tuning and Build trust in Gen-AI
A successful product would make learning about fine-tuning accessible for non-technical users while also educating them about the risks involved at every step of the process. To measure this, we defined clear success metrics that align with these goals, ensuring the solution addresses both accessibility and responsible use.
User Comprehension of Fine-Tuning
Users demonstrate an improved understanding of fine-tuning concepts and generative AI.
User Comprehension of AI Risks
Users accurately identify key AI risks, including bias, hallucinations, and ethical considerations.
Approach
Learning from Experts, Competitors, and Users
Expert InterviewsN = 12 Experts
We began by conducting semi-structured interviews with experts in the fine-tuning space to learn about the details and challenges of fine-tuning Generative AI models.
Competitive AnalysisN = 6 Competitors
We analyzed existing tools to understand user pain points, challenges, and goals with fine-tuning and Generative AI. This helped us spot opportunities to design a better solution.
Thematic Analysis
Using thematic analysis, we combined insights from the interviews and competitive analysis to uncover key patterns and themes. We identified two key themes covering why fine-tuning is challenging for non-technical users and the core reason behind mistrust in AI.
The technical knowledge required, including vocabulary and basic concepts, for fine-tuning is intimidating for most users.
Users hesitate to interact with AI interfaces due to the lack of transparency regarding their data.
Solution
Introducing U-Tune
An educational tool designed to empower non-technical users by simplifying complex fine-tuning processes and generative AI concepts. The solution bridges the gap between technical knowledge and practical applications while emphasizing responsible AI use.
Key Feature #1
Simplyfying Hyperparameters
Adjusting hyperparameters can be hard for non-technical users because it requires understanding complex terms and how they affect the model's performance. Users often don't know how different settings work together, making it challenging to make the right choices.
Decision 1 - Using Simplified Technical Definitions
The design includes simplified technical definitions that clearly explain each hyperparameter and its role in the fine-tuning process.
Decision 2 - Accordion Style Layout
The layout is streamlined, with minimal, on-demand text presented in an accordion-style format to improve clarity and reduce cognitive load. This makes it easier for users to navigate the interface and understand the settings without feeling overwhelmed.
Past Iterations
We started by experimenting with analogies to explain the function of each hyperparameter. This idea came from an observation during a presentation. In the presentation, we used an analogy to explain pre-trained Gen AI models and fine-tuning, and it made these concepts much easier for everyone to understand. Based on this, we decided to use analogies to help users understand hyperparameters more easily.
However, user feedback indicated that the analogies were too abstract, making it difficult for users to understand the connection to the actual parameters. Users also highlighted that the layout and amount of text on the screen were overwhelming.
Key Feature #2
AI Nutrition Facts
Users don't trust AI due to a lack of transparency and understanding of how AI systems handle data and make decisions. To promote trust and transparency, we introduced AI Nutrition Facts, a feature inspired by Twilio. The label provides users with key information such as: How data is collected or How long it is stored.
Decision 1 - How to Read the Label
To make the nutrition information more accessible, the design includes a dedicated "How to Read the Label" section. This section provides clear descriptions for each parameter on the label, along with explanations of what different values mean. By breaking down the details in an easy-to-understand way, the feature empowers users to interpret the information effectively and make informed decisions about the models they use.
Decision 2 - Easy Access
The AI Nutrition Facts are placed within the Model Details Page to provide better visibility and context. This ensures that users can easily access and understand important information about the model while exploring its details, making the feature more intuitive.
Past Iterations
The initial iteration of this feature allowed users to hover over a model card, revealing a pop-out nutritional info card. This provided quick and easy access to important details, allowing users to view the nutritional information without leaving the main interface.
However, usability sessions revealed that users struggled to understand how to interpret the information on the label.
Results
A definite step towards simplyfying fine-tuning
The implementation of U-Tune's key features - simplified technical information through easy hyperparameter settings, and informed decision making through Nutritional Labels - resulted in significant improvements in user experience and learning outcomes, addressing the core challenges of simplifying fine-tuning and fostering trust in AI systems.
at the end of the day i feel like I learned a new skill, better understanding of ai as a whole
love the ability to actually play around and use your own imagination
Impact on success metrics
To evaluate the impact of the designed solution, we conducted pre- and post-usability assessments with users. These assessments focused on understanding changes in their comprehension of fine-tuning and their trust in AI systems. Users were asked questions like:
"On a scale from 1 to 5, how would you rate your understanding of fine-tuning?"
"How would you rate your understanding of how your data is used by AI systems?"
Retrospective
Designing for an Unfamiliar Domain
Starting out with very limited domain knowledge, I learned how to effectively research and design for an unfamiliar domain. This experience reinforced the importance of adaptability and the ability to quickly understand complex technical topics—a skill that is invaluable for product designers working across diverse fields.
Retrospective
Bridging Design and Development
This was my first time fully assuming the role of a UX engineer. I learned how to design with tools in a way that makes development easier, ensuring a smoother handoff process. Additionally, I gained hands-on experience in translating designs into well-functioning code, bridging the gap between design intent and technical implementation.