Lessons Learned While Embedding GenAI in a Product

In this blog, I want to collect the lessons I have learned whenever trying to somehow embed generative AI into any product. I am also planning to share your experience and references. Please comment here or send me a message on LinkedIn.

Disclamer

When you are reading this article, some of this learning might seem obsulute or so obvious, but for beginners or past-us, it was something to be learned!

Intro

There are many different ways to embed GenAI in a product or application. It can be as famous as a chat user interface or just a normal button, but behind the scenes, sending some data to a large model and reusing the output in complex ways, then as an outcome (response), just showing normal data in a table, chart, or just a simple change in the UI.

In most of the use cases, there are some common or similar practices. So from one use case to another, we can build on top of our learnings, and sharing learned lessons can help us to grow faster.

Lessons

July 12th, 2024, lessons from an internal project using SAP AI Core, thanks to Mathis and Sherri for their mentorship and support.
- Use the output model for enforcing output structure (e.g., JSON) and add a description to each field.
  - Using Instructor: Structured LLM Outputs
  - More info in this Medium blog post
  - Note: Based on this it seems BAML is a better way to go
- Minimize the requested output as much as possible to get the response faster and cheaper.
- Set the scene with roles and use system, user, and assistance roles when it makes sense.
  - system role's prompt is more restricted than user and is usually used to set the scene.
- Minimalistic prompt: try to define prompt as minimal as possible.
  - Don't confuse the model with unnecessary additional info.
- Use a structured prompt. Markdown, XML, or even JSON!
- Don't give much freedom to the user if they don't know the underlying code; the functionality can be easily broken, and when it 50% doesn't work, the user gets frustrated!
  - No setting high expectations for the user
- A large output usually has more hallucinations, so we can't rely on the whole output size.

Similar Blogs or Other Learning References

Reference

Cover image generated by Canva

Lessons Learned While Embedding GenAI in a Product

Comments

More from this blog

Is "Deepfake" Just the Forgotten Name of Generative AI?

What does the retrieval ecosystem look like in the age of agents?

Never Built a Mobile App? Now We Can!

Does Knowledge Graphs Still Matter?

My GenAI Development Learning Resources

Disclamer

Intro

Lessons

Similar Blogs or Other Learning References

Reference

Command Palette

Comments

More from this blog

Disclamer

Intro

Lessons

Similar Blogs or Other Learning References

Reference