Operationalizing Gen AI on Vertex AI using MLOps

The transcription discusses the challenges and opportunities of operationalizing generative AI on Vertex AI for asset management. It emphasizes the adaptability and versatility of generative AI models, which can handle various tasks instead of separate models for each task. The white paper provides guidance on choosing the right model based on specific needs, considering factors such as quality, latency, development time, cost, and compliance. Prompt engineering is highlighted as a way to interact with the model and steer it towards desired outputs. Chaining and augmenting models is also discussed, where multiple models work together to achieve a common goal. The importance of responsible use and ethical considerations in asset management is acknowledged. Vertex Model Garden is introduced as a collection of pre-trained models, making it easier to find the right starting point. Vertex AI Studio is described as an integrated development environment for experimenting with models and build All right, so let's jump right into it. We've got a pretty fascinating topic today. Yeah, definitely. Looks like we're diving deep into operationalizing generative AI on Vertex AI. Exactly. You know, it seems like everyone's talking about Gen AI these days, but bringing it to life, especially in asset management, that's a whole other ballgame. It is, and this white paper we've got, it's by some serious Google AI experts. They really break down the nuts and bolts of how to make it happen. They even highlight some of the unique challenges that pop up with Gen AI. What are those challenges like? Well, for starters, these models can be massive, huge in terms of computational power needed, think managing, training, deploying, all of that. And then there's the data side of things. Gen AI is incredibly data hungry, so ensuring data quality and governance, that's absolutely crucial, especially for a product team and a large asset management company. That makes sense. Yeah. So this deep dive is all about understanding those challenges and figuring out how Vertex AI can help us tackle them head on. You got it. We'll be looking at everything from picking the right model to fine tuning it, keeping it accurate with real world data, and making sure it doesn't go rogue in production. Sounds like you've got a lot of ground to cover. But before we get lost in the weeds, let's start with the basics. What exactly makes Gen AI so different from the models we might already be using in asset management? The short answer, adaptability. The models you're probably using now, they're most likely designed for a very specific task, like predicting stock prices or analyzing market trends. Exactly. But Gen AI models, especially these new foundational models, they're built to adapt to all sorts of tasks. It's this versatility that makes them so powerful. So instead of having a separate model for each specific task, we could potentially have one model that can handle a whole range of functions. You're getting it. And that's a game changer for asset management. Imagine a single model that can analyze market trends, generate investment reports, and even interact with clients in a natural way. That's the kind of potential we're talking about. I see what you mean. That level of flexibility would be a huge advantage. But the white paper also stresses that there's no one-size-fits-all model. With so many options out there, both open source and proprietary, how do we even begin to choose the right one? That's where the real work begins. And it all starts with understanding your specific needs. What kind of tasks do you need the model to perform? What level of accuracy are you aiming for? What are your latency requirements? How important is cost effectiveness? All critical questions, especially for a large asset management company where efficiency and cost savings are always top of mind. And of course, compliance is non-negotiable in our industry. Absolutely. So you need to factor all of that in when choosing a model. Thankfully, the white paper provides some excellent guidance. They lay out five key considerations, quality, latency, development time, cost, and compliance. And a handy checklist for any product development team looking to navigate the world of Gen AI. So let's say we've gone through that checklist and we've narrowed down our choices. What's next? How do we actually start working with these models? This is where it gets really interesting. You need to learn how to talk to them. Think of it like giving instructions to a really smart but slightly unpredictable assistant. That's where prompt engineering comes in. Prompt engineering. Sounds intriguing. Is that like writing code? It's similar, but not quite the same. Think of it like writing instructions for the model to follow. You're crafting the input to get the output you want. Okay, so it's not just about the data we train the model on. It's how we interact with it at runtime. That's a big shift from how we usually think about machine learning. Exactly. And the white paper really emphasizes this. They say the prompted model component becomes the core unit of work in Gen AI MLOs. It's a whole new way of thinking about development and management. So we're not just building models anymore. We're building systems that include the model plus all the ways we interact with it. Precisely. And that has major implications for how we approach everything from testing to monitoring to deployment. We'll dive deeper into that in the next part. Sounds like we've got a lot more to unpack. But before we move on, can you give us a concrete example of how prompt engineering might work in the context of asset management? Sure. Imagine you're trying to build a system that can help your analysts quickly assess the potential impact of a news event on a particular stock. A real-time risk assessment tool. That would be incredibly valuable. Right. So instead of just feeding the model the raw news article, you could craft a prompt that asks the model to identify key entities and events in the article, assess the sentiment towards those entities, and then predict the likelihood of a positive or negative impact on the stock price. So we're guiding the model's analysis with specific instructions tailored to our needs. Exactly. And that's the power of prompt engineering. You're not just relying on the model's general knowledge. You're steering it towards the insights that are most relevant to your business. So prompt engineering is all about framing the right questions to get the answers we need. But imagine that's just the tip of the iceberg when it comes to working with these models. What else do our product teams need to be thinking about? Well, one thing that comes up again and again in the white paper is the idea of chaining and augmenting models. Chaining and augmenting, what does that mean exactly? Think of it like this. You're trying to build a complex system in asset management, something that involves multiple steps and different types of data. You're not going to accomplish all of that with just one model. So you need to chain together multiple models, each specialized for a particular task. And you might even need to augment those models with other tools like APIs or databases. So instead of relying on a single, all-powerful model, we're building a more modular system where different components work together to achieve a common goal. Exactly. And that's another key shift in Gen AI ML Ops. You're often developing the whole chain as a unit, not just optimizing individual components. That must have huge implications for how we approach testing and monitoring, right? Absolutely. But the payoff can be enormous. Imagine automating that entire process we were talking about earlier, pulling data, analyzing it, generating reports, making investment recommendations, all driven by this interconnected system of models and tools. That's the power of Gen AI on Vertex AI. That's an incredibly compelling vision. But before we get carried away with all the possibilities, there's one crucial question I have to ask. How do we make sure these powerful systems are used responsibly, especially in a field like asset management where trust and ethical considerations are paramount? You hit the nail on the head. That's a question that's top of mind for everyone involved in Gen AI, and it's something we'll be exploring in depth in the next part of our deep dive. Looking forward to it. See you then. Welcome back. So in part one, we talked a lot about what makes Gen AI so different and some of the key things product teams need to be thinking about. Now let's actually get our hands dirty and see how Vertex AI helps us put all of this into practice. I'm all for getting practical. So we briefly touched on Vertex Model Garden earlier. Yeah. Can you walk us through how that actually works? Like if I'm a product manager at an asset management company and I'm looking for the right model for my team, where do I even start? Model Garden. It's like this curated collection of pre-trained models all ready to go on Vertex AI. It takes a lot of guesswork out of finding the right starting point. It's like a one-stop shop for Gen AI models. Exactly. You've got Google's own foundational models, like Gemini and popular open source ones too, like Llama 2, all in one place. That sounds way less daunting than scouring the internet trying to figure out which model is right for our specific use case. Right. And it's not just about having access to these models. It's about the information you get with them. Each model comes with this detailed model card. It's like a little cheat sheet that tells you everything you need to know, what it's good at, potential use cases, even guidance on fine-tuning and deployment. It's like having a little expert advisor for each model guiding you along the way. Yeah, exactly. It's incredibly helpful, especially for teams that are still getting up to speed on Gen AI. Okay, so let's say I found a model that looks promising for, say, portfolio risk analysis. What's next? How do I actually start working with it? That's where Vertex AI Studio comes in. It's your integrated development environment for all things Gen AI. Okay, so it's like the workshop where we can experiment with these models and actually start building something. Precisely. You can test different prompts, compare outputs from various models, even explore those fine-tuning techniques we were discussing. And I can do all this without needing to be a coding wizard. That's the beauty of it. Studio has this really user-friendly interface. So even if your team is still learning the ropes of AI, they can jump right in and start experimenting. So it's like a playground for Gen AI, where our product teams can get creative and see what's possible. I like that analogy. It really is a space for exploration and discovery. Speaking of exploration, we talked a lot about fine-tuning in part one. Can you remind us how Vertex AI supports that crucial process? Of course. Fine-tuning, it's all about taking a pre-trained model and tweaking it to perform even better on a specific task. Vertex AI gives you a bunch of tools to do this, from prompt engineering to supervised fine-tuning to even reinforcement learning with human feedback. Okay, that's a lot to unpack. Let's start with supervised fine-tuning. How would you explain that to someone who's not an AI expert? Imagine you're teaching a kid how to ride a bike. You wouldn't just throw them on and hope for the best, right? You'd guide them, show them how to balance, how to pedal. Supervised fine-tuning is kind of like that. You're giving the model these labeled examples showing it exactly what you want it to do. And it learns from those examples, adapts its behavior accordingly. So we're essentially holding the model's hand as it learns to perform this specific task. Yeah, that's a good way to put it. And the great thing is you don't need mountains of data for supervised fine-tuning to be effective. Even a few hundred well-chosen examples can make a big difference. Okay, that's reassuring, especially for teams that might be working with limited data sets. Now, what about reinforcement learning with human feedback, RLHF? That one sounded a bit more complex. It is a bit more involved, but it's incredibly powerful. Imagine you have this team of experts who are really good at, say, picking stocks. RLHF is like training a model to predict what those experts would choose based on their feedback. It's a way of baking human judgment and expertise directly into the model. So instead of just relying on historical data, we're actually incorporating the intuition and experience of our best people. That seems particularly useful for tasks that involve a lot of subjectivity, like portfolio management. You got it, RLHF. It's perfect for those situations where there's no single right answer, where human intuition really matters. It's like having a digital apprentice, learning from the masters. Exactly. You're passing down that hard-earned wisdom. Okay, so we've got supervised fine-tuning RLHF, and of course, prompt engineering all is part of our fine-tuning toolkit. What about that other technique we talked about earlier, distillation? Ah, yes, distillation. So remember how we talked about these foundational models being massive, like really computationally expensive? Yeah, I'm starting to get a sense of that. Well, distillation, it's all about creating a more efficient version of a large model. It's like taking all that knowledge and expertise packed into a giant model and distilling it into something more manageable. So it's like creating a mini-me version of the model that can still perform well, but without all the computational overhead. Exactly, and that's crucial for making Gen AI practical for real-world applications, especially on resource-constrained devices or platforms. Like, say, a mobile app for our clients. Exactly. Suddenly, you're not limited by the size of the model. You can bring the power of Gen AI to all sorts of new places. It sounds like Vertex AI has really thought of everything when it comes to fine-tuning, offering a range of options to suit different needs and resource constraints. But let's say we've gone through all this work. We've found the right model, fine-tuned it to perfection, and our team is ready to unleash it on the world. What's next? How do we actually deploy this thing and make it accessible to users? Okay, so you've got your finely-tuned model. It's time to take it out of the lab and into the real world. That's where Vertex AI endpoints come in. Think of them like gateways that allow your model to interact with the outside world, whether it's through an API or a web app or whatever system you've built. So it's like setting up a dedicated channel for the model to receive requests, process them, and send back responses. Does that mean my team needs to suddenly become infrastructure experts? Not at all. That's the beauty of Vertex AI. It handles all that underlying infrastructure for you. You can deploy your model with just a few clicks, and Vertex AI takes care of all the scaling, the security, the monitoring. Your team can focus on what they do best, building great products. That's a huge relief. But once our model is out there in the wild, processing real-world data and interacting with users, how do we make sure it's behaving as expected? How do we know it's not going off the rails? That's a great question. And it brings us to another crucial aspect of operationalizing Gen AI monitoring. You can't just deploy a model and forget about it. You need to keep a watchful eye on its performance and catch any potential issues before they become major problems. Okay, so it's like having a babysitter for our model, making sure it doesn't get in any trouble. That's one way to think about it. But a more accurate analogy might be like having a team of analysts constantly scrutinizing the model with every move, looking for any red flags. Okay, I'm intrigued. What kind of things are we looking for here? What are the red flags? Well, one big thing is data skew. Remember how we talked about the real world being messy and unpredictable? What happens if the data your model encounters in production starts to drift away from the data it was trained on? Sounds like a recipe for disaster. It can be data skew can really mess with your model's accuracy. So you need to be constantly monitoring for any signs of it. So we're basically looking for anything that suggests the model is seeing data it's not prepared for. Exactly, and Vertex AI model monitoring is great for this. It's constantly analyzing your model's predictions, looking for any anomalies that might indicate data skew. You can set up alerts so you're notified immediately if anything suspicious pops up. That's incredibly helpful, especially in a fast paced environment like asset management, where things are constantly changing. Are there any other potential issues we should be monitoring besides data skew? Another big one is concept drift. This is where the underlying relationships in the data start to change over time. What was true yesterday might not be true today. And if your model isn't keeping up with those changes, its predictions are gonna become less and less accurate. So it's like the model is getting stuck in the past while the world is moving on without it. Exactly, and again, Vertex AI has got you covered. You can track various performance metrics over time, visualize trends, set up alerts. It's all about catching those subtle shifts before they turn into major problems. It sounds like Vertex AI is really taking the pain out of monitoring, giving us all the tools we need to stay on top of things. But let's say we do detect a problem. What happens then? That's the beauty of it. Vertex AI, it's this integrated platform. So if you spot data skew or concept drift, you can quickly retrain your model with updated data, adjust your fine tuning parameters, even roll back to a previous version, all within the same environment. That level of flexibility and control is really impressive. But we've talked a lot about the technical side of things. What about the bigger picture? How does all of this fit into the broader context of responsible AI, especially in a field like asset management, which is all about managing other people's money? That's the million dollar question. And it's something we'll be diving deep into in the final part of our deep dive. Stay tuned. Welcome back to the final part of our deep dive into bringing Gen AI to life on Vertex AI. We've covered a lot of ground so far, from the technical nuts and bolts to the practical considerations of building and deploying these systems. Now it's time to bring it all home. What does all of this actually mean for asset management? What are the real world implications for product teams like ours? It's been quite a journey, hasn't it? We started with this big question, how do we actually operationalize Gen AI? And now I think we're starting to see how Vertex AI can really help us answer that question. Absolutely. We've explored the tools, the techniques, the challenges, but now it's time to connect the dots. How can Gen AI be used to solve real problems and create real value in the world of asset management? Let's start with some concrete use cases, things that your team could actually be building and deploying right now. I like where this is going. Hit me with some examples. One area where Gen AI is already making waves is investment research and analysis. Imagine a model that can devour massive amounts of financial data, like market reports, news articles, SEC filings, and then distill key insights or even generate investment hypotheses. So instead of having our analysts spend countless hours sifting through all that data, we could have a Gen AI model do the heavy lifting for them. Exactly. It's like giving your analysts a super powered research assistant. They can focus on the higher level thinking, the strategic decision making, while the Gen AI model handles the data crunching. That would be a game changer for our team, freeing up so much time and brain power. What about other use cases? What else can Gen AI do for us? Think about personalizing investment recommendations for clients. Imagine a model that can tailor advice based on each client's individual risk tolerance financial goals, even their current portfolio holdings. So instead of offering generic one size fits all advice, we could create a truly personalized experience for each client. Exactly. It's like having a virtual financial advisor for every client available 24 seven. And that level of personalization wouldn't just improve the client experience, it could also lead to better investment outcomes. As the advice is more closely aligned with each individual's needs and the current market conditions. This is starting to sound less like science fiction and more like something we could actually implement. What about portfolio optimization? Could Gen AI play a role there? Absolutely. Imagine training a Gen AI model to optimize portfolio allocations, taking into account all sorts of factors, market volatility, economic indicators, even geopolitical events. So we're not just talking about generating insights here, we're talking about making data driven investment decisions. Yeah. That could potentially lead to more robust portfolios with better risk adjusted returns. Exactly. And it's not just about the investment side of things. Gen AI can also automate a lot of the tedious and time consuming tasks that bog down our teams. Think report generation, client communication, even compliance monitoring. So it's not just about making our analysts smarter, it's about making our entire operation more efficient, more effective, more client centric. Precisely. And that's the real promise of Gen AI. It has the potential to transform every aspect of asset management, from the front office to the back office. An incredibly exciting vision. But I'd be remiss if I didn't ask about the potential downsides. What are some of the challenges and considerations that our team needs to be aware of before diving head first into Gen AI? That's a great question. And it's something we need to be very mindful of. One of the biggest challenges is data quality. These Gen AI models are data hungry. They need massive amounts of data to train and function properly. And if that data is flawed or biased, the model's outputs are going to be flawed and biased as well. So garbage in, garbage out, as they say. Exactly. That's why data governance is so crucial, especially in a highly regulated industry like asset management. You need to have robust processes in place to ensure data quality, data security, and compliance with all applicable regulations. Absolutely. We can't just throw data at these models and hope for the best. We need to be thoughtful and responsible. What about bias in the models themselves? How can we make sure they're not perpetuating or amplifying existing biases, which could lead to unfair or discriminatory outcomes? That's a critical question. And it's something that needs to be addressed throughout the entire development process. From data selection, to model training, to ongoing monitoring vertex AI, actually provide some really helpful tools for bias detection and mitigation. But it's an ongoing effort. It requires constant vigilance and a commitment to responsible AI practices. So it's not a set it and forget it kind of thing. It's about constantly evaluating and adjusting as needed to make sure these models are being used fairly and ethically. Exactly. And it's not just about the technology, it's about the people. We need to have skilled professionals who understand how to use these models responsibly, how to interpret their outputs, how to identify potential biases, and most importantly, how to make sound decisions based on all of that. So it's not about replacing humans with AI. It's about finding the right balance. Using AI to augment human capabilities, not replace them. Precisely. It's about creating a symbiotic relationship where humans and AI work together to achieve better outcomes than either could alone. I love that humans and AI working in harmony. It's a powerful vision. But the world of Gen AI is moving so fast. New models, new techniques, new tools are emerging all the time. How can our team keep up with all this change? That's the challenge and the opportunity. It's about embracing a culture of continuous learning and experimentation, staying curious, trying new things, and not being afraid to fail. So it's not just about implementing Gen AI. It's about cultivating a mindset of innovation and agility. Exactly. Companies that thrive in this new AI-powered world will be the ones that are constantly learning, adapting, evolving. This has been an incredible journey. I feel like we've only just scratched the surface of what's possible with Gen AI, but I'm walking away with a much deeper understanding of the technology, the opportunities, and the responsibilities that come with it. I'm glad to do that. And I hope your team feels empowered to start exploring these possibilities, to experiment, to innovate, and to build the future of asset management. To all our listeners out there, we hope you found this deep dive into operationalizing Gen AI on Vertex AI, insightful and inspiring. Until next time, keep diving deep and keep innovating.