Episode Transcript
[00:00:00] Good morning, everybody. I want to give everybody a little update on the whole AI thing for just a couple of minutes to give you a sense of reality, what's going on here.
[00:00:13] So there's a huge. We have had a lot of meetings in New York this week with many, many companies with very high expectations about AI. The people getting their hands on Claude and building things and assuming that they're instantly going to automate recruiting or talent management or employee experience or call centers or all these things.
[00:00:36] And I'm going to be talking a lot about this at our conference, but I want to just give you guys something to think about.
[00:00:42] We have been building our AI system for three years. And so we started early Galileo and it sounded, it felt really easy in the beginning.
[00:00:55] In the beginning, we loaded it with our content, we turned it on, it immediately started answering questions and acting like a consultant and doing all sorts of great stuff. So you would think it would have been a really easy project. But of course, what happened is it started making a few mistakes here and there and we realized that there were many, many edge cases that weren't working right. And over the last three years, what we've done is many, many things to tune it, improve it, make it more accurate, build prompts around it, give it workflows to the point where today, three years later, it's a completely different system. We have now chunked the content into semantic blocks. We have an ontology behind it. We run tests against it for accuracy.
[00:01:48] We carefully label the data before we put it in, so we make sure that it's useful so that people don't get different answers to the same question, different times.
[00:01:59] The pre prompt in Galileo is now much, much longer and more complex than it was in the beginning because we've learned many ways to coach it. And the point I'm making here is, and I don't think a lot of people understand this if they haven't done this is AI is not a implement, turn on and use technology.
[00:02:18] It's very different.
[00:02:20] You could go buy Workday or QuickBooks or Salesforce, whatever, and you could turn it on and you could teach people how to use it. And the next day they'll start using it and the next day it'll start collecting data.
[00:02:33] This isn't like that.
[00:02:35] These systems take training and coaching to work well. So one of the clients we met with was, it's a very big company said, you know, we built a chatbot for employee policies and it quote, unquote, didn't work. So we shut it off and Bill and I were sort of laughing like, what do you mean it didn't work? Well, it was giving everybody the wrong answers and they blamed the vendor. I don't even know what vendor it was. And we said, well, wait a minute, you know, it's not really that easy to do this. You have to make sure that the policies are up to date, that they're easy to read, that the system can find the information in the correct way and that you have ownership for the policies so that they stay consistent over time and they don't get out of date or inconsistent with each other. You know, if you get a policy, this was a bank, you get a policy for the US and a policy for Canada and they conflict. How is the system supposed to know which one's correct? Or is it just going to merge them or average them? I mean it, it doesn't know.
[00:03:40] So there's, you know, a lot of tuning I guess is the word, or training. You can use a lot of words for this to make these things work. And you have to realize that all of these agents that are being introduced by vendors, I mean there's just, I can't believe how many of them are just popping out of nowhere, are untrained.
[00:03:59] So how do you know if you buy an agent from vendor A, that's a learning development agent, a coaching agent, a, an assessment agent or whatever it is that is going to work correctly in your company with your culture, with your data, with your content, with your people, with your business processes, you have no idea.
[00:04:17] And the chances are it's not going to work out of the box. These are not out of the box systems. These are systems that become you, these are systems that become your company.
[00:04:29] We've never had technology like this before, not in my life, that I've ever seen.
[00:04:34] And this doesn't mean that it isn't incredibly powerful stuff. I mean, in some sense this is why it's so powerful, is that the AI driven agents or super agents learn and become better and better and better at what you're trying to do. I mean, Galileo is so much smarter now than it ever was before. I mean, it's, I'm literally astounded by what it does, to be honest, because it's learned so much from what we've given it. But I would not have predicted that and it certainly didn't act that way the first week that we turned it on.
[00:05:09] Now the funny thing about this whole AI space is nobody that I can see has created a methodology for long term improvement and optimization of Your AI based applications, your agents, we're learning it as we go. We're doing a lot of cool things. We called up Anthropic and they gave us advice on what they do on how to manage these things. There's labeling, there's training, there's ontologies to manage, there's content to organize, many, there's things to clean up. I mean, one of these companies said to us, well, we took the agent, we pointed it at our SharePoint system, created a mess because we have 17 versions of everything in SharePoint. Well, yeah, right. I mean, what do you expect? It doesn't know which one to use. It just sees a big mass of text and it just indexes it all. Every version of the document going back to the beginning of time is probably sitting on SharePoint. So we're going into a world here of systems that are very smart, very, very powerful, very capable, but they take maintenance and support. This is why we're not going to run out of things to do. This is why we're not going to be laying off everybody in HR or in IT or anywhere else. Because even though you might be able to generate code really fast or whip up a website in 10 seconds and sort of throw it out onto the web and turn it on, you're going to want to maintain this stuff and monitor it and update it, otherwise it's no good to you.
[00:06:46] So there's a weird, I guess, maybe misconception amongst many people that haven't done this yet that this is stuff that you just sort of flip on and it's so miraculous it just works. And of course, you know, anthropic and OpenAI and Microsoft are selling it that way because, you know, it is amazing and it does do amazing things. I mean, being able to summarize meetings or collapse spreadsheets into artifacts, into create graphs and charts. I mean, it is amazing the first time you use it, the second time you use it, it's a little bit less amazing. And the third or fourth time it's not amazing anymore, it just is what it is. And then you want it to be better and to work correctly and not just be half right or three quarters right. So I guess my message on this little podcast here is take some time to talk to us or whoever you trust who have been through the learning curve on AI before. You get your expectations too high. I mean, for example, one of the companies I met with a couple weeks ago is building a global new hire onboarding program. You know, these are very high return on investment jobs. They hire a lot of people and they want people to become productive quickly. And in different countries, there are different regulatory requirements for these jobs. So the way they do the onboarding and the training today is they have a call center with humans who have regulatory knowledge of each country they do business with. And the call centers work with the new hires to get them, you know, onboarded into the company to fill their forms out and do their various taxes and stuff. And then they get them into the onboarding process, which is largely done by this team. And then the line managers take over after some period of time. And this is a financial services company.
[00:08:34] So they, you know, come up with this, you know, very, very interesting idea. Say, let's build an agent. It's really more of a super agent than an agent, but anyway, let's build an agent that does this.
[00:08:43] So they work with a bunch of vendors and I won't mention the names, but anyway, they build this thing and they do a bunch of workshops and, you know, internally to try to scope out how it's going to work.
[00:08:53] And they get it up and running. And one of the things they realized, which I guess is obvious, but maybe wasn't obvious when they started, is is we have to pick the brains or understand all of the things that is in the heads of these call center agents that they've been doing through their own expertise. Because it's not all written down.
[00:09:14] It's not like we write down everything we know. I don't write down everything I know, it just pops into my head because I just know it. So they've gotta find a way to extract the anecdotally learned information from this onboarding call center and dealing with all these regulatory things around the world to get the agent to know what these people know. Now, Bill and I were kind of thinking about this and we're thinking, well, you could interview these people and just record the interviews. But who's going to do the interviewing? And what questions are you going to ask them? If you're not the person that does that work? You don't even know what to ask. Like, do you think somebody else could interview you and identify everything you know? Of course not, because they don't know what you know. Only you know what you know. In fact, we don't. I don't even know what I know because my brain just sort of stores it until I need it. So to train, quote unquote, this onboarding thing to do, everything that this call center does is a non trivial effort. It's not going to happen in a week, it will happen over time. And as people use the new agent, they'll find the edge cases that don't work and they'll say, oh, wait a minute, this wasn't correct. We've got to get better information here. We've got better information here. And that's basically what these advertising companies have been doing for years and years too. I mean, you know, the Facebook ads that pop up that seem to be hearing everything you're saying, which I think they are, or the meta ads or whatever they are, I mean, that technology took a long time to build. I mean, that didn't happen overnight. And you're building those kinds of systems now in your company. I mean, every, every AI system you build in your company is going to go down the learning curve of the advertising industry and how long it took them to be able to personalize your ads. Now it'll happen faster because the systems are much better, but you're going to have to be involved in that on an ongoing basis till the end of time. Because these applications we're building in AI, they don't freeze in time. If you look at recruiting in a, you know, a high volume retailer or say a restaurant chain or something, and you have a bunch of questions you're asking people and you have screening stuff and you know, you have some, maybe some cultural things you want to identify during this, the selection or the screening process, you know, you could probably figure that out relatively quickly. You could get recruiters to help you define what those things are.
[00:11:39] And then, you know, you could get the AI to learn that, you know, maybe in a week or less, wouldn't take too long. But you know, let's suppose the company goes in a new direction, like I'll give you a good example, Starbucks is going through a turnaround. And I go to Starbucks usually in the morning to get an espresso. I mean, they've changed the culture a lot. I can see this as a customer, when I go to Starbucks now, they know my name, they say hello, they greet me, they're very, very different. Something has changed. They have changed the way they go to business and the way they're dealing with customers. It's good for them. I'm very happy about it. Well, that decision was made in corporate somewhere and then it proliferated down into a whole bunch of training stuff and then they had to do hiring stuff and they had to decide that with this new cultural experience we want to create in the stores, are we hiring the right people? Well, if they had an Assessment of some kind, which they probably do. And I know they do a lot of sourcing through intel, through AI because they know a lot about that stuff they're doing there that's gotta be tweaked so that would have to be retrained. So these systems that we're building are miraculously good and powerful, but they need ongoing support. And so we're going to have these really interesting jobs in our companies to maintain support, train, improve, update, modernize, whatever word you want to use. The intelligence systems that we have and the applications are just spectacular. I think I told you guys about the media company that's going through risk assessment of different skills and jobs. I mentioned the onboarding yesterday. We had a whole bunch of people from Talent Acquisition talking to us about various applications. One of them is a chain of fitness centers and they're, you know, really, really trying to ramp up the hiring of trainers and you know, certified fitness professionals and there just is such a shortage. They're going to build their own academy and their own certification and they're worried that when those people get certified they're just going to pick up and leave, which they probably would, right? I mean, they go look for another job. And I said to the, you know, the head of ta, I said, yeah, but you're a brand. I mean you're going to have to, this is good for you because your brand will create career opportunities for these people going forward. So who do they select now? You know, does that change their selection? I mean, for example, in our company I get a lot of inquiries from people that want to work for us, that are gurus or you know, maybe they self proclaim gurus, but they're gurus in different things. We don't really want to hire gurus. We want people to become gurus the way we're gurus. We don't want them to bring their guru ness to us and try to change us. We already kind of know what we want to do. So there's a very fine line between a guru who fit with us or not. And so I would have to train Rai on that. So, you know, my point is that this is a very, very interesting technology and, and take some time with. And this is, we're more than happy to talk about this to you directly if you'd like and learn from others what it took to make these agents that you're buying work. Well, if a vendor throws something at you and says, oh, we have so and so using it and such and such, using it, call them, ask them what happened. Ask them what went well. Ask them how long it took to get it into a stable state. Ask them how well it's adapted over time.
[00:14:59] These things are very powerful, but they take operational support, and they will help you decide how to staff the project so that we'll improve over time. Okay, that's 20 minutes for today, almost.
[00:15:11] I just really felt I needed to send this out because we had so many conversations about this last couple of days in New York. We're meeting with Sana on workday today at a big AA AI conference, and I promise you I'll tell you all about that in the next podcast. Bye for now.
