Episode Transcript
[00:00:00] Good morning, everyone. I'm recording from the Workday Rising conference in Barcelona this week with many, many observations about lots of things going on. But I'm going to try to focus on one topic which I think will be educational, which is the making of an agent. The Galileo story. And the reason I want to tell this story is that it's actually very educational for all of you in your efforts to buy and build the AI solutions you want for your company and for your personal life. Now, going back 25 or 30 years, what we've been doing and I've been doing is vast amounts of research on all aspects of human capital management, technology and hr. And that has included research reports, maturity models, surveys, benchmarks, studies, individual case studies, speeches, debates, radio roundtables, and many, many, many other methods of getting to know how a person or a company solved the problem and then diagnosing what worked and what didn't work. And that's essentially what we've been doing for 30 years. And what happens when you do that is you accumulate a vast amount of experience and knowledge in this domain and you begin to see the interconnections and the patterns. And so over time, our quote unquote research library was, we've always called it has become more and more diverse and integrated and interesting and important.
[00:01:32] But except for a very few of us like me or many of you I know are raving fans. So a lot of you do read a lot of the stuff we do. It's hard to see the relationship between all these different innovations without an expert to answer for you. And so we decided years ago that we were going to be a customer, intimate company, not just a publishing company. This was a decision I made earlier, early on, and that we would listen and talk to our clients a lot. And so what happened was most of our corporate members, we used a membership model, would call us or we'd meet with us, or we'd go meet with them and we'd have roundtables and different ways of talking to them. And we would learn from each other the intricacies and subtleties of these different areas of HR relative to that company's needs and that company's situation. So the fundamental value problem was collect a lot of data, make a lot of sense of it, and apply it to a given company.
[00:02:32] It's sort of what consulting firms try to do. But we focus much more on the research side than the consulting.
[00:02:38] And then many offshoots come from that. The analyst work we do with vendors, we believe that vendors are a part of the problem's solution, not a vendor as an end in itself. In other words, we're not an analyst firm like Gartner that studies vendors just for the sake of extracting money from vendors. We do it as a part of understanding the market and what best customer solutions would be. And we realized maybe almost seven, eight years into this, that what people were telling us was we like reading your research, but would you please teach us this stuff, put together courses, become an educational institution. And I resisted that for a while and then realized this was a big gap that we had been missing is to take this material and put it into, in instructional form so somebody didn't have to read a hundred page report. They could sort of go to a class or watch a video or get taught what we had learned. And of course, the intricacies and subtleties of all these things of, you know, all the things in HR are sometimes easier expressed visually or verbally than they are in writing because it's just too time consuming to read. Okay, so that's all been going on. And then the challenge that we always felt we had in doing all that was the search engine. If somebody searched a website with a problem statement, how do I X Or what is the best practice for Y? Or how much money should I be spending on Z? Or should I use this or that vendor? The search engine never quite worked. I mean, it kind of found related material, but it never really could answer that question.
[00:04:18] So Fast forward to 2022.
[00:04:21] You know, I've been doing this now for 20 some odd years and ChatGPT comes out and within a few days I thought to myself, maybe ChatGPT could replace our search engine. And I have a good family friend who's a software engineer. And I said, you know, and I told my co founder Bill that I was going to do this, but he wasn't particularly interested in it at the time. I said, I'm going to try a little project here. I'm going to try to see if I can get some of this AI technology from OpenAI, which was new to work against our data. And we built a thing which we called an HR copilot. At the time we didn't have a name yet and we got it working. And I was absolutely astounded at what I saw because we just took our PDFs and other reports and we put it in there. And all of a sudden I realized it wasn't a search engine, it was a digital consultant, it was a digital educator, it was a digital advisor and actually a digital coach. Now, it was early days. We didn't optimize it a lot. And we showed it to everybody at Irresistible that year, I think it was 2023. We probably showed it to them. And the. Maybe it was 2022, I can't remember. And the reaction was astounding excitement. And the first question somebody asked was, how do I put my data in there? And I thought, aha. Okay, this is a little more complicated than I thought, but that makes sense. If our intelligence database is so good, why wouldn't somebody put their leadership model in it? Or their behavioral interview or their employee data, which I'll talk about in a minute. So I scrambled around for a month, called a lot of the VCs I know and said, you know, here's what we're trying to do. How do you guys think I should do it? And one of them put me in touch with this little company up in Sweden called saa. And I immediately talked to Joel Helemarch. And he knew who we. He knew who I was, and he knew what we did because he was selling into L and D. And we hit it off almost instantly because they saw the opportunity for what we were trying to do. And we were desperately in need of an expert to help us. And so over a very short period of time, we had a big meeting in Europe with our management team. We went up to Sweden and met them, spent a day with them, which was great, got to know everybody. They started building this new thing. And at that meeting, we literally were having dinner, drinking a couple of glasses of wine, and I came up with the name Galileo. And it just felt like the right name, and we marched ahead. And so Galileo started as a file folder in a sense that there's much more going on under the covers of all of our research in the SANAA agent platform. Now, we are lucky that we're very well organized. So all of our research is very carefully tagged and organized around domains with lots and lots of metadata. Most of you don't have that in your corporate data, but we do. And that's partly because I used to work for a database company. It was something I always wanted to make sure we did, but there was more to come, so we. So anyway, let me finish. So we go through launching Galileo, we get, you know, a fair number of people really liked it, and we didn't really have a team that knew how to manage this. So we have, you know, a number of really smart people in the company. And sure enough, one of the people, Gracie, who works for us, who used to be a schoolteacher, actually volunteered to be the one to manage the corpus, the data. And we needed some infrastructure to keep the data up to date. So my computer software friend built us a real time update mechanism so when we would publish something onto our website, it would immediately go into Galileo. We later added the podcasts, lots of other data from other sources. So we needed a data input technology which we custom built. And then Sana, of course, began to look at the kind of stuff we were putting in there. And as we added more and more different types of content, some of it worked, some of it didn't work. And they were continuously optimizing the, you know, the internal process that the AI used at that point in time to accommodate all our data. And I'll explain more about that when I get to Jupyter. So anyway, we're moving ahead here. We got a name for it, we got a brand for it, we start doing some marketing and somebody says to us, well, you know, I was asking it a question and it kind of hallucinated. And we go and we start testing it and we realize, you know, this doggone thing is making stuff up. It's mostly right, but every now and then it kind of wings it, it guesses what a vendor does because it thinks it knows. I don't know how it found out. And we realized it was getting polluted from the outside Internet and that we needed to focus on trust because that's the business we're in. I mean, there's. Nobody's going to buy any research from us if they can't trust it. So we go back and we look at it again and we, we close off the Internet. We talk a lot with Sana about it and we start tuning it for quality. And we found that the most of the reason it was hallucinating had nothing to do with us because our content is very, very domain specific. But it was getting data from the Internet outside. So we now see the need to be careful what third party we put data we put into it. Meanwhile, people are using it and they're asking us all sorts of questions about things we don't know. What is the vacation leave policy in Croatia if I hire an employee there? Or what is the salary that I should be proposing to a chemical engineer in Cincinnati? And is that salary higher or lower than if I hired them in Houston? And we realized we need more data. So we embarked on a new initiative to build trusted content partners. Lightcast. We. We found some great regulatory data from a company called Oyster that a lot of you know, we partnered with Vizier on a bunch of benchmarks and then Korn Ferry for a leadership model, SHL for a robust model for job capabilities and competencies. And in discontinued, we now have Equilar and therefore executive compensation. But each time we did this, we had to test to make sure that the answers were correct and that it wasn't making mistakes. So we ended up building 50 prompts, which we call now golden prompts, to test to make sure that Galileo doesn't become less trustworthy than it should be or that it just needs tuning. And we learn many, many things about why it would make mistakes. It has something to do with data labeling. If you give an AI a bunch of data, but you're not clear about what the data means, it's going to guess. And it might get it right, it might get it wrong, and it could make a completely wrong conclusion based on simply data labeling or explaining what these numbers are about. So we're moving down that path, getting involved with more partners. And then a bunch of you guys say to us, we don't want Sana. We've already got the Microsoft copilot. What about Claude? What about Gemini? What about this internal thing we developed? You're going to make us buy another agent? Well, it turned out nobody minded that too much because we had such a unique solution that they would bring in Sana anyway. And we ended up selling Galileo to, you know, more than a thousand companies. It's probably more like 2 or 3,000 now, and they're doing amazing things. And we ended up getting very unique situations where, for example, iata, the International Airport Airline Transport association, built their entire competency model for airlines in Galileo. And so we started to get those, you know, really cool use cases out there. But the demand was starting to, I guess, wouldn't say moderate, but it was starting to bother us that if we couldn't get this thing to work on multiple platforms, we were going to get cut out a lot of these big companies because the companies are standardizing their AI technology.
[00:12:21] So we got that problem. We have this never ending problem for more data. And then we have the problem that the system itself isn't really portable because Sauna is a harness that sits on top of the LLM in Sana, you can switch which LLM you want to use. And the code that SANA had built to help us is proprietary to sana. So as much as we love Sana, we couldn't do it all ourselves. We needed to run on other platforms. And our Academy, which is now running in A legacy platform by the name of Nomadic, which is gone. You know, what are we going to do with that? Because the new content's coming out faster and faster and faster, and we're building these courses in this other platform. So the first thing we did is we licensed Sana Learning and we built an academy, which we ended up calling Galileo Learned, which is a partner to the agent. So when you buy Galileo from us, you get the agent and the learning platform together.
[00:13:21] This is something Sana is not selling through Workday, but this is part of their product.
[00:13:26] So you can go into Galileo and you can ask a question, and on the right side of the screen, you'll see courses that you can take to help you answer it if you decide you want to learn more. And through the AI capabilities of Sana Learning, and only six months of work with them, we built 800 courses on all of these things that we had been working on for years in hr and a process for doing that so we could build them very quickly. But meanwhile, we got this problem of platforms. So, you know, somewhere along the line, we hired an amazing human being who's our chief architect. And he said, well, you know, the problem is the corpus itself is very primitive. When we did a project with ServiceNow, where we wanted to move Galileo into the Now Assist platform, which has been replaced by a newer one, and the only way we could get it to work was to take all of our PDFs and spreadsheets and everything else and stick it in their infrastructure so that they could index it into their platform. And actually, that was our first implementation of the copilot version of Galileo 2 is we literally worked with Microsoft for a year to get all of our content optimized into the Microsoft cloud so they could input it into the copilot. And our new architect, Rom, says, you know, you guys, this is not the right way to do this. We need to decompose our content into a more portable form where it can be moved and integrated more quickly. And then, of course, MCP was coming along, so we really needed to build an MCP server as well. And so what we did next is we said, okay, we're going to keep producing reports and PDFs and, you know, spreadsheets and all that, and podcasts and all that, because that's a different part of our company, which, by the way, we're going to redo all that too, but that'll come later. And we said, let's apply an AI engine to decompose our content into chunks. Now, if you Read about the Jupyter release.
[00:15:27] There's a white paper that describes this. And we actually got advice from Anthropic. We talked to some of my friends who work at Anthropic, and they explained to us that this is what they do. They don't only take raw things on the Internet and just stick them into the LLM. They have to decompose them into pieces and tag them. So we ended up building, you know, fairly quickly a very, very comprehensive, cleaned up, in a sense, optimized version of our content, which we call ARC Agent Ready corpus. And by doing that, we stripped out a whole bunch of data and the reports that nobody really can read or wants to read, tagged every single thing in there, added amazing amounts of metadata so that the agent will quickly know when you ask it a question, the answer, or where to find it without guessing and scrolling through a whole bunch of tables of contents of different reports to try to find something. Because before Jupyter, the agent would literally scan through a report one at a time with lots of duplicative information about, you know, the, the footnotes, the sources, the authors, and somehow to make sense of all that. So we made that easier by telling the agent what all this stuff was. That's what Jupiter is. And the doggone thing became much more fast and effective because those 50 golden prompts, which we've been using now for a couple of years, all of a sudden became really intelligent answers. And Galileo, you know, the reason people like it is it's exceptionally good at doing the kind of things you do.
[00:17:05] I mean, it will give you very insightful answers to the degree that we now have prompts in there, in the, in the library that will coach you to solve problems. It's, it's, it's literally a consultant now. It is a digital HR business partner, as is, without adding any more value at all. It does that right now, today. And once you add your own content into it, it becomes the HR business partner of your company or a digital twin. And we'll about digital twins on another podcast. So now we're doing that, and sure enough, we hook it up to the Microsoft Copilot through MCP.
[00:17:43] And the score of the, of the 50 golden prompts is the highest we've ever seen. It's even higher than sana. In other words, the MCP Jupyter optimized version of Galileo is even more accurate and it uses almost no tokens because we've eliminated a lot of the scrambling around looking for stuff that the language model had to do before and really optimize the corpus. Meanwhile, while this is going on, Galileo Learn is taking off and we're adding more and more courses. And we realized that the solution in a corporate account in any one of these HR domains isn't just an HR solution, it's a manager solution.
[00:18:26] And so we start talking to a bunch of companies that want to use Galileo for their employees and managers. And we build a version called Galileo for Managers. And Galileo for Managers is the same corpus, but with a totally different language and set of prompts and context so that a non HR expert can ask a general question like, I've got an employee who's underperforming and he's driving me crazy and he keeps showing up late. What do you advise I do? What are the regulatory issues I have to run into, what's the best way to handle this grievance, et cetera. And so we, we don't, we essentially expand the domain of Galileo into employee support, employee services, employee relations.
[00:19:09] And that's another domain, of course, where we have lots of information and research and we can get more content partners involved. So you can see where this is going. And, and then we launched the Copilot version and along comes Workday and SAP. SAP wants Galileo to work on Joule. We do a prototype with them, they sort of lose interest and get working on other things. And along comes Workday. And Workday says we want to work with you guys directly. And that's what led to Galileo inside of Workday. Now, meanwhile, while all this is going on and we're moving through this path, this all happened in three years. So there's a lot of activity behind the scenes here. One of the things I always felt was important that we couldn't do alone was connecting the Galileo data to your HR data. Not just your HR policies, but the real data about employees. And we had been talking to hi Bob about doing this and they were building an agent of their own. So we, I said to one of our people here, Sarah, I said, sarah, I got an idea. Why don't we just build an HR database of a simulated company? We can get Galileo to help us build it. And let's copy the architecture of the Workday database just because it's one. We know there's a lot of different ways to do this. We just sort of copied their model and we built this company called North Shore Outfitters. And it's a retail company and it has all the business functions of a regular company. It has about 15,000 employees and it has a bunch of retail operations in different functional areas. So we built all the job titles and the names of the employees and their levels and their pays and all that stuff. And it wasn't that hard to build. It turned out Sarah, I don't know how she did this, but she built it in, you know, maybe a month or less. And we put it into a big folder inside of Galileo and said, okay, Galileo, what do you know about this company? Tell us about the span of control, Benchmark its salaries against peers.
[00:21:08] How would we reorganize it to improve productivity? And we started talking to Galileo, and sure enough, now Galileo isn't just an HR best practices advisor. Now it's an organizational consultant. And we realized something that we kind of knew in the beginning, and certainly I did, which was, the more valuable data you give it in the given domain, the smarter it gets. Because now if I ask it a question like, how do I improve the career model or the retention of our retail employees, it knows more about the company and what are the options it could do. It could propose an individual has a new career by looking at adjacent skills because it knows everybody's skills, because it has access to lightcast. It could advise a manager on how to reorganize a retail division or department because it knows how the rest of the company is organized. I mean, the stuff you can do with it is mind blowing. Sarah or I or one of us can give you demos. And then I realized, this is what's going to happen when we connect Galileo to an hcm. And of course, hibob's working away on their thing and we start connecting to hibob and voila. Galileo now is an organizational consultant for anybody that uses Haibob. So along comes Workday. Meanwhile, in the middle of all this, Workday decides to acquire sana. And we were a little bit perturbed in the beginning and because we didn't want to lose our best friends. And of course, the SANA people never stopped supporting us at all. They're wonderful, wonderful people. I mean, I just cannot say enough positive about SANA as a company. And we obviously know Workday really well, and I know Aneel and a lot of people there, and we get back equated with workday and we tell them what we're doing and we explain it to them. And I essentially said to Garrett and to Aneel and a few others, I said, look, you guys, this is a transformational change in the definition of what workday does. If you had Galileo embedded into workday, Workday becomes an organizational consultant or an organizational coach, or an organizational development tool for the business, for the cfo, for a manager, for an employee, for an HR department. I mean, it isn't just an HR advisor. Workday itself becomes a dynamic platform for business growth.
[00:23:33] Meanwhile, they're working on this on their side too, because they're building all sorts of new agents and we just hit it off. And so what we've been working on, it's, it's available to purchase now and it'll be in general availability in February, is an integrated version of Galileo into Workday. Now we are partners with Workday. We are not owned by Workday. We still operate on multiple platforms. You can get the Galileo, you can get Galileo on the Copilot. So if you have Workday in the Copilot, you can have a seamless experience. And the way we think about Galileo now is it's not just a digital business partner or an HR agent that does all sorts of great cool stuff for HR people, or a digital coach for you as a leader or a manager, but it is an intelligence system that makes your agents more intelligent. In other words, if you're building the world's best onboarding agent for your company, or a development agent, or an agent that assesses candidates or builds onboarding programs or whatever it may be when you connect it to Galileo through MCP or through whatever platform you use, Galileo will make that agent smarter about what it's doing because it has almost 2,000 examples of companies solving problems in HR. There's 1,800 roughly today, but we're doing a lot more. It has models, it has maturity models, it has vendor information and it has benchmarks. It has world class intelligence about this. And it's not as random or haphazard as you would get on the Internet. If you go to the Internet and ask any HR question, it'll answer it. But you're probably getting a blog from a vendor written by a journalist. You're probably not getting real research. You have to dig around to find it. And you may or may not ever find it because it isn't always clear why it says what it says. With Galileo, you get a trusted answer. And anytime either you or the agent you're building wants to know why such and such a solution was recommended, it tells you why and it shows you the source. And you can read the source and validate that. Oh well, this does apply for us in the healthcare industry, or us in retail, or us in oil and gas, because it came from real information, from real companies who've done this in this domain. So, you know, there's the story. Now, the reason I thought it would be interesting to share this with you is that every one of these steps we went through is exactly what you're going to go through. When you build an agent, you're going to have to define the domain, you're going to have to define the use cases. You're going to have to find the source of truth. It's not going to be easy to find it, especially in a big company. Some of the source of truth isn't written down. Some of it's in somebody's head or a whole bunch of people's heads. The data that is written down might be old or there could be multiple versions of it. And what you'll find is you're going to have to do a lot of work getting that information together, interviewing people, recording data, cleaning up data, and then feeding it into the agent to teach it to do exactly what we did with Galileo. Now, one of the miraculous things in our story, and I think this is true for you, is we had very, very little software engineering support involved. Sana, of course, helped us a lot, and as did Daniel and the other people who worked on the Copilot and the integration tools. But for the most part, the work of building an agent is understanding the business, understanding the problems, understanding the domain and understanding the information and where it came from, and also understanding the rules and guardrails so that the agent itself does not have access or does not accidentally give information that it shouldn't give to the wrong person. If you have an agent that's doing sales compensation for salespeople, and it knows the salaries and the pay structures and the commission structures of all the salespeople, and you plug it in to your employee services agent as a source of information for salespeople to find quarterly or monthly reports on their compensation. You gotta be very, very, very careful that that thing doesn't accidentally tell somebody who's not supposed to know why or how much a salesperson's making in a given situation. And because HR is so interconnected, the value of AI in HR is these interconnections between these systems. So we kind of want to connect them together, but we want to do it in a very architected way. And that's what HR 2030 is all about, is we've kind of built out a framework and a blueprint to show you how to do that. You will have to tune these things. When new data comes in, we test the prompts against it and make sure that it's working that we've labeled it correctly. You're going to have to look at different platforms. When the LLMs get upgraded, which they seem to get upgraded every day, sometimes the behaviors change, sometimes we have to change the prompts just to accommodate the new features. And then you're going to want to probably look at performance. In our case, performance is not an issue because Galileo takes very few tokens to run. But if the way the data is structured is creating massive expenses in your token costs, you're going to want to work with it on stripping out the non essential data and tagging it like we did with the Jupyter release. And then of course you want to give it access to more data.
[00:29:12] So going back to the North Shore Outfitters example, I think, and this is my hypothesis, that the big ATM providers are going to become essential parts of your AI strategy. They don't have all the data you need, they have some, but they have a lot of critical important registry data that you need in order for your AI to work. Who is this person? What job are they in? What level are they? Where do they live? What jurisdiction do they live in? What functional domain do they have? What skills or jobs have they had? In the past, that sort of routine HCM information is really valuable when you start using AI. And we've played around with North Shore Outfitters so much that we know now that the depth of that HCM information becomes more and more valuable when you do a merger, when you do an acquisition, when you do a reorg, when you do a layoff. All of those things you do in HR will leverage that information.
[00:30:14] So that's the story. I think a lot of you can learn a huge amount about AI by using Galileo because Galileo is a platform for you to build your own agents on top of, because it has this corpus of authoritative domain knowledge built in. You get, in a sense, a highly trained, world class consultant to start and then you build your agent on top of that. And so whatever agent you build will get smarter and more determined and more pinpointed by using the Galileo corpus as part of its intelligence as well. I hope this is interesting and educational and I thought it'd be kind of fun to put it together and just talk to you about it while I'm here at Workday. And if you're in Europe at the Workday Rising conference, please come up and say hello. It's going to be a big group of people, but I hope to get a chance to meet many of you here. More to come and more great examples and case studies coming in the next couple of days. Bye for now.