64: Leading AI Adoption: From Faster Tasks to Redesigned Work

Pete Behrens:

What is the point of AI? I get it. It's fast, it's smart, and we're in an arms race. Not only companies developing the latest models, but every company seeking to leverage those models for competitive advantage.

But to what end? Where is it leading us?

To answer that question, I invited two people who are at the tip of this spear, AI change agents, let's call them, hired specifically to drive AI competency across their organizations. Together, we peel back this curtain beyond AI use and enablement into what I could only call a maturity model towards an AI-native organization.

Now, I get it. Those are big, complex words. My guests, they simplify that message. They take us inside what's really happening across organizations, what's actually going on behind closed doors with leaders who are pushing on this more than I've seen in my entire 30-year career.

You might remember Eric Kihn from a previous episode. He now leads AI innovations and solutions for Hach, a water quality company. Charles Fleet is vice president of artificial intelligence for Neumo, a technology company servicing governmental agencies.

This is the "Relearning Leadership" podcast. I'm your host, Pete Behrens, CEO and founder of Agile Leadership Journey, guiding leaders to places we've never been before. And I think this qualifies as one of those places.

So without further ado, let's dive in…

Interview

Pete Behrens:

So I use the term AI change agent for what you two do. Uh someone hired by a company to help uh I don't know um develop maturity, a competency in in AI. But I'm curious, how do you define yourselves to colleagues and friends? And maybe Eric, I'll start with you.

Eric Kihn:

Um, yeah, that's a great question. So, when I talk about what I do here, you know, my title is innovation and solutions manager, right? So, it's about bringing AI innovation into the organization and it's about managing what we then develop. So, you know, the innovation side, we started out getting people curious about AI. So, I've been with this company for 15 months. Um, it was when I got here, a lot of people were aware but not curious, didn't want to get involved with it. Then moving it up to enablement, you know, hey, this is how you can get this kind of stuff going. Ultimately, you know, we're headed towards being a truly AI native organization. We want it fully not even just integrated, but but really so that we're redesigning the organization to include the concept of AI fundamentally. And that, you know, is ultimately where I am. It it's almost more on the organization side now than just, you know, demonstrating technology.

Pete Behrens:

Hm. Charlie, how would you uh respond to that?

Charlie Fleet:

Yeah, I I couldn't agree more with Eric on that. I shift even in the last couple years where, you know, an AI change agent was somebody who came in and said, "This is how you use AI or this is what it does for you." Um, but we're reaching, I don't know if it's a tipping point or not, where it's more of people use it on a day-to-day basis, and maybe an organization has it or doesn't have it available to them, but they still are using it on personal accounts. And so it's more about why are you using AI? Um, you know, so it's it's more about, oh, you could use it here or here in your organization. Which one would be better? And so, um, and why would that be better? Oh, that's going to help us grow more. Oh, that's going to help us have a higher customer service. So like it it's it really gets into the operations the business case like the business improvement and less more the less so these days about this is how you do prompt engineering.

Pete Behrens:

But if you had 10 seconds in the elevator, say, "What do you do?" What do you say, Charlie?

Charlie Fleet:

I am here so that our organization uses AI most effectively.

Pete Behrens:

Eric?

Eric Kihn:

I'm here to make sure that AI gets integrated into all of our workflows and done. Sustainable, reliable way.

Pete Behrens:

Oh, now we get the big words in here. Ethical, sustainable, reliable. Love it. Uh, which we'll which we'll dive into. So, Eric, as you were describing the first time through, you you use words like enablement, you use words like nativeness. You're inferring and Charlie, you kind of talked about this too. You're inferring a a maturity model. There's there's some developmental stage you know I think back to the CMMI days of of you know software maturity or you know you can look at other industry standards is there something you're following is there language or or tools are you developing your own is that even a is that even a thing in your companies that there is these stages and there are these definitions or is it looser than that and maybe Charlie I'll switch and start it with...

Charlie Fleet:

Yeah. So you can ask Claude to write you an AI adoption maturity model and you'll have a new one in five minutes. And so googling one or using one that's published I found to be a bit futile because within a week there's a new one that's flooded the LinkedIn in network. To me the most important part is coming back to this why. Uh and I I right now have a walk a crawl walk run model. So if using AI helps you in a task and and you know you can see it that's kind of crawling. If you can show that you AI with some type of number like I process le less tickets and I went 50 to to 25 this okay great that's walking. But if you can show that your customer satisfaction went up, your conversion actually went on on sales went up, that your actual P&L improved in that quarter, now you're running. And so if we think about the long view of AI and it's the cost of tokens and and like what's the cost of consumption, we are going to have to start intersecting those with the actual, you know, business performance of the company, not just the the the crawl and the walk model, the actual run model. And we're going to have to intersect those two. We're have to be able to forecast and predict that. We're not there yet as an industry, but that's the direction we're heading.

Pete Behrens:

So I'm interpreting crawl, walk, run into terms like, well, it helps me in a task. It maybe speeds me up or creates some kind of momentum improvement versus it actually is doing some outcome-based improvement.

Charlie Fleet:

Right.

Pete Behrens:

Is that a fair interpretation of some of those terms?

Charlie Fleet:

Correct. Absolutely.

Pete Behrens:

Interesting. Yeah.

Charlie Fleet:

Yeah.

Pete Behrens:

Eric, do you see that differently or do you approach this maturity model differently?

Eric Kihn:

Well, I I totally agree with Charlie that you know, if you follow the literature, the AI maturity models. So we have consistently since I got here had kind of a very generic one which is start with curiosity. Curious is the first step, enabled is the next, then governed, and that's where we are now. So we're transitioning from enabled like we have a lot of tools and things out there we need to get into the governance phase, then integrated, and then we were calling it AI hybrid but for this conversation AI native um as the kind of the top level and we know that there's steps to get to each of that you know at first it's about getting people excited, second's about you know getting some tools out there that people can feel productive with, and it gets harder as you go up. I mean governance, you know, it's always like, oh, the fun police have arrived, but how do we create a governance that lets the safe path be the quick path for people so that they're not only getting on in a safe and ethical and reliable way, but it's going quicker because they're they're willing to follow our road. So, that's a lot of the work now. And then the next levels get even harder because you're going to be asking leadership to make major changes as you become an AI native organization um around you know what it is a leader does, how the organization functions, changing you know redesigning whole structures and workflows, which we know is a big task. So I when I got here, you know, since I was starting a new program, it was definitely the easier, funner phase and it's getting, you know, more and more lift as we go.

Charlie Fleet:

Eric, I want to I want to jump in on something you said there because it's it's actually our model is almost not incongruent, but it's it's different than yours. We have four focus areas. We have um you know what we build, that would be our products and our capabilities. We're a um scrappy govern um government tech company. We build software for for local governments that serve communities. But then there's the how we build. But then we have two more layers. We have enablement and governance. So we enablement governance. We put them right beside each other because the enablement's about is the organization primed to be able to do this, the AI and chai thing that that you and I have done, those kinds of programs, separate from do we know how much money we're spending, are we working on the right programs, is this ethical, are we are we using these in an ethical way in our practices, and that governance layer completely independent from from enablement. So interesting how we use the same words and and very differently. Okay.

Eric Kihn:

Yeah. I mean, yeah. And part of it is I think when we thought of enablement it was really about getting people to engage with the tools and do some some basic good. And then one of the... We wanted to make sure that that was happening reliably. It's now become much more of a focus because it is like our uptake has been huge. People are submitting use cases after use case and approaching business problems every day with it. So now you know we do have questions around cost management. You know all the token get expensive and you know duplication, are they this application to that? So it's a stronger governance but you know beyond that getting it truly integrated into our work so that it is you know built on enterprise data that is integrated with all our workflows and all that will come from having that good governance structure is our perspective.

Charlie Fleet:

Yeah.

Pete Behrens:

Yeah, I can definitely see a a challenge. I love the curiosity you bring in, Eric. I think that's a really cool way to think about the crawl, like a child curiosity. Uh just just explore. And where I get a little bit more huh is in that government governance integration nativeness like I I'm seeing a lot of nuance and maybe a Venn diagram overlap between those that maybe is less clear to me and you know versus maybe Charlie's you know crawl walk run has has a little bit more clarity or to me I just think of it as kind of a two-phase. It's use and then redesign. You know, I think of all right, how are we using it and is it is it impacting kind of task and work and and our jobs? And then to me, the the the nativeness really comes in. How are we redesigning everything? Governance, roles, responsibilities, jobs, you know, all these other things an organization has to do to survive. My guess is the nativeness kind of touches all those governance, the policies, the risk management, the the roles, the responsibilities, the org charts, everything else.

Eric Kihn:

Yeah.

Pete Behrens:

No question in there, just comments.

Eric Kihn:

I mean well I mean...

Pete Behrens:

Yes.

Charlie Fleet:

No way. It's lots of head nodding.

Eric Kihn:

We definitely agree with that. I mean one of the things when you when I'm thinking about AI native we're intentionally redesigning everything around the human and machine capabilities, understanding what is fundamentally going to be machine-based going forward, that the concept of developing hybrid teams and ma how do you manage a hybrid team changes a lot of what leaders are going to be asked to do. The one that you know really stuck with me was now you would in traditional development you would go out to your people and say, "Hey, go learn a skill like, you know, we really need some good Python developers on this team. Go learn Python. It'll serve you well. We're counting on you to support the organization with that." Now, maybe the machines are are your Python coders. So, what is it you need that human in the loop to do? And you definitely need them. And if you understand your workflows... So you're going to have to rethink a little bit even what you're asking your people to do as a leader for their development. So that's the fundamental challenge with getting to AI native as I see it.

Pete Behrens:

Charlie, you wanted to jump in.

Charlie Fleet:

I I yeah I mean if I think about I mean yes and Eric everything you've said um there there's a there's this pit that I have in my stomach of an answer I question I don't answer which is um in the past it's been pretty easy for us to take responsibility for the our own work quality like when I'm doing my job and I write an email and I draft a document and so there's an defect in it or something I can be accountable for it and so there's this emerging like conversation where it's like well I asked Claude to do it and it made a mistake, I still responsible for that error. And the line has been, yeah, yeah, it's your job to prove it. It's your job to prove it. As as the the interdependence that we have on AI systems builds over time, at some point that breaks because we've built so many skills that we use. It's kind of like a colleague. This this skill, this agent becomes, you know, somebody I'm not responsible for. It just becomes a part. Am I then responsible for its work anymore? How does this play out? I I I don't know.

Pete Behrens:

Yeah, I think what you're getting into Charlie is this level of authority you're handing and the way you know I've seen that described is there is the you know delegation construct where I delegate task and work but hold on to authority of decision and ultimate responsibility.

Charlie Fleet:

Right.

Pete Behrens:

I've personally, you know, in our company and some of the coding projects and other projects we work on, I've been exploring switching that model into a delegating what I might call product ownership like like decision authority over a a small part of this project like you decide.

Charlie Fleet:

Yeah. Mhm.

Pete Behrens:

And so far I've been disappointed. Maybe I'm not setting up my environments properly. Maybe I'm not controlling. But, you know, I've seen things like I start to distribute project management and it just starts going into the weeds and like I'm like, whoa, pull back here a little bit. This is this is not working. But is that maturity level? Is that my my control level? But I think what you're getting at is we're teetering, I think, in AI a lot towards this. And I think certainly the companies that are making AI models are trying to push them more and more into the place where they could take more responsibility. Uh I think right now what we're seeing in practice, humans stay ultimately responsible.

Charlie Fleet:

Mhm. Yes. It's coming though.

Pete Behrens:

I want to get in the meat of our conversation and I I use a tool called the Catalyst Canvas to kind of focus how we might think about something. And we've already really talked about the center of our canvas which is the goal, right? The goal to be more AI competent, to be more AI native, to be more AI enabled and improving better outcomes through AI. The the most important piece of this though and Charlie you started to pick up on this is the why. And the next piece of the Catalyst Canvas we often hit or we do hit is not only the why but the why now. It it it focuses both on an importance and an urgency. And when you get that question why now? How do you respond to that in an organization or when you're dealing with parts of your organization that are maybe more traditional or maybe more operational that are just are not in this AI realm today? How do you answer that? And I'll kick it off with Eric here.

Eric Kihn:

Well, I so when I think about the why now, I mean certainly there's the standard third answer is if you're not doing it, everybody's doing it, right? It's an arms race and the businesses that you compete against are moving better, faster, stronger with an AI enabled workforce. And so in some sense, you're getting towed along in the wake. But bigger than that, I mean, what we're really trying to do is one, improve the business, you know, increase our organizational capacities, improve consistency, um, learn how to scale better, be be more agile. So, one of the things with AI agents is, you know, pay the tokens, you can double your workforce in a given area in a given time, right? And in the old days when you had to go out and hire and bring people on board, that was a very grueling process. We're also, you know, interested in honestly improving the workspace for people. So, can we reduce the low value administrative overhead? We have plenty of examples already of where we're, you know, stuff that was just people grinding through documents has gone away. Um, we can do that. You get the ability to improve institutional knowledge by having these repositories with an agent to guide you on it. So, we're improving the knowledge across the organization. And then ultimately at the end of the day, it's about improving our organization's ability to change. So as we have to adapt to new things in marketing, new ways of selling, new service paradigms for our customers, can we be ultra agile? AI really gives you that in all those areas. It lets you organizationally and functionally flip on a dime if you're set up right and if you understand what you're doing, you know, in your workflows, it is really, really powerful.

Charlie Fleet:

Yeah. I, Eric, I think hiding in each of your examples for me is the why now. There's some internal or market event that you have to react to. You know, we've got to hire for these people. We've got to figure out how to win this customer. Um, we came up short on our, you know, our our earnings that for the quarter, some some event like that. And and the response is, well, the tools I've had in the past aren't solving the problem. Maybe some new to new tool sets are going to work. And that's where I'm seeing companies turn to to AI, but some of them are self-generated where it's like, well, we just spent $55,000 in tokens with Claude last last month. Did we did we get anything out of it? And so you it forces you into this governance mode where you're like I I I actually don't know, we we need to go actually put... You know, that would be the event that's forcing the focus, the why now.

Pete Behrens:

Do you think it's obvious? Like, do people get it or do they look at you and just say, "Just because everybody else is doing it, we should do it." Isn't that just peer pressure? I mean, it do you get do you get death stares or or like ignorant responses? Like I'm curious how that plays with the people in your organization.

Charlie Fleet:

I mean, I can't speak for Eric, but like the people I talk to already have interest in AI in general.

Pete Behrens:

Okay.

Charlie Fleet:

It's a selection bias.

Pete Behrens:

All right. So you so you've got an organization that curiosity is there and interest is there. Eric?

Eric Kihn:

Um, I'll give you two answers. So, you know, there is a if there's a demonstrable ROI. So, we had a thing where we were outsourcing some kind of document processing. We're spending $20,000 a quarter on these getting these documents processed and we now spend $20 in tokens instead. Right? It's the same quality of result we're getting it back, man, all of a sudden you know the C-suite is like wow, that you know, win-win. The other side of people that are kind of unhappy, it turns out one of the most surprising things to me is when we say hey we're here to free you from your burden of administrivia and processing documents and grinding through stuff, a lot of people like that, they like that, that is their job and they don't feel freed, they feel you know the death stare or whatever, like, "Hey, guess what? We figured out how to automate, you know, all those forms you had to do last month. Isn't that great? You'll be able to do higher thinking and strategic visioning and all this stuff." And they're like, "No, I like that. Asking that. Job." So, it depends on who you're talking to a little bit.

Pete Behrens:

Yeah. And and that's certainly, you know, you think about the human dynamics and, you know, we've we've studied culture and organizations for, you know, a few decades now. And I'm not one of those people, but you've got a lot of people in the organization who like predictability, stability, control, and you're taking in many cases all of that away from them. And you're basically pulling the rug out from under these people who have done this for 10, 15, 20 plus years. Now what?

Eric Kihn:

That that that was my honestly that was our message going out when I originally got here was to tell everyone, hey, we're here to free you from your drudgery and all this this grinding administrivia you're doing. And we found out that really landed poorly, that they a lot of people do not want to be freed from that. That that is what they do and they enjoy coming in and doing that in their office.

Charlie Fleet:

I think Eric, there's another layer to it, which is um in the news cycle in the last two years, a lot of these big tech players when they do major layoffs, they sell it as an AI layoff. I don't I don't I don't buy it. I don't buy it. But they sell it as that. And so when when someone comes in, I'm going to save you from your your your you know difficult task so you can do higher order thinking. They go, "Yeah, that I don't buy it. That's code word for you're going to get rid of my position." Um and I've talked to to enough people that I that that's the real pattern whether it it happens or not at our companies. It's happening up. It's a fear that it will happen.

Pete Behrens:

Well, we're getting into the second piece uh of the Catalyst Canvas around this goal, which is obstacles and the thing, you know, the things that are going to slow this down, the friction, the impediments. I'm curious as you look at your job or as you've experienced your job and Charlie you've now been in one and I'll say one and a half since you just started the second and Eric you know you've been pretty deep for for over you know a year now. How would you describe maybe the top three obstacles? And maybe I'll let you kind of do ping pong here. Pick one that's in your top three and pass it off to the other here.

Charlie Fleet:

I'll go first. Too much work. Too much work in process. The the the mantra is we it. We need to do it. And so the organization spins up a thousand things which is a good problem, good problem to have, that curiosity take take hold. The the problem then is what do you do with it? How do you know it's working? How do you find your bets? How do you get organized? And I think that follows the arc, Eric, that you were talking about, governance. Um, so so I that's my first one is just work in process and grab a pen.

Pete Behrens:

So Charlie to connect that one I mean we have a small handful company, right, we're really small, we work we work and scale through colleagues, but even our little small company between a few of us working on product I see that happening with AI like it speeds up everything. And so all of a sudden, you know, if you have a weakness in your process, if you have a weakness in your review cycle, a weakness in your governance, AI just blows right through that and all of a sudden you've got 20 things that have that same problem. And we're trying to do so much because we can go so fast.

Charlie Fleet:

Mhm.

Pete Behrens:

Yeah, that's awesome. All right, Eric, what's what's on your top three?

Eric Kihn:

Uh I'll give you my number one since we're ping ponging it. Data that's not AI ready. So ultimately you have to build your AI from the data up and you know whether it's your sales data, your engineering, your telemetry from your instruments, whatever, having you know we have a treasure trove of data but it's if it's not AI ready, meaning you don't have good metadata for it, you don't have a a semantic layer, you don't have you know if you don't have an understanding of what that is, all of a sudden you have people interacting with raw ugly data and AI is going to do something and it's going to do it quick and then what do you got? And you know we we have the dream that you know our president on a Sunday night when he's watching football could sit down and ask any question and in the old days that meant calling up a business analyst who then would get in, create a dashboard, do the data. We want that to not be, but to do that you've got to have AI ready data. If you don't, you're just asking for trouble. Because AI is super democratizing and that who can ask for what, who can create plots and applications and tools. But if you don't if you don't understand what that is underneath that you're working with, I it's the number one problem I face for sure.

Pete Behrens:

Well, Eric, and I know we talked worked together for a while and and you come from a background of, you know, government climate research data, climate data, space weather data, and you know, you talk all about the the model is based on the effectiveness of the data. And if the data has black holes, then our analysis has black holes. And uh and so it's interesting that you know I know your job specifically as you've been promoted has brought data under your wing as important or as an equal player would you say to AI.

Eric Kihn:

Yeah, I mean it's it's definitely equal and I I really appreciate the vision they've had here to see that that is the case, right? If it AI can't be better than the data that drives it and for almost all the companies, the LLMs are fascinating. They can handle language, but mostly you're asking questions about your business, your domain driven by your data. It will be no better than you know the state you leave that in. So having what we call triple A data, authoritative, available, accessible data, um really changes the game. And once you do, it's amazing what AI will do for you. But if you don't, then the real problem is it tries so hard to give you an answer no matter what. Whereas maybe in the old days, a business analyst might come back and say, "Uh, we can't answer that question with this. You know, it's it's insufficient. It's broken. It's missing records. I'll give you something." And then what are you doing? So yeah, I it is it is I think very visionary to put those together because no good data, no good AI.

Pete Behrens:

Very good. Very good point. Charlie, uh, back to you or comment on his data.

Charlie Fleet:

Yeah. Know, I think if we're just going to keep ping ponging, my uh second one is really around tool tooling. Um and this is really around well which which provider are you using? But there's so many kind of splinters of this it's like um well which do we pick? And then it turns into a religious conversation uh you know around which we should be using and then the lock-in that's afforded once you pick one and since things are moving so fast fast it it presents a problem and and especially then smaller tools show up you know like Salesforce strategy was a VP with a credit card, that was who they were selling to. A lot of the AI you know capabilities today you'll see departments go oh well I used I need this capability and I just paid $15 a month and I was like, what did you upload to it? Uh, and so we're starting to kind of see this, it's just still early days and and and because it's so unsettled, I don't think it's I don't know if it's slowing us down, but it's friction. And so speed up, slow down, it's just a lot.

Pete Behrens:

I mean we could go back to the agile days and there's this constant battle of well is it Scrum or Kanban or you know you know I think this concept of tools and theology and what's right often gets in the way of just doing something and so it's interesting you're feeling that with AI. I I know talking to Eric you have a little different strategy on on tooling. Maybe you could share before you jump into your next obstacle.

Eric Kihn:

Well, actually my next obstacle fits very nicely with that one and so I'll just use it to build off it.

Pete Behrens:

Okay.

Eric Kihn:

So I this is one pilot success that doesn't scale. So when we have people bring pilots in and they're successful and they show wow I could do this amazing thing with AI but then you know they're not software engineers. One of the again one of the things about AI is it's very democratizing. You don't have to be a professional software engineer but do you understand scale? So one example is we had a tool that was to help sellers you know manage our, we have a 100,000 products, manage that portfolio. But and it was like, "Wow, this is great." Then they started pushing it out. Well, it burned through all our tokens like allocation for the month in a week, right? Just that one application. It didn't scale. And so, and you know, when we got in there and looked at it, we're like, "Oh, it's reindexing hundreds of files every time. No software engineer in the right mind would do this." But it wasn't a software engineer that built it. So, you know, and to what Charlie was saying about, you know, the VP brings in a new tool and says, "Hey, I saw this at a conference. It looks great. Put it in play." It's almost the success of these then causes more problems. It's an obstacle because then you kind of have to fight it back and say, "Well, it's not really going to work across, you know, a global corporation."

Pete Behrens:

It reminds me, Eric, back in the day when we were working with your scientists and they'd write these little PHP models or list models and and then they'd work great and then your IT team's like, "Well, that requires this, this, and this, and it doesn't run on our servers, and it's not safe and secure." It sounds like you're at that same problem again, but now it's not just scientists. It's like everybody in the organization's a coder.

Eric Kihn:

Wait, that's it. I mean, and we're try, you know, we're trying to figure that balance in our in our governance because we want people to be engaged and want them to feel empowered, you know, to work with the data to create new models, but also, you know, there's some basics of software engineering scalability. What do you do about maybe you just prototype it and then how do we hand it off to a process to bring it into to global beings? So, but just the success of those pilots gets people so excited and they're like, well, why can't we just roll this out? Well, there a lot of reasons.

Pete Behrens:

Yes, that's that's gets into that space I think we talked about earlier which is can I hand over control and decision-making yet? Uh still some problems there without some oversight in that.

Charlie Fleet:

Yeah.

Pete Behrens:

Yeah, interesting. Charlie, comments on that or jump to your next one.

Charlie Fleet:

Uh I mean just amen. Amen on that. Yeah. The... Okay. So, my third one is um AI is still kind of like those 1980s um arcade games. You know, you could play you could play Pac-Man and they said you could put two players in, but you each played after one another. It was never a two-player game. And I fundamentally AI is still a single player game. The ecosystem is still fundamentally a single player game. People are figuring out how to make it multiplayer, but all of the ma primary interfaces and the way that you use it today prevent you from true team collaboration. This will change as as we figure it out, as it scales over time, but it's a real barrier. And so we have these acts of, oh, I can write a skill and I can share the skill and you... Then they have to download. It's it's kind of like I you would write an app back in the 2000s and then send you the .exe file and then you had to download the .exe f on a floppy disk and a five and a quarter. I mean, you know, it's just and it that's how it feels. And so this is just this is temp this is temporary, but it's a barrier nonetheless.

Pete Behrens:

On diskette. Yes. Well, and I think if you take that and you start to extrapolate, right, do people become more isolated? Do they start going down directions and rabbit holes that you don't catch right away until now all of a sudden not only they've burned, you know, 10,000 tokens or 100,000, whatever, but they've spent three days down this thing that wait a second, this is happening over here, right? So imagine that insulator isn't just a problem about sharing a tool, but also productivity and alignment and other things.

Charlie Fleet:

No, absolutely, Pete. And if you don't want to, you know, you could fall down a rabbit hole of like these interfaces are designed to get you to use tokens. So the more it can reinforce you to try and solve the problem in that chat as opposed to stand up, talk to a colleague or bring a colleague into that conversation. So it's a two-on-one that would work against token usage.

Pete Behrens:

Yeah.

Charlie Fleet:

So I again I don't know why it happens but it certainly seems like it's just not something commercially the providers need to optimize for yet.

Pete Behrens:

Yeah. They're not incented to that yet. Eric, comment on that and in your last uh obstacle.

Eric Kihn:

Well, I I thought that was really insightful actually that it is like, you know, playing Pac-Man. You're both on the same game, but your turn, my turn, there's not really, we're not together scooping up the little dots. We're, you know, on separate screens doing our own thing. And it is really kind of isolating. I I think it also tends to make people, you know, more isolated because you feel like you're interacting with something. You feel like you're getting some feedback and you don't seek out whereas, you know, the patch might say, "Hey, I'm seeing this in the data. It's kind of weird. Just ask AI and it says it's fine." So, I think that's really insightful. Um, my last obstacle is that governance is interpreted as bureaucracy. That everything we try to do to put in some governance and show safe paths and help with sustainability is interpreted as bureaucracy and the fun police arriving to, you know, ruin the day. And that's so hard because I I've been on the other side of that, you know, in my career where it's like, oh, here they come to shut down everything. But so we're really working hard to understand how to present governance as, you know, a bulldozer that's making a smooth pathway that if you get on it behind us, you know, you'll have some roadway. And it's hard because you know, just feel like it's it's squashing the fun out of it. So, uh, we have to be really careful in how we present that.

Pete Behrens:

Yeah, it's a delicate balance of, you know, speed and safety and ethics and innovation. I can imagine there's a lot of tensions being pulled uh in in that journey. Well, let's jump to the third of the four around the the goal here in the Catalyst Canvas and that's measuring. And usually the hardest thing when we start thinking about goals is how do you measure this? And we typically look at two types of measurements. Leading and lagging. Leading means okay it's a it's a predictor that it's it's going to mean better outcomes. And then lagging is it's actually a better outcome. So I'm curious how you're being measured as individuals or how is your work being measured in in a way that says we should keep you around next year because you're actually doing good in this in this world. So maybe both from a leading lagging kind of indicator here, how would you approach that or how are you approaching that?

Eric Kihn:

And I'll go a little bit. So, one of the easiest ones is adoption, right? So, we're doing adoption metrics around how many people are using AI. Do we see increases in Copilot use? Are are users moving moving from curiosity into routine application use? So that it's a good metric because it's easy to count, right? I can see I get a report every week about how many tokens were spent and all that. So adoption is a big...

Pete Behrens:

So your enterprise models are sharing basically they can see usage per person.

Eric Kihn:

That's right and we can see how many people requested the full Copilot license and we're like okay we're getting we're getting the word out there. Next thing is business outcomes kind of, you know, what are we seeing in terms of improved quality numbers, time to complete tasks, that sort of stuff. That's another one that we can measure and again that's a lagging indicator, right? So we're seeing these things are out and in the business...

Pete Behrens:

Yes.

Eric Kihn:

So we're getting it. The next thing that I think is our... I'll go ahead.

Pete Behrens:

And well, stop there a second, Eric, for just a second. Leading indicator really easy to connect directly to AI. Quality and other outcomes, my guess is a fuzzier connection to AI.

Eric Kihn:

Yeah.

Pete Behrens:

Are you just assuming a connection or are you doing anything to make that connection?

Eric Kihn:

I mean we're in luck in that there's a lot of areas where they do have good long-term baselines of like quality standards...

Pete Behrens:

Okay.

Eric Kihn:

...you know, let's say with particular instruments and returns and things like that.

Pete Behrens:

So if we have trends that that's a better connector.

Eric Kihn:

So if you bring in AI at a point and you start to see it, I mean, can you attribute it all to the AI changes? No, because we're constantly evolving process and we're, you know, an organization that's into constant improvement. And so there's parts of that, but if you see some kind of delta, that's something you can measure across many parts of the organization.

Pete Behrens:

Okay. And you were going to do one more, Eric.

Eric Kihn:

Well, the big thing for leading indicators is our portfolio health. So, we look at how many use cases are coming in. We have a series of tollgates for them. How many are getting through the tollgates? Um, you know, what are we seeing duplicate things coming in? So, we're starting to get the word about what's out there in the portfolio.

Pete Behrens:

When you're saying portfolio, you're talking about products or you talking about AI pilots and usage?

Eric Kihn:

Um, the AI portfolio. Yeah, the AI portfolio. So we keep a collection of all the use cases that are coming in and see are those growing? Is the time to get through tollgates to operations quicker? Um, that's the stuff that's leading like hey we're getting better as an organization at understanding this stuff and getting it through the system.

Pete Behrens:

Okay, cool. Uh Charlie, how would you how are you approaching this these measures?

Charlie Fleet:

Well, first of all, I want to say as someone who's only three weeks into their their new role, um, and I'm proposing an AI portfolio. So, I'm glad to hear Eric, you have one because it was the obvious next step for me as to how do I actually, you know, put in place any kind of measurement system. So, that's good. Thank you for for that uh a vote in favor of of my proposal. Um I I would say the this comes back to the the crawl walk model for me for measurement. The crawl is just hey um I've been able to to you know measure something like I'm just like I I've been able to use the tool and I'm and I'm I'm there. The walking is hey I can see that I'm producing you know more work for the same amount of time or I'm processing more tickets.

Pete Behrens:

So you might you might use a survey or something for that to get an employee opinion.

Charlie Fleet:

Yeah. And I mean this is really getting closer into like value stream mapping of like how many tickets do we process in a in a in a month. How many people do we have and how many required manual touch? What was the average time? Th those are things which you can start to measure. But it's hey actually this department has fewer headcount despite a 20% load of tickets or 60% of our tickets were redirected to auto-resolution by you know because of our AI chat chatbot.

Pete Behrens:

Yes.

Charlie Fleet:

So we didn't hire under the forecast that we expected, we were able to redirect that spend toward a more valuable part of the the organization. That's when you're running. And so the don't get real and I think you know my role really is is is valid until you're getting at that run level because anything before that can get lost frankly. It's like oh I you know I saved more time, uh I didn't have to write that email, great, what did you do with the saved time? Did you go smoke a cigarette? Did go play a video game? Go play Pac-Man? I don't know. Uh the others you can't... Those those are real.

Pete Behrens:

Yeah. The elusive productivity measurement. Yeah.

Charlie Fleet:

Exactly.

Pete Behrens:

It it kind of reminds me of the the you know, yeah, we invent vacuum cleaners, yet we still spend all of our time, you know, cleaning, right? We we invent all these things, but we still spend a lot of time doing these things. Yeah.

Charlie Fleet:

That's right.

Pete Behrens:

Well, the last part of our conversation, I'm going to skip over actually the last part of the Catalyst Canvas, which is next steps. Um, it's more tactical thing. I want to zoom out a little bit on leadership. And you know, Eric, you mentioned something back at our podcast two years ago now. And I think and I'll I'll read this quote. It said, "You said in episode 59, in five years, you will if you are not bringing AI into your leadership toolkit, you are going to be run over." I'm curious how you see that today. Do you still believe that? Do you look at it differently? And then Charlie, I'll let you respond to that as well.

Eric Kihn:

Yeah, I definitely still support that. I mean, I think there isn't really a leader that's not using AI to help with their staff emails, their calendar management. There, especially when you get to that high leadership level, time is such a precious commodity and every minute is scheduled. And, you know, if you're not using AI to help you with that at this point, you're just falling behind your peers and you're really doing a disservice. I I think I would sharpen it today. The the issue isn't whether leaders personally use AI. I think whether can they can redesign their organizations effectively to use machine intelligence and embed it in their workflows and it's it's a great question because that means they have to understand the true workflows in their organization. They have to understand where AI can integrate with it and where AI can integrate with it changes every three months. Like earlier in this discussion we were talking about uh today it doesn't do something well. Well, you got to check back in at three months or six months for sure because it might integrate with that part of your workflow very well. So, as a leader, the fundamental thing now is can you create an organization that understands its workflows and how you embed machine intelligence in with your team. That's what's going to give you the competitive advantage.

Pete Behrens:

Yeah. So, it sounds like you've shifted from that personal view of leadership to leadership's responsibility. Yeah. Interesting. Charlie, how would you respond to his... Yeah.

Eric Kihn:

I think it's changing, well...

Charlie Fleet:

No, go for it, Eric, 'cause I have what I want to hear already.

Eric Kihn:

I think it's changing a lot of what the leadership tasks are and when we work together on a project about AI and leadership I always had the question what is it that leaders actually do, like what is what is it to be a leader? I mean there's some motivation, there's some you know prognostication, there's all this stuff. And AI touches all that like you know are you using AI to help you predict the future, no longer just you know being the big brain that that's divining it from somehow, but are you a data driven organization now when it goes to you know future hiring states, are you visionary enough to know what key elements of your workflow will always be human and that you've got to find people with those skills which may not be the traditional skills. I think it's just so fundamentally changing what's going to be an excellent leader. It has and is going to going forward.

Charlie Fleet:

Yeah, I'm going to go. So I think the irony of being at the executive level of any company is that your work right now is a little less compatible with really using AI um than the people who are deeper in the organization. And so as a result there's this divide. Think of a CEO, right? They have to, they're all about connection. They're all about... and we just talked about AI being a single player game. Can use AI to advise or to help them take notes or to help them connect, but fundamentally that's just a job aid. They're not using it in the way that a lot of the organizations are getting real value from from AI. So, so they spend their day not getting to learn about AI and use AI in a way that the rest of their organization gets to. So there's a fundamental different perspective that they have of what AI is and how it affects the organization. So it I don't know what it's called. We're we're playing around with this ELT road show, but this idea of like the executive team needs to go do some, you know, some gemba, and they need to actually get out and see what it's like, and they need to touch, feel it in a way that's different, because if they don't, it won't be as connected as they need to be to properly sponsor and, you know, drive um the abilities organization to grow.

Pete Behrens:

I love it. Go touch grass for a bit.

Charlie Fleet:

Yeah.

Pete Behrens:

I'm gonna... We're at coming up the our end point here. I'm going to leave you ask one final kind of question uh for leaders as you're talking to leaders. What's one thing you think leaders have to give up in order to achieve the goal you're trying to get?

Eric Kihn:

The thing that jumps out to me is I think leaders have to be have to give up the comfort of being the person with the answers. More and more the knowledge of something is less and less valuable. And you know if you're doing a good job of documenting it, pushing it out, everybody can have access to it. So being the one that knows everything because you've been there 20 years or you have experience in the field, it's a really comfortable thing to be the one that has that knowledge. Instead, you have to become the architect of a system that learns and adapts and you know understands what it's trying to accomplish with AI. So I think for me the biggest thing is just being comfortable not being the person with the answers but really you know how do you architect a system? And I I think that's a big ch...

Pete Behrens:

It's awesome. And we call that catalyst leadership from expert leadership. But yes, it's a it's a great skill. It's a great skill to have, AI or otherwise. Charlie, what's what's one thing you say leaders need to give up?

Charlie Fleet:

I mine is much poor much more poorly formed as I don't know how to say it, but I'm going to try anyway. I I think leaders have to give up or or trade less in perfection and and details. This this idea of perfection and spend more time in, and I just wrote the word down, intent. It's all about intent. Intent of what do they need the organization to do? What was the the what was the intent of this work product? Because if you get caught up on the details of something was created by AI as you trade emails, you'll get lost. But what the intent behind it and how do I provide intent to the systems, that that's the fundamental, is is focus getting rid of anything that isn't around intent.

Pete Behrens:

Yeah, perfect is the enemy of progress and you know great's the enemy of good or good basically trying to be, you know, it's going to require an iterative agile kind of nature to to move in this world and and you know I talk a lot about that in in the book, so appreciate appreciate appreciate you mentioning that as well. Well, Charlie and Eric, I appreciate your insights here. Um, I appreciate your willingness to share those insights with our community and, uh, I will offer to those listening or watching. Uh, we will put their contact information if you want to reach out to them on our website and I'm sure uh, they will be more than happy to respond. So, thank you.

Charlie Fleet:

Absolutely.

Eric Kihn:

Yeah, thanks for having us.

Charlie Fleet:

Please be...

Pete Behrens:

Well, thank you both. That was awesome. Appreciate uh the the insights and the comments and uh anything you wish you would have said you didn't say.

Charlie Fleet:

No, I loved the ping pong. That was fun.

Eric Kihn:

Yeah, that that worked pretty well.

Pete Behrens:

Yeah, you guys you guys played well. Nobody dominated and spent too much time. So, I appreciate appreciated that as well.

Eric Kihn:

You're welcome.

Pete Behrens:

Um but I will be in touch with you guys uh once this gets out and as well as letting you guys as much as you want um contribute to the article or as little as you want and and we'll go from there. But um but thank you. This was awesome.

Charlie Fleet:

Yeah. Thanks, Pete. Good to see you, Eric.

Pete Behrens:

I'm Pete Behrens, thank you for listening or watching.

Relearning Leadership is the official podcast of the Agile Leadership Journey. To learn more, visit relearningleadership.show.