Why Trust Is the Missing Link Between AI and ROI

Trust in AI: How Do You Build Trust That Creates Real Business Value?

Companies are pouring billions into AI, but investment alone doesn't deliver value. In this episode, Ben Parker sits down with Paul Drennan to unpack why trust is the missing link between AI spending and real business outcomes. They explore what builds — and breaks — trust in AI systems, how organisations can balance encouraging adoption with maintaining healthy scepticism, and why technically excellent AI can still fail without user confidence. From managing hallucination risks and human-in-the-loop decisions to measuring genuine ROI, Paul shares practical steps leaders can take within 90 days to move from AI experimentation to measurable, trusted results.

Timestamps:

1:59 — Quick introduction: Who is Paul Drennan?
4:25 — How much of the AI value problem comes down to trust?
8:26 — What needs to happen before employees and leaders are comfortable relying on AI?
13:13 — Can technically excellent AI still fail without trust and adoption?
15:35 — What are the biggest things that cause people to lose trust in AI?
16:34 — How should organisations manage the risk of convincing but wrong AI answers?
20:36 — Finding the balance between encouraging AI use and questioning its outputs
23:42 — Where should human judgement remain as organisations automate more decisions?
29:22 — What is the relationship between trust, adoption, and ROI?
33:12 — How should leaders measure genuine business value from AI initiatives?
34:40 — What should organisations put in place across technology, data, governance, and people?
43:25 — How should organisations continue testing and monitoring AI to maintain trust?
47:42 — What are successful organisations doing differently to build trust and adoption?
53:17 — The first meaningful action to improve trust and ROI in 90 days

Connect with guest: https://www.linkedin.com/in/pauldrennan/ 

Connect with host: https://www.linkedin.com/in/ben---parker/

Thank you for listening! 

Transcript

Copy Transcript Copied

guest 0:00

we're asking people to essentially rewrite their job description using a tool that they don't fully understand, but they're still accountable for what happens. And the opaque nature of AI makes that a pretty steep hill to climb. I mentioned before that at the end of all of this, we're asking someone to rewrite their job description, which is a bit of a leap of faith and what we need to do is we need to turn that leap of faith into more of a hop of faith. Confidence and correctness are two different things, but we aren't always great at noted-- knowing the difference. Traditional software fails loudly, right? You have a bug, it causes a problem. It's usually pretty noticeable. AI, I'm gonna say, fails with a straight face. It always sounds confident, even if it's making things up. Trust yields adoption. Adoption yields confident users. Confident users create ROI. I think that's the path.

ben parker 1:15

Hello, everybody. I'm your host, Ben Parker, and welcome back to Data Analytics Chat. Today, we're going to be talking about something almost every organization wants from AI, the magic value. Companies are investing heavily in AI, obviously launching new tools, running pilots, but investment alone doesn't guarantee ROI. One of the biggest questions is whether people actually trust the technology enough to use it, rely on it, and obviously, change the way we work. So today, we're gonna explore the relationship between trust, adoption, and ROI. Joining me is Paul Drennen. Welcome, Paul.

guest 1:56

Hi, glad to be here.

ben parker 1:58

Cool. I guess before we dive in, do you wanna give the listeners a quick introduction to who you are?

guest 2:05

Yeah, sure. I'll keep that brief. I have recently retired from my corporate career after about 30 years of building analytics and AI practices in a number of industries, most recently in insurance, and now interested in helping people navigate the change and understand what it takes to really be successful. And I'll say that after all that time, the, the main lesson for me is... and you already hinted at it, Ben, the data is always gonna be a challenge, but the real work is in building trust with users, and then once you've got that trust, how do you keep that trust once they've integrated the solution into their work process? So that's really what I'm about.

ben parker 2:52

Oh, I I'll see you touched on that. So why do you think it is trust is the issue? Is it just lack of like data literacy or is it just change? What do you think is the sort of key component

guest 3:06

here? There's a number of different factors going on. You mentioned literacy. I put that pretty high on the list that people's awareness of what's going on inside these algorithms and these platforms that have been created isn't very strong, and we're asking people to essentially rewrite their job description using a tool that they don't fully understand, but they're still accountable for what happens. And the opaque nature of AI makes that a pretty steep hill to climb. And if we don't climb it and we don't get that trust from the user, there is gonna be hesitation and at a certain point, every time it makes a mistake, there's a bit of a scorecard that's being kept by individuals and again, that the fact that people are still responsible for the outcome means that they really want to believe in the tools that they're using, and creating that transparency and trust is difficult when the algorithms are opaque and analytical literacy isn't always the highest

ben parker 4:21

I guess so look, let's focus on trust. So c- companies now are investing more and more in AI, but I guess still struggling to create meaningful value from it. How much of that problem comes down to trust itself?

guest 4:37

I think most of it, honestly, the investments that are being made right now are primarily in building and establishing the platforms and hiring the engineering talent. Those things are important, but the cost of AI, and I'm gonna be inclusive here, Ben, of traditional AI as well as these frontier model-based tools. The cost was never in the build. It was always in the maintenance of it and the monitoring and the drift detection and the repair that hap- has to occur, and that was always where most of the effort was. And now with these frontier models and the fact that they're not deterministic in their outputs, trust maintenance is actually harder than it used to be, and the skills required to do validation are... They're not widespread. There's a high opportunity cost to using your subject matter experts to test the AI's output. So I feel like the disconnect is everyone is chasing the technology and wanting to make sure that they are on pace with the world and that they've got the right tools and platforms and talent, but building an AI is not that hard. Owning one that you're proud of in the future takes a lot more effort, and that disconnect, I think, shows up in the high cost of development today, but the relatively low rates of adoption and and consequently impact on business processes.

ben parker 6:27

Yeah. So what you're saying is it's a lot of emphasis on more the, the technology instead of like people that actually build it. 'Cause I guess now you... I guess the value you're gonna get is people that actually know the domain. 'Cause obviously you've got the technologies there now, isn't it? It can do a lot of the heavy lifting. It's actually g- I guess having the, being the middle person to actually under- I guess blend it all together. Is that do you think where businesses should get more value from? I

guest 6:55

do. I think we're treating this AI emergence primarily as a technology deployment, and I think it's really more of an operating model change and the focus on the creation of AI you certainly have to do that, but value isn't created when I build it. Value is created when I deploy it, and in my definition, deployment means I have a set of confident users who have now integrated it into how they do their job. And that's where the value gets created. And so much of the distance between a f- a, a frontier model which is becoming highly commoditized now, and we're all renting the same models from the same vendors. So there's nothing very distinctive about what's happening in the LLM. All the distinction is happening in the rules and the wisdom and the experience and the judgment, those things that our best operators, our best decision-makers have. And how do I encode that so that the frontier model and that context layer are partnering well? And if I do that, I'll earn trust. And if I don't do that, I won't. And if I don't earn trust, I'm not gonna sustain a population of confident users for very long

ben parker 8:20

So obviously you're highly regarded in the market. I spoke to many people that, yeah, really rate you, I'll be honest. What, what happens where before employees or leaders become comfortable reliant on AI as part of their day-to-day work then?

guest 8:34

So this is a big question, and I think there's a number of things, and I often use a gardening analogy where I think about I have to prep the soil, I have to plant the seeds, I have to nurture them, I have to keep the weeds out. I have to do all these things and so this preparation is important. I mentioned before that at the end of all of this, we're asking someone to rewrite their job description, which is a bit of a leap of faith and what we need to do is we need to turn that leap of faith into more of a hop of faith. And so some specific ways that I have used that have been successful is a couple things. So one seems obvious, but you wanna have the users and the builders co-invent the solution. When the user feels like they've had equal authorship over what's being built, then their path to adoption is much easier the models are opaque, but you can make many of the moving parts as transparent as possible. And so you want to make the system visible and transparent so that people understand how it works. There's a moment where the algorithm is gonna do something that's difficult to describe, but before and after that, we can make things very transparent. I got a lot of usage and value out of what we called the, the blind acceptance test, where we would score something and then show it to our experts without telling them what the model had said, have them reach their conclusion, and then reveal how the model compared to that. And, if you're lucky, it performs very well, and if not, then you iterate until you get to a point where the user believes in the quality of the outcome. And then the last piece, which I think is really important, and I'll probably say this more than once while we're chatting, once I've built it I really have a lot of duties that come with ownership, and AI is going to drift. It's trained on the past. The future will look different. As that difference widens, the confidence we have is going to erode, and you have to watch it, and you have to repair it. And I believe that at a certain point, as a user, I wanna know who's accountable for that activity. So I wanna know who is watching and who is going to take action when it drifts. Because without that warranty, the drift is accumulating, and it's happening invisibly. And at a certain point, I'm gonna start getting answers that don't make sense. And once trust is lost, it is very difficult to retain. So those are the things I think you have to bring people in. You have to create transparency. You have to put everything on display that you can, and there needs to be that warranty that says, "We're not just gonna build it and chuck it over the wall and hope for the best. We're gonna stand behind this." So I summarize all that in, in my practice as the safe to own promise, which is more comprehensive than a safe to buy promise because, as I said before, the real effort is in the long-term trust maintenance and ownership of these assets, and so a safe to own promise comes with those things.

ben parker 12:13

Yeah. So I'm guessing s- we need leaders to be good coaches, really, to do this constant education for inhe- or I guess helping people understand AI really then?

guest 12:24

Yes and we're all on a learning curve together. Even the professional data scientists are having to climb a learning curve with these new tools. So acknowledging that we're all learning together I think is an important fact. And a willingness to try, a tolerance for the iterations and cycles as we calibrate, and then studying how the machines are built, what they're good for, what they're not good for. I think inoculating ourselves a bit from the hype so that we can navigate more practically all of those things and leadership has to not only sponsor that, but actively participate in that

ben parker 13:09

So then obviously looking at technology c- can a AI system be, like, technically excellent but still fail to create value because people simply do not trust or adopt it?

guest 13:23

Totally. I've seen that before. And before, before the current frontier model-based AI in my own practice, one of the things I've talked about monitoring, so we paid a lot of attention to the shape and value of the data that was feeding the algorithm that we had trained. We spent a lot of time monitoring its outputs against references. But we also added a we called it an engagement rate, which said how often is the user agreeing with what the model is recommending, and we trended that over time. And what I have seen is models aren't perfect, and so every now and then they're going to recommend something that the user disagrees with and if the user is anxious about it, they're probably keeping score. And at some point it's gonna give them an untrustworthy response one time too many, and at that point you're gonna lose their confidence. And so I have seen algorithms that were performing well, the monitoring results were strong, but where I was losing the confidence of the user, and that will end value creation just as much as a model that has gone off target

ben parker 14:40

So would these, is changing behavior sometimes harder than actually building the technology, Tim?

guest 14:46

I think that is the hardest part of the job, honestly. You have to earn trust. You have to convince someone, "Here's what's really happening. I'm building a tool for you. I'm asking you to rewrite your job description. I can't fully expose exactly how the tool works." That is a difficult request And Yeah. That's just a really difficult request, and if I don't manage that part, the psychology part of it well, and I don't convince my users that I'm standing behind it, I'm gonna guard their commitment and keep, ke- keep earning the trust that they've given, then I risk losing that trust. And as I said before, once it's gone, it's very hard to get back.

ben parker 15:34

So what are the biggest things that cause people to lose trust in AI?

guest 15:38

Accumulation of wrong answers that they are still accountable for would be at the top of the list. And look, I was doing AI before the frontier models, and we had algorithms that were deterministic. So I would put the same inputs in, I would get exactly the same output out, and trust building and trust maintenance was the hardest part back then. And now you've got algorithms where the same input can generate a different answer. And so that, that breadth of what is it gonna say this time, I think creates a, a, a bigger trust barrier. And as the answers come through and as that variance accumulates, it's not only that I disagree, it's the unpredictability of it that can be a real threat to my confidence

ben parker 16:32

Okay. Then I guess, AI can sound convincing even when it's wrong. Yeah. And, yeah, And AI, it can produce answers that sound extremely convincing even when they're wrong. And so I guess how should organizations manage that risk without making people afraid to use the technology?

guest 16:51

Confidence and correctness are two different things, but we aren't always great at noted-- knowing the difference. Traditional software fails loudly, right? You have a bug, it causes a problem. It's usually pretty noticeable. AI, I'm gonna say, fails with a straight face. It always sounds confident, even if it's making things up. And in most business contexts, we're gonna, we're gonna manage the context layer so that the ability to just make things up wholesale is very controlled. But even when it starts to drift off target, it's always gonna sound sure of itself. So I would say the, the fix is not we have to condition everyone to be nervous or afraid. You have to design safety into your system. And so again, this, this idea of ownership and warranty is, I think, central to the answer here. I really need somebody dedicated to context curation, not a part-time job, somebody who is familiar with the domain but also understands what it takes to teach the context layer what the truth is. Validation cycles with your subject matter experts, not as an ad hoc or crisis response, but as a planned workforce capacity, a protected duty that you build into the way you think about staffing. I, I created a job family years ago. It was one of the smartest things I ever did that we ultimately called an asset owner. But think of this person as a, a combination of the best sales engineer you've ever met, the best scrum master release train engineer you've ever met, but they own the solution. And note, I'm not saying the model, I'm saying the solution because the solution often has multiple components, but they own it and they're accountable for its performance. They're accountable for how the trend is going. They're accountable for scheduling repair sessions in. And then the last piece I'll point is, you don't wanna, you don't wanna scare people. You build those controls and you build that visibility into the full life cycle And then you invite scrutiny. You ask your users to give you honest feedback and then you have to respond to that hon- honest feedback with gratitude rather than defensiveness, because you want to encourage that honest feedback. And if we all understand that every tool we're building is on an evolutionary path, it's not perfect, it's gonna get better. If I don't watch it, it could get worse, but ideally we're gonna pay attention and we'll make it better. So there are gonna be issues, and you wanna have that combination of ownership in- invitation of expert validation, invi- invite scrutiny and feedback, and you wanna respond to the challenges with gratitude and a commitment to learning from those challenges and investing those learnings back into the solution so that it evolves in a positive direction

ben parker 20:23

Okay. Good. I like the, obviously, yeah, the outcome focus, 'cause, yeah, that's... Enda, you wanna be progressing, don't you? You want it to be running efficiently. So I guess, how would you... how should organizations encourage people to use AI while also making sure they continue to question its outputs when necessary?

guest 20:45

One of the trends that I see happening, and I'm not a fan of, is this kind of enterprise forcing function on AI that says, "30% of your code has to be generated by AI," or, "X percent of your work has to use it." I'm not a big fan of that. The future says, Paul's opinion, but I've said this out loud more than once, every job family is going to be AI-enabled in the future. So we want people to have hands-on experience, and we want them to start getting comfortable with what it's good for, what it's not good for, how do I get the best performance out of it? We want those things. But applying pressure to people to hit contrived milestones, I don't think is the answer. And you wanna have... we can't eliminate the human judgment, which is where the value's created. So I said that before, and I'll repeat it, like what's encoded in the front- frontier model is a commodity now. That's not where the value comes from. It comes from the wisdom of the people who have been doing this and have learned what works well, and you wanna get that embedded into a system where AI is a piece of the system, but AI is empowering the human to make better choices we had a lot of conversation about this, and one of the things that I actively championed was an enterprise-wide literacy campaign to get people hands on keyboard, but not just go try it, but to orient our entire workforce to how it works. Not down at the level of weights and biases and deep learning training. We don't need to go that deep. But what is it doing, and how do you influence the path it's following so that you can stay on target and not not get the hallucinations and those types of things? So I'm gonna say practical user literacy, not necessarily deep engineering or science literacy, and I felt like that was an important factor in starting to take the mystery and reduce that, black boxiness of it to where people felt like I understand how I can apply this now. And any program that does not take on the literacy challenge, I think, is just making the slope of the hill they have to climb steeper.

ben parker 23:29

Yep. You touched on an interesting point about human judgment then. So obviously, as organizations are automating more decisions with AI, where do we balance the... Where should human judgment remain? Do you think that's the challenge now?

guest 23:46

I think we're all still trying to figure out. We all understand and believe that human judgment is critical. Where does it show up? How do I integrate the tools and the judgment in a, a graceful way so that work gets done efficiently and we're proud of the output no one's asking the human to outperform the model at what the model was designed to do. One of the things that we implemented, and I think this was the right decision, was and honestly, this predated even the generative AI emergence. We only let systems have the authority to approve things. We never allow the system to automatically deny things. And I do think that as a hard rule was a smart one. I'm also a big fan of this idea that most of these solutions now, as I mentioned, are a combination of a frontier model and a context layer. And so investing deeply in how do we build, how do we sustain, how do we nurture, how do we clear the rot out of the context layer as a dedicated assignment, as a protected part of our workforce? If you do that well, then you really get the human in the loop at the foundation of your context layer, and that's where that judgment, it just flows all the way through the life cycle if we do that well. If we naively just grab a bunch of documents from a library and embed them or ingest them in a wiki or something like that, we run the risk of there being contradictions in that. It's drifting over time. The hum-- for me, the human in the loop adds a ton of value if I embed that very deeply at the front of this whole life cycle in the way we build and manage context. Then we allow the agents to be strong and have authority over easy yes outcomes. We don't let them have the complicated no outcome, and we engage the human judgment when things are ambiguous or if I'm making a decision that I'm gonna have to defend later. And so for me, human in the loop, human judgment, really powerful at the foundations of knowledge and in the ambiguous cases. That's where I think that's going to land. And I'll say it in just summarizing all of this, everything we've talked about so far, just one of the things that, that I said to my executive team and my board of directors is, at the end of the day, we wanna be proud of what we did, not just the day after we did it, but three years from now, I still wanna be proud of this thing that we created. And that's the kind of final exam score on all of this

ben parker 26:55

Yep. No, I agree, like I said, that you wanna, you want fa-- like you, it gets, you, if you get more value out of it, there's-- things are gonna, it's constantly gonna evolve, isn't it? So you need to keep building and it's obviously, it's one of them tools, isn't it? Technology where you need constant development on it. But yeah, you need, like you said, you need to be actually getting value from it so you... It's not every time you gotta start projects again or, 'cause it's just, that's, obviously that's where a lot of the investment's gonna go, isn't it?

guest 27:26

It is, and I think one of the, one of the things that we're all wrestling with is our familiarity with traditional software. And this feels so similar in a lot of ways that there's this sense of this is just a, an extension of what I'm familiar with. And I think superficially there's a lot of truth in that. But where it gets different is that in traditional software, I wrote a bunch of logic and the same activities produce the same outcome every time. And so I could have a team of people who built software, and then I could have a totally different team of people who maintain software. And and with AI, building is not where the hard part is, it's in the maintenance. And that is under constant daily threat of drift. And you have to know it, you have to recognize it, you have to repair it. You have to re-credential the solution with your users over and over again. And that part is very different And it forces you to think about your workforce and your capacity and the conversation you're having with users. It forces you to think about that very differently, which is why I said earlier the illusion that this is technology is, I think, strong. But honestly, in my opinion, this is an operating model change, and it-- in a lot of ways, it's forcing us to pay more attention and to be more deeply engaged with our users than we've ever had to be before, which I think is a challenge, but also a really positive thing

ben parker 29:00

Yeah. No I agree with you on the operating operating change 'cause, it's... You're embedding it across the whole business now, isn't it? It's just AI's, yeah it's critical for every business now. Every bus- every department wants as a business use case now, doesn't it? So I think it's, you're fl- flipping your business upside down, aren't you? And building again.

guest 29:18

Yes.

ben parker 29:20

So listen, I want to talk about trust, adoption, and ROI. So what's the relationship between trust, adoption, AI for an organization so it's, from AI?

guest 29:31

So I think it's a chain that runs in one direction, and if I don't have trust, I'm gonna struggle with adoption. If I don't get adoption, I'm not gonna get an ROI, and it doesn't really matter how great the algorithm or the tool we've built is. So you really have to start with trust. You have to earn it. You have to gain confident users, and then you have to protect the decision they've made to adopt over time. And as you do that, and as confidence grows, they'll integrate the tool into the way they do their job more and more deeply, and that's when the ROI will start to show up. And I mentioned this idea of a final exam grade and so I'm gonna say, my opinion, the final exam grade is not that I got a 30% productivity number, it's did they adopt it and were they still using it the same way or more a year from now? And if that is true, I'm gonna get the productivity numbers, but I'm also gonna keep the productivity numbers. And so I think there's a little shortsightedness at times where we set a target, and we work really hard, and we get to that target, and we declare victory, and we move on. And what I'm saying is, I want confident users more than anything else. I think having a set of confident users is far more valuable than a bunch of solutions and models that I, hit some target number on a report card. And the recap. Trust yields adoption. Adoption yields confident users. Confident users create ROI. I think that's the path. And if I lose trust, the whole thing collapses

ben parker 31:08

So do you think, everyone has a goal if they wanna get A to B. So do you think op- adoption is the missing link between AI investment and business value? 'Cause, 'cause it's... Adoption is the hard bit, isn't it? It's just you cha- if you're... Whenever you're going through a change journey, you are, like I said, you've gotta build trust with people, you've gotta do the change management piece, which is obviously changing people. Do you think that's the... Adoption is the hard bit for businesses?

guest 31:41

I do, but I would modify it, Ben, only to say I think it's sustained adoption. Because we can apply management pressure to a workforce to get them to hit a number on a report card. I want them to hit that number because they believe in it. I don't wanna give up. When management attention shifts s- somewhere else, I don't wanna go backwards, right? So it's really sustained adoption that matters And look it's hard to quantify some of these things, and we all want, a, a, a, a bunch of KPIs on a report card that I can generate and I can show to my leadership team on a quarterly basis and demonstrate the trend and so on. I understand why we all want this, but we're talking about people's belief systems and how that manifests in how they behave. The metrics are often a lagging indicator on belief. Belief is what I need. So I think we all have to be a little more comfortable with this isn't that easy to measure directly. But when I check in with my users and when I watch how they are engaging with the tools, how often they are agreeing with it, are they starting to abandon the use of it? Those things I think are how we measure whether people believe or not, and that's ultimately what's gonna be the most important.

ben parker 33:11

So how should leaders measure whether an AI initiative is creating business value rather than simply just cre- generating lots of usage or experiments?

guest 33:22

You certainly want some, some direct mechanical observations if you can get them what I don't think works is saying something like, "30% of your code has to be generated in AI." It's pretty easy for me to hit that target and have not really integrated the tool into how I do my job. So you have to be careful. I don't think usage measurement is going to translate into value very well. And we have lots of usage measurements out there because they're mechanically easy to produce. But I think that there's a gap between usage and value, and we're gonna have to pay more attention to engagement and how well has my workforce integrated the tool. I would put those higher on the list. And then I think we also have to accept that for a while it's going to be more qualitative than quantitative. And so on a qualitative basis, do my users trust it? And are they trusting it over time, back to my sustained confidence point those are the key things I think.

ben parker 34:36

Oh, interesting. What should organizations put in place across technology, data, governance, and people to build AI that employees and leaders are prepared to rely on?

guest 34:50

So I did a couple things that I think were pretty impactful and so you mentioned governance, and I will tell you that I had tried to rebrand governance internally to, to safe enablement. Governance often feels like friction and second-guessery, whereas safe enablement feels like an accelerator for me to get where I want to. So I I brought together a group of individuals who historically represented that risk of friction or that risk of the 11th hour challenge We all know and love them. I call this gr- group collectively the guardians. It's compliance, it's legal, it's procurement, it's risk management, it's architecture, it's security. To deploy a system, we have to gain their approval. And I brought the group together and I said I want you to join me on Team Yes. And Team Yes's job is to approve use cases and solutions, but not as a rubber stamp. I'm not asking you to abandon your duties. I'm asking you to help me encode what does it take so that you can say yes, because saying yes is what you wanna do." And so I think this shift from governance as a post-hoc inspection or evaluation, and instead invite that group to co-author the safe perimeter. And then as we got better and patterns started to emerge, we were able to document what that safe boundary set looked like, and use cases that were inside that safe boundary got approved without a lot of oversight, because we had already we had already published what the boundaries were. And that actually opened up the use case pipeline a great deal, and then the guardians could use their time and skill and wisdom to look at novel things, new use cases, new patterns, things we hadn't seen before, which is where they add the most value. So I'm gonna say on the governance side, excuse me, bringing the guardians in at the front of the process and having them co-author what I call that, that safe boundary set, that was really valuable. I've mentioned this idea of an asset owner before, and I do think that having a named person who owns the solution and is standing behind that safe to own promise over time is a great idea from an accountability and a visibility standpoint. It's also a confidence builder for users, and we've talked a lot about building confidence in users as really the, the key milestone here. And I think having that named asset owner was really valuable in that regard. Anything that we do in a repetitive way that can be engineered into the platform and the methodologies that we deploy is well worth doing. And, one of my taglines that we use was safety through design, not supervision. And so whether it's governance or technology, just taking recurring things and shifting left and being very deliberate about what they are, how they're supposed to work, what the safe approach is and engineering that into our systems and our methodologies I think that those are all really good things to do. And then on the data side, I'm gonna beat this drum really hard, Ben, but the context layer is where all of the value gets created. And so having dedicated, qualified, and protected capacity to manage the context is key. And so safety through design, not supervision, bringing the guardians in at the start of the life cycle, and really committing to that safe to own curation of context, those are the key moves, I think.

ben parker 39:18

So do you think that with projects before you ha- always had project coordinators to keep projects on t- task. Do you think that in data they need more, I guess p- like more people that do similar type of role that's got, obviously, to take the ownership, so they are like the glue. They make sure everything's gets the outcomes. Do you think that's what's missing or What do you think?

guest 39:40

Certainly when I made that a full-time job, it accelerated everything. And I'm gonna call back to this idea of the skills of AI have gotten much easier. The duties of AI have not, and in some cases, I think the duties of AI are actually more difficult now because of the non-deterministic nature of how the models work, and they're more opaque than they used to be. And so that, that accountability to not only get the, the solution over the finish line and deployed, but to stay with it throughout its life cycle and to safeguard its accuracy and be hungry for two things. As I gain operational experience, can I use that knowledge to improve my solution so that feedback loop gets created? And can I take this solution and reapply it to adjacent use cases so that I can multiply the value that was created? And, in the old, older traditional AI days, we did that but the trans- the transferability of the solutions was a little more brittle because, you had the trained algorithm. W- with a generative or agentic solution, we're orchestrating components, so there's a lot more fluidity. So this idea of I've deployed it, I'm standing behind it, I'm watching closely, I'm inviting feedback, I'm taking those learnings, I'm improving my algorithm, but I'm also looking for the nearest neighbors that I can reapply it to. If nothing else, the pattern can be reapplied. But in many cases, if I've invested in a deep context library on some domain, the ability to reapply that to a number of different solution opportunities is quite wide. But if that's not somebody's job, it's not gonna happen with the same speed or durability. It's... so I don't know, Ben you asked me a bunch of questions. I keep coming back to this idea of accountability for the context layer and its commitment to safe to own over time. These, these things end up being at the center of most of the questions you're asking

ben parker 42:05

Yeah. No, I think it's also once something's... once you build something, people wanna move on to the next project, don't they? It's just, just natural. People don't

guest 42:16

wanna do something new. I think that's one of the legacies of traditional software, and I'm not, I'm not-

ben parker 42:22

Yeah

guest 42:22

I'm not trying to beat up traditional software but we had matured the pipeline of traditional software to a point where you had dedicated builders and a different group of dedicated maintainers. And because drift was not the challenge in traditional software, it is in AI, that's the pattern. We all understand that pattern. We've all lived through that many times. But AI inverts that. Building is not challenging. Sustaining is very challenging, and we have to adjust to that. We ha- we have to factor that into how we think about who's involved in the build cycle, how do I create dedicated skill for maintenance? And I don't mean maintenance like just mechanical maintenance. There's certainly a piece of that, but it's this idea of trust and confidence maintenance. That's the really key job here

ben parker 43:19

So then I guess once AI is deployed, how should organizations continue testing and monitoring it so trust is maintained rather than just assumed?

guest 43:32

So this is gonna echo back to what I just said, but you have to treat an AI solution as a living system. It's going to drift every day. It's going to experience some level of microscopic damage to its logic, and if you aren't monitoring it and you don't have a regular repair mechanism, eventually it will drift too far and it will start to give you bad answers. And I've used the word drift a number of times, but let me just spend a minute defining it. There's a couple different sources of drift, and so one of those things is very much external. So if I'm building a, a pricing model for an insurance product, climate change, geopolitics, supply chain disruptions, these are external things. But what they mean is tomorrow looks different than the last 10 years because one of these things have gone in a different direction, and I have to know about it, and I have to make adjustments for it, and I have to layer that knowledge into my system. That's external drift. And then there's internal drift. So maybe we're going to market differently, or we've changed our appetite for risk, or we're applying different incentives to our workforce, which is gonna cause a shift in their decision-making process. Again, these all represent a difference in the future from the past, and my algorithms, whether it's a traditional AI or the foundation model, they're built on examples, past examples. And so that separation between the training data and current and future reality, that's drift. And so you have to watch for that, and there's several ways to do that. You can certainly, and you should, watch your data sources and pay attention to how the shape of your data is changing over time. My mix of product or my mix of geography is changing. The cost of repair of an automobile is changing. All these things. You have to watch that. That's really a, a data monitoring. But those data that are inputs to my solutions have to be watched. And then the other piece of this is- How frequently am I running my repair? Because if I can do microscopic repair missions, that's better. More frequent repair is better because the performance of my system is the average of its best day, the day after the model was deployed, and its worst day, the, the day that it has had the longest to drift. And quality of what I'm getting is somewhere in between. And so if I can shorten that window between when I notice it's drifting and when I adjust my algorithm, my performance of my overall solution is gonna be much improved. And so I think watching it and then engineering that detection and repair. And by the way to do that requires a commitment of capacity that's often at the expense of building the next new cool thing. And so that's where I come back to, like there is this idea of protected capacity, which is specifically protected to monitor and fix and improve my existing solutions with the same fervor as I put capacity against building the next new thing

ben parker 47:19

You know, and I'm guessing this all tie back to what you mentioned earlier about having, yeah, ownership on each part. 'Cause if it's just too much, then obviously you are gonna, yeah, face challenges as a business

guest 47:33

I'm completely convinced of that, Ben

ben parker 47:36

Okay, so wh- so when you look at, say, organizations successfully creating value from AI, what are they doing differently to build trust and adoption?

guest 47:44

I, th-three big things, and I-I'm, I'm afraid I'm gonna end up repeating some things I've said, so I'll try and be more succinct about it. But I think treating this as an operating model change and not a technology deployment at the start. Work and the accountability that comes with the choices we're making is getting redesigned. We're not doing component replacement. We're not picking the biggest friction generator or the biggest error generator in the process and replacing that. We're actually thinking about how work gets done with AI enablement end to end. So that's an operating model change, and that comes with rewrite of job descriptions. It comes with the creation of new job descriptions. I mentioned this idea of a dedicated content curator. Not a lot of places have that as a dedicated role yet. So the operating model change is gonna make us think about how work actually gets done and which skills or which sets of duties as a defined role need to be reinforced or introduced. That's one. Directly related to that, as I create those roles, my, my asset owner, one of my favorites, but this content curator, and then this idea of the subject matter expert as the validator, how are we staffing that? Because if we're staffing that as a part-time job probably not gonna work. I think about SMEs as validators as a really strong example of this. The non-deterministic nature means that validation often requires an expert to look at some sample and give an agreement rate. That's how we validate that the algorithm or the solution is performing well. That's an expensive thing to do in a couple ways but not the least of which is that there's a high opportunity cost for taking some expert out of their day job and having them play validator of an AI. And w- I think we have to rethink how we do that validation. That's-- So when I talk about staffing these, these duties, ownership and curation and validation as real jobs and not something that I do as a part-time by the way, that part-time assignment always shows up at the worst possible moment for that individual because they're probably really busy on something else. So how dedicated are they? How focused are they? Those things are gonna matter a lot. In my practice, the asset owner was, so valuable because the business thought of the asset owner as their agent inside the AI building shop. But the IT folks thought of that person as their inside agent on the customer side. And so that two-way, that two-way role is a trust and adoption mechanism by itself because it's building confidence in both the builders and the users because they each feel like they've got their advocate, the person who gets w- what their day is like in the middle of the loop. And so I feel like that one's really key. And then I mentioned this before, but I think the third answer is really Th- this idea of a blameless feedback loop. And so what I mean by that is we're hungry for feedback, and we're gonna treat the inevitable mistakes that are made not as, how do I hunt down who was responsible? But instead, th- these are how we're gaining our operational learning to reinvest back into the system. And I'm using the word system very deliberately because the foundation model is part of the system, but I don't have the ability to change that. The context layer is part of the system, I can change that. But also, how is the information being presented operationally? What policies have I put around user behavior? All of those things are available as part of the system. And when we make a mistake, how do we use that to upgrade those things? And we want to establish an appetite for that feedback. We're hungry for those observations. And if we react to those observations with, "Thank you very much. I'm now gonna use that to return that insight to the system," that is a flywheel for confidence building as well. And as long as we're behaving that way, then we're gonna encourage people to be very forthcoming with what they see, which is what we want. So I think operating model, staffing those duties as real jobs, and this idea of a blameless feedback loop, those are the three power moves in my opinion.

ben parker 53:00

Yeah, no, there's some good feedback there. So I guess for a practical takeaway I know you should provide some value, but if, say if a, a leadership team wanted to improve both trust and ROI from AI, I guess in the next sort of 90 days, three months what is the first meaningful action you'd encourage them to take?

guest 53:20

I think there's two things, and the first one is gonna come right back to where we started, which is trust. And so I would go to a business unit that has a deployed solution, an AI solution, and ask, "Do you believe in it?" And be very prepared for an honest answer, which is not really. I think that there's an illusion that says, "We built this thing, it took six weeks, we deployed it to a team of 200 people, and we've got a 20% increase in token usage among that population and believe I've achieved something." But if you go and interview the users and say, "Do you believe in this answer?" I think we should be prepared for many of them to say, "Not really." And so I would want a very honest assessment of my deployed solutions from that perspective. That's the first thing I would do And then the second thing I would do is maybe pick one where the answer was yes a lot, where people generally do believe in it. And then it would start to create that middle measurement layer between the token usage and am I seeing a metric that's really moving in the right direction? So if, I was an insurance guy, so am I seeing claim duration come down? Because I'm getting through my research tasks, I'm making a decision faster I'm resolving this thing sooner. So a- am I starting to see that manifest in a metric that really implies a change in the business outcome? So is it happening faster? Am I having a lower re- representment rate? I'm not reworking as many things as I used to, which would be more of a qualitative assessment. But the, but those are things you can measure. There's gonna be a lag between deployment, adoption, and those metrics changing. So that's why I say step one, go figure out which things you've deployed that people are confident in, and then pick one of those and see if you can't build that bridge between adoption and a business process outcome that I care about. And stay with that until, until I feel like I've got a story that we can express well and that we can defend and then build trust in my other solutions and repeat that process and start to generate a report card that says, back to the question you asked me earlier, like what's the chain? Trust, adopt, ROI. So step one, figure out what people trust. Step two, pick one they trust and start to build that adopt to ROI pathway, and then you'll have a great diagnostic tool that says, "I've got a trusted one I can't measure. Let me go work on measurement. I've got an untrusted one, not worth my time to measure it yet. I gotta go solve my trust problem." But if you inventoried your solutions from that trust perspective first, I think it'd give a pretty straight line roadmap for what you need to do next

ben parker 56:45

But I see, sure, I find it fascinating. I've seen, I've interviewed a lot of people on the podcast and I just, a lot of it is just down to people problems. It's like it's just seems like that's the hardest challenge for businesses.

guest 56:59

Look, I think human psychology is always- Yeah important for us to acknowledge and try our best to get an honest assessment of w- what, where we're starting from. And then trust comes from transparency, so bringing the right people in and having them truly participate. And in order to get them to truly to participate, I have to be willing to accept answers I don't like If I'm not willing to accept answers I don't like, people are probably not gonna tell me what I need to hear. So it really we get so fascinated, Ben, by the technology breakthroughs and so on, and I go, those things are great, but what's hard about this is still back to what I said. At the end of the day, I'm asking someone to rewrite their job description, to use a tool they don't understand, to be accountable for the outcome anyway, and in the worst case, I may not even be giving them enough grace to learn the new tool and integrate it into their process without adjusting their report card. And those are really hard things to ask another person to do. And I have almost never failed to build a solution because the data was too hard. Technology has never been a gap. Convincing a human to do their job differently has always been the hardest part, and I think that will stay true for a very long time.

ben parker 58:28

Yeah. Okay, brilliant. No I've enjoyed the conversation. And I guess, so if anyone wants to continue the conversation with you, is it best to connect with you on LinkedIn?

guest 58:39

Yes, definitely. Okay, perfect. And I would I would be delighted to follow up with anybody who's listening and wants to go deeper on any of this.

ben parker 58:46

Amazing. Yeah, like I said, I'll... I can I'll put, I'll add your LinkedIn to the podcast. And also if anyone wants to connect with Paul, just, yeah, please do reach out to me. And yeah, finally, Paul, it's been a pleasure having you on the podcast, and you've provided some great advice and knowledge.

guest 59:00

I'm so glad you invited me to do this, Ben. I'm really grateful for the time and the conversation.