How To Define Success ROI

In this episode of Data Analytics Chat, we welcome Michael Shaw, Senior Vice President of Data and AI at Dow Jones. Michael shares his fascinating career journey, which includes experiences at Google, Facebook, and Instacart. 

He discusses the key challenges he faced, such as managing hypergrowth during the pandemic, transitioning into leadership roles, and driving ROI in data projects. 

The discussion examines how to measure success in data initiatives, manage stakeholder expectations, and strike a balance between achieving quick wins and long-term strategic benefits. 

Michael provides insights on retaining top talent, fostering cultural shifts, and leveraging AI for business growth. 

00:00 Introduction and Early Career Lessons
01:08 Welcome to Data Analytics Chat
01:58 Michael Shaw's Career Journey
08:28 Transition to Leadership
17:49 Challenges and Overcoming Them
28:08 Defining Success and ROI in Data Projects
31:26 Leveraging Data for Business Objectives
32:04 Building a Core Data Foundation
33:05 Managing Stakeholder Expectations
34:14 Balancing Quick Wins and Long-Term Benefits
35:46 Tactical Approaches to Data Management
38:15 Overcoming Data Quality and Governance Issues
44:31 Driving Adoption and Cultural Change
49:48 Capturing and Communicating Softer Benefits
54:31 Evolving Expectations of ROI with AI
01:02:11 Setting Realistic Goals for Businesses
01:04:58 Conclusion and Final Thoughts

Thank you for listening!

Transcript

micheal shaw: 0:00

I found my way to a startup that completely failed in the one year that I was there, which taught me a really lot about red flags and startups. Found my way to Instacart, where I shocking grocery delivery, good business to be in, in a global pandemic. That business line came in with a really small team and a mandate for hypergrowth. Grew a team that was maybe six people when I joined. At some point I was managing over 30 people there. And this was a company that the business had just five XD in probably the three months before I joined. Then we were trying to build a sustainable business out of it. If you talk to people in consulting or accounting roles in the years before Excel, there's a really lot they did that they don't have to do today. And it's not that we have fewer consultants in the world today, it's that these. Explosion of things you can do with this new technology means there's so much more opportunity

ben parker: 1:09

Welcome to Data Analytics Chat, the podcast where we discuss the world of data, ai, and the career shaping it. Today, my guest is Michael Shaw a senior vice president of For Data and AI at Dow Jones. In this episode, we'll explore his journey, the challenges that shaped him, and the insights he has learned during his career. Today, the data topic for today will be on how to define success, ROI. Michael, welcome to the podcast.

micheal shaw: 1:39

Thank you. It's great to be here.

ben parker: 1:42

Yeah. Loving the topic today. I think it's crucial that investments in AI drive measurable business value, and it's a lot of chasing the shiny toys. So I'm gonna look forward to Yeah. Diving into that topic. But I guess before we do that, I guess obviously you've had a fascinating career working with some great companies like Google and Facebook. For the listeners, do you want share your career journey?

micheal shaw: 2:09

Yeah, happy to tell a bit of my story and I hope we get to something useful on success in data and how to measure that is a hard topic. I don't know how I let you, you talk me into that one for today, but before we get there, so who am I? I grew up the like math and physics geek. I was a. Math major in college because that's what you do when you're smart and like math and enjoy these things. And then switched into physics when I realized that math was this theoretical pursuit that I was just not gonna go into. And I was not, I was never gonna be a math professor. So I'm studying physics. This is like early 2010s in the Bay Area. I'm at Stanford at the time and I realized like many people, I don't really want to go into academia. I wanna try something in the real world that's actually applicable and useful to people. And what do you do in, early 2010s Bay area? Get a job in data science. I, I found my way to a startup that completely failed in the one year that I was there, which taught me a really lot about red flags and startups. Then I found my way to Facebook where I got a job on their growth team. I was responsible for helping people find their friends on Facebook, and it was great. It was playing with recommendations, algorithms, product, how to actually integrate things in a product drive success. Worked on that for a while. Then switched to a team of thought about Facebook's family of apps, and this is Facebook, Instagram, WhatsApp, and Messenger. It was right after they had acquired WhatsApp and they wanted to understand. Who their users were across the, this suite of products that they have. So I got to play with both the, like accounting, what, how do we pull a number together that we can put on the 10 Q and K for a large public company and go through all of the fun with internal and external audit on that. And then that once we have these numbers, how do we actually think about the strategy? What should we do with users and how should we make the product better for people across all of these different apps? Turned out that was a big job. And what was a one person job? When I started, it became a five person job very quickly. So they asked me to grow a team, hired a bunch of people under me, and that was my like transition into management and leadership. Played around with that for a while. Worked on a couple other projects in Facebook, thinking about sort of performance and reliability of the apps. Worked on a new website redesign, which was really fun. And then. I was there for five years and started thinking about what was next. I had realized that if I stayed, I was gonna become a lifer and I was gonna just be a real deep expert in this company and wanted to see what else was out there. So I started a job search in about February of 2020. And that was a fun time to be looking for. The first month was very normal, and then it became a really interesting time to look for a job. So I started looking around for what companies were an interesting place to be during COVID. Found my way to Instacart, where I shocking grocery delivery, good business to be in, in a global pandemic. I joined them in that summer and I was running their consumer data science team. So let's think everything. User facing. I worked on growth and marketing and the apps and everything from like, how do it make the search engine work so people could find their groceries to what's our top of funnel and how do we think about that? That business line came in with a really small team and a mandate for hypergrowth. Grew a team that was maybe six people when I joined. At some point I was managing over 30 people there. And this was a company that the business had just five XD in probably the three months before I joined. Then we were trying to build a sustainable business out of it. I was there for a couple years. Was there longer than pretty much the entire senior leadership team. I saw the founder replace all his directs and then he got replaced himself with a new external CEO and built that, stabilized. It had done a fun thing was looking for my next thing where I took a job managing one of the data teams in Google search. It's running data for Discover and then for some of their vertical efforts around sports and other things and how they play in search was there for about two and a half years and got really to see the like core of this trillion dollar business through the beginning of AI transformation of search. That was just absolutely fascinating experience to, to live through. And probably what I'll be thinking about a lot more when I see when this is being written about in the history books and I was looking for my next challenge. I'd been in big tech for many years and wanted to try something new. I did a bit of a broader search. I had mo recently moved from the Bay Area to New York and was looking for what is interesting in New York and what is New York. New York is media and finance and poked around it like are there inter, what are the interesting places there? Where can I bring fun data to bay and found this opportunity at Dow Jones where this company is, it's been around for over a hundred years and they have IP from the 19th century, which is just mind boggling to me. Coming from big tech. There's a new CTO who joined a year or two ago who is working on this like big digital transformation of what we're doing in the business, and it just seemed like the right place to be. I that, so I, I took this role. I'm running a bunch of the data teams, not just the sort of data science and analytics that I've played with in the past, but the full stack of data engineering and also the data from our sort of content and data sources that we actually use in the products as well. And playing around with what we do on search, on personalization, on ai. It's a bunch of really fun engineering centric stuff that I had never gotten the opportunity to do before. And they're letting me play and hopefully build some really cool stuff for the business.

ben parker: 8:03

Amazing. And some, yeah, obviously fascinating companies and yeah, companies with large data sets as well, so a lot to play with.

micheal shaw: 8:12

It's true. We talked back at, when I was at Facebook at how something is really small and rare and doesn't matter much, and then you hit, it hits you that hits a million people every day. And when you start rethinking of million as a small number, it's a, it's an interesting mindset in

ben parker: 8:27

Definitely. So obviously, was it great that you got push or not you got the opportunity to move into leadership. Was that some fear? Was that, was that from your, was that a dream of yours or was it, did it just naturally occur?

micheal shaw: 8:44

the moment, naturally occurred. I'm a really strong believer of something we do in data and engineering roles in tech of having these parallel paths for IC and manager development. I think the ideal is that all the way up through VP plus levels, they are parallel and you really can have people delivering that level of impact in an IC capacity. I don't know, maybe that could have been me. I fell into management and really ended up loving it. I like people, I like talking to people. I'm here'cause I enjoy talking to you, Ben. And it's management ended up being fun. I like getting to play with, to help grow people's careers to see their success, whether it's through, through them as much, much as through me. And yeah, in an alternate life, I could have could've stayed on the ICR. I still think about it occasionally. Maybe my next job is gonna be as an IC somewhere to go play deeply in that at some point. But I think for me the opportunity fell into my lap to, to become a manager. Something I realized I'm good at and I think it's something that I enjoy and probably wanna spend mo most of my career doing.

ben parker: 9:51

Yeah, amazing. And obviously there'd be a lot of people want to move into leadership or have the fear factor. So how, how did you, obviously you've managed some, some great teams. How did your learning occur? Occur around like leadership?

micheal shaw: 10:06

I heard this great advice years ago that always stuck with me, which was do one new thing at a time. So when I started as a manager, I was working in the area that I was already an ic and for the first like year or two of my team, I was probably still the best IIC on my team. So in some sense it was easy. I was coaching people and doing the thing I knew how to do. I could help them, I could step, I could shadow and reverse shadow them, play all of those different roles. At some point, that's a ceiling in management. You can never become a truly amazing leader if you're also trying to be to maintain yourself as the best see on a team. But I find that a. Just really great way to step into management a little. I know people want I get this question all the time from people about, how do you become a manager and there's no one path, but maybe being in the right place at the right time is a lot of it. And that's not just luck. The, you can plan that step into growing companies. When a company is fast growing. You can very quickly become one of the few deep experts in that company and they'll want you to share that expertise and to scale yourself. And they bring in lots of other people, and they are, it is so necessary to find people who can manage and grow. That's a place where you can do it. If you step into a place that already has enough managers, it's a tough sell for them to want you to be a manager when you've never done it before. Tho those places you go into once you are a HAA. Skilled and experienced manager, you've shown that you know how to do management and now you're bringing it to a new place. So the one new thing you're doing is managing in that place rather than first trying to be a manager for the first time.

ben parker: 11:50

Brilliant. And then I guess on the flip side, how do you grow your, so you've worked up a fantastic career. How do you grow team members then? How do you go about doing that?

micheal shaw: 12:00

I love to give people the opportunities, and what I mean by that is I will step back and let them do my job if they can. I, my what I tell my most senior people is usually some version of look, if you wanna do my job, if you wanna be in every room that I'm in and speak with all the leaders and write these, and I the these analyses yourself. Go do it and I will find something else to do with my time. I told someone, I probably should never have said this when I was interviewing for a job, but I told one of the people I worked for, that my skills as a leader are my humility and laziness. And I, that's I'm quite willing to let someone on my team out outshine me. And I, I think that's a really important skill for a manager.'cause your best people will outshine you. They will be better than you at some at things and not just deep technical things. Some of them will be better than you at public speaking. They will be better than you at getting alignment among stakeholders. Like these things that feel core to your job and your sense of self. Let them give them that space that is how you can become a great manager. You have to be humble. You have to be able to get outta their way, and you have to be willing to let them take over chunks of your job when they can. Now you also need on the other side to know when you can do that. And my mistakes as a manager have often been going too far in that direction where I've stretched people and given them opportunities well beyond what they were really ready for. You have to, there's a balance to all of it, but I would much rather lean on the side, make, I would, if I'm gonna make the mistake, I wanna make the mistake where I bet too much on my people. Then the other mistake in the other direction.

ben parker: 13:43

Yeah, amazing. Like obviously you need to Yeah, have the delegation, isn't it? And trust in your team.'cause if not, why is, why have you hired them or why are they working with you? And you can't as an individual, you can't be, especially data, like you can't be an expert in everything. It's so diverse now. I think you do need to just, yeah, I think your team will respect you more in that regard as well, won't they? If you've just given the responsibility, it's definitely gonna work in your favor.

micheal shaw: 14:10

One of the biggest challenges of management is retaining your best people. This is an amazing career and people who come into data and people who are good at this, they can go anywhere they want, and that's true within the company, and that's true at other companies. If you want those people to stay on your team, you'd better give them better growth opportunities than they're gonna find in the rest of the world. And that means giving them those opportunities, betting on them, taking those chances. If not they'll find, if they're good, they'll find those opportunities. They just won't be working with you.

ben parker: 14:40

Yep. I do know that there's a lot of opportunity especially in data at the moment. So if you had to pinpoint one or two defining moments that change your career, what would they be?

micheal shaw: 14:50

I feel like we almost just talked through one of the big ones of the, where I stepped into management. Maybe I'll, so maybe I can go in a, to a bit, bit more depth about what that was. I, so day to day I'm sitting there, I'm just overwhelmed. My, I'm working 12 hour days, I can't get done. Half the things I need to do and. I had talked to my manager about it, I had talked to some of the stakeholders and you do all the standard things. You prioritize, you ruthlessly prioritize, you start dropping really important things. You find other people who can take on some of that. And in parallel, we had a conversation about my own career and does it make sense for me to be a manager? Does it make, are there other ways to grow? Did my manager or someone else have bandwidth to hire people in to do some of this work? And what we ended up doing is that they hired in one person before I switched into management, who I was mentoring and helping coaching, and was able to hand some of my work to them. And then after that, I really did just have to step in and go through all the paperwork and be what it takes to really be a manager. And the worst part is, for a while, it didn't help. Yes, we knew we were gonna hire, we were going to, we had the open roles, we had done that, we had done everything we needed. But it takes a while to hire, as I'm sure you well know, especially if you wanna hire good people. And then even once you get someone good, it takes a while to ramp them up. And for, if you think about your, they're like net effectiveness, you start pretty deep in the negative if you, of all the time and effort that you go into to bringing someone great onto the team. And that just means more and more trade offs on the short term deliverables. But it was the right thing for the org. And we were able to get out of that. We brought in a, we were able to go to a larger team, bring in some really good talent, and at that point I could start bringing in more specialists. So I had someone who was like my accounting expert on the team. And if you're not into this sounds like a miserable role, but for some people I found this person who absolutely loved the, like thinking like an accountant as a data first and a deep digging really deep the how do you get these numbers in the right way? How do you. Speak the language of your auditors. How do you it's it felt like someone unraveling the mysteries of the universe, but it was the mysteries of accounting systems in that sense, and they loved it and I loved working with them on it. So you build these things. They're able to grow and scale and start. The team at that point starts doing things I could never have done myself. And it was, it becomes an easier and self-reinforcing machine.

ben parker: 17:28

Yeah, definitely. When you've got the right foundations in a team, when it clicks, doesn't it? It's just. It's amazing to see what can happen.'cause then everyone's got the job and then it's, you're not you're not wearing mul. Everyone was wearing multiple hats. But then obviously when it's more distributed, it's a lot easier for everyone to succeed.

micheal shaw: 17:47

Most definitely.

ben parker: 17:48

So what were the sort of, what's been a tough challenge for you? What's been one of your biggest challenges in your career and how did have you overcome it?

micheal shaw: 17:57

Yeah, one of them was when I stepped into this role at Instacart. So I come in like early summer of 2020 I, the month or two that I, before I joined. So as I was interviewing with'em was all of the, literally, can we keep the servers online tonight? Our business is growing multiples over weeks, which is like not a thing that happens to billion dollar businesses at scale. And maybe it does now. And like you, anyone who's worked at Open Eye is just gonna laugh at these stories. But like for in 2020, this was new for a business to grow in this way. And of course it makes sense. People were just in love with anyone who could bring groceries safely into their homes. And that's exactly what this company was able to do. Now we get there. We're now at this. Business at a scale that we had no idea how to really understand and match. I had a six person team thinking about the entire consumer facing product, and we had to just really quickly step in and scale. I needed to hire ics, I needed to hire managers. I needed to build a culture and a team and get all of this into a state while we are maybe not keeping the servers up all overnight, but figure, but like trying to deliver an early marketing stack and trying to deliver on some of these core mechanisms of like, how does this product work at this? Does it, can we grow it? Can we continue to make this a sustainable business? And there's just a ton of trade-offs of balancing things. It was probably one of the hardest times I ever worked of just try, just trying to pull all this together, doing a few jobs myself before I was able to actually hire people into those roles and get over that hump from can we survive tomorrow to how are we actually growing this? Do we understand what is truly incremental about this business? And do we understand how that will change as these sort of global conditions are changing? And we're going from a place where people are almost desperate to be buying our product to a place where there are now many competitors out there. And it's, the business is really trying to figure out what it, what is, what a sustainable version of itself is a business that was ready to go public and wanted to get itself into a position where it could make, send the right signals to the markets. And that means a lot of data maturity that we just didn't have. So getting in, digging in understanding what they did and. For me, this was the first time I ever switched companies into a leadership role, and that's a whole different set of challenges when you are not the best IC on the team when you are trying to understand how a business ticks and what are the core parts and what are the risky parts, and how you have to think about it and do all of this while you are doing your day job and while you were managing people and while you were scaling and delivering on everything, it was a fun challenge. I, I learned a really lot in those first, first few months there before I got my feet under me a little bit. Yes.

ben parker: 21:12

So it's a good challenge to have a good problem. So how did you overcome it then? Was it just a lot more thinking sessions or how did you get around it?

micheal shaw: 21:23

I don't know how much deep thinking there was. It is a lot of running quickly and trying to make sure that I was running in a reasonably correct direction. Prioritization was a big piece.

ben parker: 21:34

Yeah, and I guess I've got this, sorry.

micheal shaw: 21:37

No I feel free to, to follow. I, it's, to me it was, do I prioritize the things that drive the medium and long-term outcomes that I need? And how can I do that when everyone is throwing things at me for what they need today. Time blocking alignment, figuring out what are, what is the work around hiring that's gonna be truly incremental to be able to scale myself and my team and what are the things that like, even though it doesn't matter long term, I need to get this delivered'cause this is what matters today. There's, you gotta do some of both and you gotta figure out the right balance. But I think getting alignment on working on the most important things was probably the biggest piece.

ben parker: 22:15

Yeah, no, I like that. And also, it's similar today, look scale companies, scaling AI is a big challenge for companies right now. And it's, again, it comes down to mentioned, prioritizing the right requirement. So many people are, take Gen AI as certain examples. A lot of companies are going down that route where maybe they should be price hires in somewhere else. So yeah, I do think that's a, yeah, a massive positive that what you've done there.

micheal shaw: 22:40

And you gotta avoid the noise. I as maybe this won't even be true when by the time we release it,'cause AI today is this funny world of it's in the middle of a boom and maybe in the middle of a bubble. I think it's really easy to deliver these things in a way that don't actually add business value. That when a company is growing so quickly, everything looks good, all the numbers are going up and to the right. You need to figure out what are the things that are actually incremental and what are the things that are gonna deliver real ROI. When you start to look into that and really deeply question it.

ben parker: 23:14

Yeah, no, I agree. And I guess, so obviously you been successful, but progressing into leadership, what sort of skills or mindsets have helped you become the person you are today?

micheal shaw: 23:25

I think what I haven't mentioned yet is intuition around data. One of the key things that I've been able to grow, and I credit a lot of the leaders I worked for early on for this, because I, they were just brilliant at it, is being able to look at numbers and know when you should call BS on them. And you do this really just by a lot of deliberate practice. You've pulled a bunch of these numbers yourself. You know what numbers look reasonable and what ones look a little bit iffy. You're not you have to get out of the mindset of like local optimization. Optimization. You're not looking at does this make sense based on the last step you're looking at. Zoom out. Is this a reasonable number? So for one, one almost hilarious example I won't say which company I was at for this, but I was looking at an analysis of usage, and someone talked about the opportunity size. You have this five year out projection of how many users will have on this product. And they told me that they would have about 400 million active users in the United States. And if you think about that number for half a second at in the right context, you realize that is more than the number of people in the United States. And if this product is maybe restricted to 13 plus or any number of re of other reasonable things. You're just not, it's not possible. And what's going on, of course, is that there's some issue in their modeling or data. Maybe there's a factor of two somewhere, or some very unrealistic growth projections in there, but it's not obvious when you look at the projections and when you look at the models, you have to really dig into see the miss. But when you look at 400 million in the United States, like of course that's wrong. So getting that intuition, figuring out when to question numbers, how to question, how to think about it, and just practicing that stuff a lot is I think one of the real skills that you need to be able to coach, guide lead things in the right direction and know what's actually worth sharing up.

ben parker: 25:32

Yeah, no, I do think. With the way the field's going as well. And it is a lot more, obviously, tools out there that can do the heavy lifting in terms of the technical aspects. I think if you've got the especially like the business domain expertise, that the knowledge around what's actually the data means you are gonna stand out and understand the business problem so much more. And like I said, be able to identify where there is correlation and things like that.

micheal shaw: 25:58

You have to understand the business and you really knowing which pieces of the world to bring in to think about, whether it's understanding the broader context, the company's in, the competitors, the world knowledge, your team, the technical systems you're in. They all give you different ways of thinking about a problem. And great leaders can do all of them and know which ones apply in any moment.

ben parker: 26:24

Brilliant. Then, what advice would you give to someone who's aspiring to become a top leader?

micheal shaw: 26:31

It's so hard to do this generically. Everyone's story is different. Everyone's situation is different, and I really think people should lean into their own strengths and take advantage of opportunities in front of them. Maybe one thing that's always true, if I could say it is don't be selfish. Care about and grow the people around you. This is critical If you want awesome people to work for you,'cause they're gonna know and no one wants to work for the manager who's gonna take credit for what they do. But it also is about your peers. If you're a, many of the ways people get into leadership are opportunistic. Maybe a leader departs and they offer it to someone on the team. The people who are your peers today might work for you, Nick, tomorrow, and they're not gonna stay and they're gonna question the decision if they don't think you're someone who will grow that them and care about their careers. It's a surprisingly small world in data. I've worked with a lot of people multiple times that I had no idea we would ever work together again. And the reputation you build early on sticks with you. So care about your reputation, help people around you. I think that is always gonna help if you wanna be in a leadership role.

ben parker: 27:53

Yeah, I think it is obviously everyone's on a different journey. Everyone's got a different path. I think it's also being a self-aware of what, where your skills lie and not follow, trying to follow someone else's career, like just stick to your journey. It's, I think, so important. So let's move on to the data topic. And I know yeah, an interesting one, a challenging one how to define success ROI and it's obviously yeah. Gonna be interesting to see what you say. So this with many companies struggle to agree on what Riss in data projects. From your perspective, Michael, how should organizations define success in a way that aligns both business and data teams?

micheal shaw: 28:36

This is such a good question, and if it's not obvious, it's Why is this hard? This is hard because when you're building engineering products, you know exactly how to value success. You look at the data, you use data, you use the metrics to say that this worked because it helped our users achieve the outcomes that they were trying to achieve. It hit this business objective. All of that comes from data. So then the business stakeholders who love this go back to the data team and say, Hey, cool. Can you evaluate the success of your own data projects and get a bunch of blank stares? Because until we do the data work, we don't know, we don't see the numbers in advance, we don't understand the ROI in this. And then even when we do it, the data project isn't what's driving the metric. Engineering is doing that data is just measuring it. And you almost think that should be free, but of course it isn't. Data, product products, some of the hardest products to build. So how do you do it? There's some, a lot of this gets back to intuition of being able to, with much less data than you want, estimate and intuit the value of things. Understand which. Is gonna work at scale, what matters in various different ways, which is gonna, what, which parts of these projects do line up with real business outcomes and what things couldn't you do without this data product. And some of this you can get with generic intuition and just understand how to think about data. And some of it you really have to evaluate with a specific project in mind. So one thing we're playing with at Dow Jones right now is we are doing this massive data migration. We are moving all our data pro products from a just variety of places into a single data lake. We're we have this new data lake on Snowflake. It's gonna be fantastic when we finish it. And it, anyone who's worked on one of these before, this is a multi-year project of bringing. Data from all sorts of legacy systems into one place that we're really excited about. We're gonna build key and coherent metrics for our new business lines on here. We're gonna align it with experimentation, tooling. It's gonna be great. And I know I think about the value of this in how it will drive our strategic thinking, our business decisions over the next years and maybe, maybe even over the next decade plus, where we will be able to make better decisions as a business. We'll be able to invest strategically as a business because we have this core data foundation that, that we're able to use for it. And that might be the core of it, but I also, it's like, what else can this do? You're saying like, Hey, we're solving this one business problem. That's great. But one of the great things about data is that often these projects can be useful in a variety of different ways once you have built this data foundation. Are there other business objectives that you can meet? One bet we have is we're gonna do a lot more ab testing than we had done in the past because we're make, we're setting up a system where it's easy to do this. So it's not just an efficiency bet on the testing that we do. It's saying that we are going to improve the product development for a variety of engineering teams.'cause we're building this foundation that lets them do it in a better way. Dow Jones is also a data business. We are, we sell data products as part of our risk business, as part of our energy business. And building a core data foundation will allow us to do a better job with those product lines as well. And just, we don't, we're not talking about like analytics data that we're selling. We're not gonna tell someone what you're reading on the Wall Street Journal. What we sell is an understanding of what prices someone should buy energy at and what are the markets that they need to think about if they're in a. Risk sensitive business. Are there customers they need to be careful doing business with? Do we have other information on various areas of their world? And all of these are data products. We're building a data lake. Can we do a better job building and selling and cross selling and all of this? Of course we can. So you have to think what is the big picture and how can you think of a data product as more than just a way to do a little bit more efficient analysis, but as a way of really driving business outcomes as well

ben parker: 33:04

Yeah, I guess it's important for you as a leader to, to manage expectations for stakeholders'cause that's a big problem for data projects. Obviously some of them, like you said, they're not just quick projects. They're gonna take time. And the ROI is gonna be, yeah, down the line. So it's obviously, I guess a big challenge would you say is ma managing expectation.

micheal shaw: 33:30

with the right stakeholders and the right setup. I wouldn't say challenge but yes, it's something you have to get right. Something you have to care about and get that alignment in advance and not just try to back into it in the last minute.'cause some of these do take a long time and the real wins can be in the out years rather than right away. And that's something that you need high. You're always gonna have pushback in the moment. Stakeholders always want the urgent thing that they care about, but if you start it right and you have high level alignment on what are the most important things, is this project actually the right thing to build, then yes, the, you can you can move mountains.

ben parker: 34:11

Okay. Brilliant. That sort of moves on to the next question as well. So lot big challenge is balancing the demand for quick wins with the longer term strategic benefits of data projects. So how would, how do you go about managing that tension? Be a bit strong of a word.

micheal shaw: 34:26

You asked the tough questions about data leadership. I give you that. So the key thing that has most worked for me is changing where I make this decision. If you ask about what should I work on today, everyone wants the origin thing. If I ask about what should my team focus on for the next year or for the next two years, I start to get the zoom out answers from stakeholders about what they really want and what we can really deliver for their business. This isn't unique to data, by the way. If anyone is interested in urban planning and zoning, it's the exact same thing. Nobody wants the, factory built on their street, but everyone might want it built in their country. Nobody wants the loud, noisy thing near them, but that is not an indication of their feelings about it being built. And that's true of data. No one wants their project to be pushed off the priority list today, but they, you can get great alignment when you talk about the importance of building this foundational effort over the next year or two years, or even, and it doesn't have to be years. I'm giving the extreme, like if you talk about priorities for months and quarters, you still do better than if you're talking about priorities for the day. So strategically, yes. Think big picture, get alignment there, and then you need a bunch of tactics to how to actually execute on that. So one thing I did at Google, I had inherited this team that had a really lot of inbound and they just had this task list coming in. Everyone was asking them for analysis and. They would work through it and yeah, they would prioritize a little bit and do the ones for a more senior stakeholder before the others, but they so a core part of their job is working through this inbound list. So what I did was flip that a little bit and I said we're gonna time box. We're gonna say that maybe 20% of your time is on these inbound tasks. And I let everyone do it the way that worked for them and their life and their schedule. For some people it was, Monday is inbound day and you day, all of Monday, you just work through these things coming in. And then you close that and you do not look at that window Tuesday to Friday. For others it was a few mornings or a few afternoons. You could find different ways of doing it. But the key is you don't let the rest of the asks on the inbound side filter out past the time window that you're giving it. And that my job is then aligning with all the stakeholders Hey, this is how it's gonna work. This is how much of this we're going to be doing. And if you want more than this, you're gonna have to find some other way of getting it. There's a bunch of other tactics. I'm a big believer in game theory, in organizational politics, and if you think about making it just a little bit harder to ask the team for something, you tend to get fewer asks. And there's just like a lot of tactical ways to do this. I've gone with intake forms that just ask for a lot of detail and requirements and you really try to align it that the person who wants the thing does most of the work on the thing and the team that feel that maybe is the core expert in a system that has to be involved in delivering it but isn't as invested in the outcome, they have the easier job. So they should only get it once all the requirements are pulled together and all the details are there and you, this is a win in two ways. It's a win because it's easier for the team when all the details are there, but it's also a win because. The lower priority ones never actually get asked. Once it's expensive to ask a question, you find that a lot of the questions that maybe just weren't actually worth investing time and energy in don't make it to the core team. So that's a way of solving both ways you align a bunch of incentives and get the tactics a little bit more aligned with the sical decisions.

ben parker: 38:11

Brilliant. I like that. Let's move on to the next question. So with business obvious poor data quality silos lack of governance often, or ROI, how can businesses overcome these issues to unlock like the real value?

micheal shaw: 38:29

So I think of this in two ways. One is trying to just start with simple things. When I was at Facebook, I inherited a team that had, was thinking about some of these ideas around content. And one thing there was this metric a year before I got there about content production that probably shouldn't share the whole thing, but I'll tell you, it had a square root in it. And sure, that's not an especially complex piece of mathematics, but when you start bringing stuff like that into a metric, what you end up building, maybe it would be a good objective function for an algorithm. But as a metric means no one understands what's going on. It goes up 5% one week, it goes down 2% the next week. It makes no sense to anyone because it's this big, messy, complex algorithm. So start building really simple things. So I was working on a dashboard for exec executives there where we just counted likes, comments. Other really simple bits of engagement. We didn't try to add them, we didn't try to value them. We just had the raw counts. And that became a pretty darn valuable view for execs because they could see exactly what was going on and they knew and could understand the week to week changes. So this gets across some of these complex, messy issues because you're just getting done something really simple that everyone, no matter where they are, can intuit the, I think the other piece is maybe a bit more of a technical answer. So do Dow Jones is this we have this fa absolutely fascinating history. And I say we've been around for over a hundred years. I inherited a team. I have people on my team who have been working at Dow Jones longer than any company I've worked at before, has existed. And, I haven't worked at especially old companies, but Google is not a brand new company and neither is Facebook at this point. And it's just astounding to me to think about the like level of history that we're dealing with. It's a company, Dow Jones is a place that has acquired all sorts of businesses over its history, and that's one of our strategic bets is that we can be a, we can do a better job meeting what our customers need by bringing new business lines in some, through building them ourselves and some through strategic acquisitions. So what that means is when you acquire a bunch of companies, you acquire a bunch of technical stacks, you acquire a bunch of data systems, and it is amazing to me how many data, I mean you could call them data silos, but it's more than just silos. It's like we have multiple different systems to log data. We have different systems to store data. Not only do we have dashboards with inconsistent metrics, but they're built in separate dashboarding tools. So one of the bets that we're making is building single platforms, aligning it. This is the way that we're gonna build for the company. We already did this with a data lake on Snowflake. We're talking about it for analytics tools up and down the stack. And we're gonna get to a place where we have a standard tooling answer that we're gonna slowly bring everyone on no matter what their history of ac of acquisitions, we're gonna build standard metrics. And it's not that this instantly solves the data quality and silos, but it al it gives us the platform that we can solve it on.'cause once everything is consolidated, once we're on single platforms, once we're all talking the same language a little bit, we can then start to have those conversations about, Hey, is this the right way of measuring search success in this product and that product? Maybe I'll just take the same metric in this, and we've built on the consistent logging infrastructure. Look at it in both places and start talking about it, and you can get to a much cleaner understanding of what's going on once you've done some of that technical consolidation work.

ben parker: 42:16

So do companies, do you feel switch metrics too often or what your thoughts be there?

micheal shaw: 42:24

Oh, some companies definitely do. And I was on one team I probably shouldn't say where this was. I could say it's not Dow Jones, a place where I was there for multiple years and every year we had a new North Star metric. And if you think about what a North Star metric is, it really should be consistent for a very long period of time. Definitely more than one year. And they built a bunch of dashboarding in for all of this around new North Star metric each year. It was crazy. It was just a big waste of analytics time and it meant the entire products and engineering team was pivoting left and right without any real direction. But this isn't, I don't put that as a. Metrics problem. Exactly. That is a strategy miss. And the reason this happens is because these teams aren't actually aligned on what they're trying to do, and they are trying to use metrics to solve something when there is a real strategic alignment issue. One of the jobs that I think is critical for data leaders is that we are the people who sometimes step back and tell you when to use and when not to use data. And this is a place where if you are flipping metrics left and right, you might wanna step back and have a conversation that's a little bit less about the actual data and a little bit more about what are we actually trying to do? What does success look like for this product? And can we understand that in, natural language first, and then talk about how that translates into an actual metric.

ben parker: 43:47

Yeah. No, I feel, I do feel that's a big challenge for businesses.'cause you should have a good leader that knows where they're heading and then you hear about, obviously so many projects not going to plan. So it's, it must be the strategy or the execution, the communication must be the big problem there because I guess. When you move the goalposts and data, obviously it keeps moving, doesn't it? So I know it's obviously never ending, isn't it? As we can obviously see in the market as well. So yeah, I think strategy is like a big challenge for businesses today. So I think with projects, I guess even when the tech works, projects often fail because the wider business maybe doesn't adopt or act on insights. What would you say is key to driving adoption and cultural change?

micheal shaw: 44:35

Oh, that's a great question. I think you need to approach this top down and bottom up at the same time. Top down is maybe an easier, it's easier to discuss anyway. You need a real sponsor in somewhere at the place where they're actually making decisions at the company. A lot of, I find a lot of these things come from a data team or from a junior product person or from the team building it, which is great that they should, I'll get to a second. Like they, you absolutely need them involved, but if you really want to change strategy, you want to change major decisions in a big corporate, you need someone who is in the room when those decisions are being made. And that's a different flavor in a different, in different companies in a. In Google. That means you need someone who's in sort of the SVP Senior Leadership team. You need to be involved at the search leadership or knowledge and information leadership. In the worlds that I was in, or ADS leadership, YouTube leadership, those are the places where they're making the big strategic decisions for the company. At a place like Dow Jones, you need both tech. You need the product and engineering and sort of tech leadership, but you also need the business leadership in the room and aligned. And that means that you need a sponsor who's gonna be in the room with the senior leadership on the business as well as the technical side. And that person has to really, I call them a sponsor.'cause you really do need someone who will sponsor this project. You do not need someone who's bought, who's interested or likes it or will say yes to it. You need someone who will put their own career on the line and help convince others of it. And if you don't have that and you don't have a path towards doing that. You are probably not going to make cultural change in a large company. And I don't mean that to dissuade. Let's say go find your sponsor. Go convince someone and look around until you find the right person in leadership who's gonna be in the room and help push that. Now, this is really different in startups. If you're at small companies, you're places where everyone knows the CEO EO, go sponsor it yourself. That's great. But if you want to do this in a big company, you just need someone in the room because if you are not there, you will be it will silently fail in a way that you don't understand. Second thing you need is true bottoms up, alignment and interest. Are we aligned to how people think and work? And is this something that they actually want to do? So one of the things I came into Dow Jones with is I come from tech. I want to help bring some of the best practices of tech here. And I want, I. Wanna make sure that everyone on my team is able, has the ability, and op has the opportunity, and is able to actually use these, all these modern AI tools that are just a, it could be a huge step change in how we do our jobs, but I want it to come from them. I don't want this to be a top down, like I'm yelling at people about things. So I just surveyed for the team and this was like a good team health survey, and I asked all the usual questions. One of the questions I asked was, how often do you use AI tools in your job? And I had set myself a KR of, I think it was around 70% should use AI tools every week. I thought that was a bit sandbagging, but I really wanted something that we'd actually hit by the end of the year. Of course I run the first survey and nine, I think it was over 90% of my team says that they use AI tools every week. And that's fantastic. So I'm really glad that my team is ahead of me on this. I clearly screwed up in K writing. I'm gonna get probably gonna get that from my leadership. And we're gonna change the goal and say, it's gonna be about either daily usage or about the depth of usage are being, or how these tools are actually driving to business outcomes. And not just that people are using them, but the key is, I want it to be something that comes from the team. If I tell everyone your goal is to just use AI every day, maybe they will because I'm the boss, but they're not gonna get anything useful out of it. We're not gonna actually deliver business impact. They're just gonna get, go wake up in the morning and ask Gemini a question about their work and that, say if they used AI that day, what I want is to find ways that this is matters to them in their day to day. That they have, they're unblocked, they have access to all the tools they need, that we have the right partnerships when there's an external vendor tool that they want and that we've. Set things up and give them the time and space to try and fail and iterate and all the things that they need to do to hit it. But it's about getting them excited. It has to be some, if you want to do real cultural change in a place, the people whose culture is gonna change have to actually be bought in. This is something that is aligned with their own career success and something that they wanna do.

ben parker: 49:17

Yeah, I agree. They need to see the benefit, don't they? Because people don't like change. Like I see. Why? But if it's gonna benefit like their career or it's gonna make their job a lot easier. Cool.'cause people would spite your hand off.

micheal shaw: 49:32

It has to be the benefit to them, not just to the company, not just to their boss, but like they will do better. They will be happier. They will get their life back. They will grow their career because they are do working in this new way.

ben parker: 49:46

Brilliant. So I guess, look, not all value is financial. How do you capture and communicate the softer benefits, like improved decision making, speed, customer experience, or risk reduction? When calculating ROI.

micheal shaw: 50:02

Some of that is financial, or at least can be quantified and connected back to financials. Look, if you want, PE companies pay people a really lot of money to make decisions and a lot of the, top executives, what do they do is they make hard decisions. They spend time thinking about and aligning with people on it. And if you can speed up and make them more effective at their jobs, I think you can pretty clearly connect that to financials. And so one of the things that you should do is find all of the things that you can measure. Actually measure'em, quantify them, and include that in the sort of financial side of the ledger here. And I think this is something that good data people should know when is possible and know how to do, even if there's some estimates or guesses around it. But yeah, that's not gonna be everything. You're never going to get in the language of your financials or in a way that shows up on an accounting statement. Long tail risks, the customer perceptions that maybe drive financial results in 10 years, but not in a way that you see today. And I think one of the roles of data leaders is to know when you need more metrics and when you need fewer metrics. Folks who are less familiar with data might try to quantify some of these things that just can't really be quantified and end up missing the core value of. Of some of it. So one to make this a little more connected to reality. So I have worked on two search products in, in, in my career. When I was at Google, I worked on a, product. A lot of you may have heard of it on google.com and they have too many metrics. They have made mistakes in the past by trying to use data to measure things that could not actually be measured by data and have ended up with a search experience that people perceive as worse, even as their metrics start to say it's better. And that's a big risk. They've thought about it, they've found ways around it, and they've done all sorts of complex work over time of building in human evals and other ways of measuring success that are not purely with data because when they were just using too many numbers and looking at it in the wrong way, I. In my job here, one of the things I'm responsible for is the search products we have at Dow Jones. This is the search on Wall Street Journal and wsj.com. And in our apps, this is the search on Factiva and some of our B2B products. And one of the, bet bets we have is like building a new co foundational tech under this building, a single search tech that we can use across our business. I, I think that'll be amazing when we ship it and we're in the process of it. We've shipped semantic and now hybrid searches on both our B2C and B2B products. And when we shipped, the first one of these I was about two weeks into my job, so I can take no credit for the actual work that went into this. One of the first questions we got from our CTO was, how did it do is the CTR is better than it was before, and the team didn't know because we weren't logging. Or at least we weren't logging in a way we trusted. And so we have the opposite problem. Here is a case where we just need more metrics. We need to actually dig in and understand and see just a couple key ways of measuring the value of the products that we have. And we do have CTR say it is significantly better than it was before because we're shipping a better experience. And that shows in the numbers. But this is where you have to figure out the right balance. There are some things that should be in numbers and as a data person, your job is to, our jobs are to make, to bring it into numbers and to get the right numbers, to be able to make decisions. And there are some things that should not be in numbers. And I actually think the data people are best positioned for to say that because we are the ones with the, who would know if it could be done. And we're the ones who should build that, who should have that trust in the organization to be able to say this shouldn't actually be a data problem.

ben parker: 54:16

Amazing. It's, yeah, it's amazing to see how AI's transforming just products, services across businesses nowadays. It's so much going on and I guess, with that. I guess expectations rise, don't they? So I guess how do you see expectations of ROI evolving?

micheal shaw: 54:36

I think every leader I've ever worked with has been thrilled to do more with less and to increase expectations every year while they decrease resources on a team. And look, AI is just the next opportunity for doing that in a lot of people's minds. And it will be that there, there are all sorts of ways where things that were things are now possible with smaller teams than they were in the past. But the way I think of it is. Use AI to compliment your most important work, not to re to replace it. So I use AI tool. I just use the consumer AI tools every day in my job. I'm always asking Gemini and chat GPT and Claude and just about things that I'm not an expert in, but it would be useful for me to know more about for my job. And I find that they're just really fantastic at that in just accelerating and speeding up learning and being able to work in areas that maybe otherwise I would've had to hire someone or to talk to someone about, I can start by talking to an AI and learn a little bit more before I waste someone's time. I, I challenge my team to do the same. I think we can all be more full stack than we were before. We can make decisions around AI coded prototypes and demos rather than around a mock. And that will just make us. Better at making early, early decisions. We're able to cut off off bad ideas and accelerate good ones really quickly. And that's not saying that AI is gonna write better code than our best engineers, but I think AI is gonna definitely write better code than our best designers and data analysts. And this lets them prototype something as if they were an engineers. There's like a bunch of of amazing opportunity there. It also lets us scale things that we never could do in the places we're. So one of the things I have found tremendous value in is actual people looking at a really lot of results and just doing something that doesn't naturally scale. So when I was at Google, we did figure out Google did scale this and they have this amazing scale set of human evals. They ha they have a lot of people who are paid to look at. Results and interpret them and grade them and rank them and figure out what, how to, how well something is doing and how it is different after you've run certain changes. They also can run surveys on their products at scale and get a lot of this actual data in. Years ago when I was at Facebook, I one of the mo the most successful intern project that I ever had on my team is I told the intern of Just go spend a day, look at hundreds of results here and do some simple analysis o of what it looks like because we couldn't figure out a way to get clean metrics on it except for by someone just sitting down and looking at it. And it turns out that was just a really impactful pro project to get done. And I'm thrilled that we're able to do it. And I look back on it and it's funny'cause today. We do this all the time at Dow Jones, and it's not because we have an army of of interns. Our intern, we do have interns. They're fantastic. They have to work on much more exciting things because we have AI to do that. And it's using these LLM evals is something that we've been doing in search at Dow Jones. It's something that we're use doing on some of our new gene AI features that we're hoping to release on our B2B product. Soon. We're building I'm not sure we've announced these. I'll probably be a little generic reading some summaries of really conversational features, like some amazing ways to interact with our data and content. And we're evaluating them at first by running it through an LLM as a judge and using AI to tell us how well this is doing. We'll, of course, look at hu human testers after that, but this way we do one round before we have to waste people's time. It's faster, it's easier, and it's something that scales in a way that I never imagined would happen at a company like this. There's so much more that we can build with ai. We're we're building these summaries, conversational features. We're gonna build agentic experiences into our products. We're gonna use AI to help take our data and content and surface it in new ways, whether it's in our tools in cus in customer owned tools on the B2B side, and really get it deep into people's actual workflows in their jobs. I think there's just so much more, the analogy that always comes to me is spreadsheets. If you talk to people in consulting or accounting roles in the years before Excel, there's a really lot they did that they don't have to do today. And it's not that we have fewer consultants in the world today, it's that these. Explosion of things you can do with this new technology means there's so much more opportunity. And I really see a lot of data and data adjacent roles feeling that way with ai. It will take away some of the, it will do some of the rote work for us. That is fantastic. We all get to do more interesting things and there is so much more that's going to be possible with AI that I think the opportunities are look pretty bright. At least in my world,

ben parker: 59:57

Yeah, agree. And I think it's obviously like we mentioned earlier, isn't it a lot of the heavy lifting is being done now and whatever your job is, it's like zooming into what, how can you get deeper into that? How can you improve it? It's getting more strategic in that regard, isn't it? Just how can we take this further pushing, pushing the level.

micheal shaw: 1:00:18

we all get to be more strategic. I in some ways you go deeper. In some ways. AI actually lets you go broader. And one of the patterns I see in newer companies is they, everyone is a little bit more full stack than they were in the past. If you look at some of these startups that are just living in the AI world, it's a designer who can build a working prototype. Not because they can code, but because they and AI can code and have a meeting where they discuss whether they should actually build it with proper engineering practices, but they have it by looking at and playing with a real thing, rather than by talking about something in Figma or in other design centric worlds. And I think that's the direction I wanna see data scientists get out of this. Answer a question for your product person because. We've u set up AI on our data sets in the right ways that every product person, even the ones who don't know any SQL, can start asking their questions of AI and getting those simple things done. And when they come to a data scientist, they're asking a really interesting, deep question about the future of our product that requires them to think strategically, that requires them to connect the dots between things and make the right assumptions about the world, rather than just knowing which tables to join. So that to me is the opportunity with ai, it's, we all get to do the easy parts of each other's jobs in a way that a lot of these trade-offs and handoffs get easier and we get to focus deeper, but also focus broader in the world.

ben parker: 1:02:00

Yeah, and I guess. I've not, obviously, I've not pretty not gone easy on you today. I've asked some challenging questions and like it's great, some great advice we shared. What advice would you give businesses trying to set realistic goals today?

micheal shaw: 1:02:15

Talk to your data people. If you don't have great data, people hire them. And I say that a little glibly, but I do think getting to the right goals really depends on a deep understanding of the business, where they are, what they are trying to do. In Dow Jones, we I, we can I can think of our own goals and what we should do and how we should move them. And we're already moving our thinking from these like individual business lines that maybe had. Come in through an acquisition are still sitting there in some of our minds to thinking like, how are we understanding our products as a platform? How are we meeting all of the needs for our B2B customers? And are we able to deliver that through whichever product line they have happen to be? And how do you translate that strategic idea into specific metrics that the team can actually go on to make sure that our KRS are actually aligned with the, with these ideas For other companies, it's an, it's a different place. It's setting different layers of goals, different levels, and I think it's all about if there is generic advice, a think align on your strategy, get metrics that best measure your success towards that and iterate over time on this. Don't think that you're gonna get it right the first time. Don't build out some big complex system until you're ready for a big complex system and make sure that you are involving the right people in these decisions. What if I'm, when I'm asked by business leaders about what to do, one of the things I really do push on them is if you want to be a data informed business, if you want to think about data as a true first class citizen, you also have to think about your data teams as a first class citizen. Where does data report into? Is it layered down three layers under some product person? Is it hidden in the finance org and is only gonna think about finances perspective on the world? Or is it treated as a first class citizen, the same as the other key functions? And then, and you have a data person who's in the room with you and you don't have to go ask some external consultant how you should think about your goals.'cause you have the expert right there.

ben parker: 1:04:33

Yeah, I think it's, there's unlimited opportunity out there now. There's so much going on. It's obviously, like you said, you need to obviously focus on your business priorities. But yeah, I think it's an amazing time we're in. Obviously it's so much that can happen now and it's, yeah, I guess it's not necessary a race, but who can get it aligned quicker, embedded with their business as quick as possible. So brilliant. Michael, it's been a pleasure having you on the podcast. Yeah. Loved your passion, your insights, and I'm, yeah. I'm sure the listeners will really value your time.

micheal shaw: 1:05:08

Thanks so much for having me. It was great getting to talk to you and if anyone is looking for data roles, we are always hiring.