Leveraging AI

256 | How to build business strategy for the AI era, A detailed blueprint with Andrew Rabinovich, CTO and Head of AI + ML at Upwork

• Isar Meitis, Andrew Rabinovich • Season 1 • Episode 256

Share episode

📢 Want to thrive in 2026?
Join the next AI Business Transformation cohort kicking off January 20th, 2026.
🎯 Practical, not theoretical. Tailored for business professionals. - https://multiplai.ai/ai-course/

Learn more about Advance Course (Master the Art of End-to-End AI Automation): https://multiplai.ai/advance-course/

Will AI automate your workforce or amplify it?

That question isn't just theoretical anymore. For business leaders navigating the shifting tides of AI, understanding how to remain competitive without losing the human edge is now essential.

In this episode, we go deep with Andrew Rabinovich, CTO and Head of AI at Upwork, on how one of the largest talent marketplaces in the world is strategically evolving with AI, not just to survive, but to thrive.

Andrew shares how Upwork is moving beyond simple matchmaking to delivering outcomes, what it really takes to integrate human-centered AI, and how AI agents and human experts will coexist in the near future of work.

Spoiler: It's not about replacing freelancers—it’s about transforming what they (and businesses like yours) are capable of.

In this session, you'll discover:

  • Why business leaders must rethink efficiency in terms of outcomes, not processes
  • How Upwork built “Uma”, an AI agent guiding clients from vague ideas to precise deliverables
  • The surprising results of benchmarking AI agents on real Upwork jobs
  • Why task-level automation is only step one and why full automation is still far off
  • How to identify which parts of your business workflow AI should (and shouldn’t) touch
  • What human-centered AI actually looks like in practice—and how it drives both trust and scale
  • Why AI agents will soon be collaborators, not competitors

About Leveraging AI

If you’ve enjoyed or benefited from some of the insights of this episode, leave us a five-star review on your favorite podcast platform, and let us know what you learned, found helpful, or liked most about this show!

Isar M:

Hello and welcome to the Leveraging AI Podcast, the podcast that shares practical, ethical ways to leverage AI to improve efficiency, grow your business, and advance your career. This Isar Metis, your host, and I have a very unique episode for you today. As you probably know, the improvements in AI capabilities is putting a lot of question marks on the value that many companies are generating for their audience. For many companies, the risk is relatively low, like if you're running a, you know, a. Car body shop, then the chances this impacts you are relatively low. For some others, the risk could be existential. Like if you are running a company that generates content for others, mostly written content, then you are facing some very serious risk for your future. So there's, and there. The entire range, obviously in between. So every single company is now need to be considering, okay, how is AI gonna impact what I offer and the value that I provide? Now, dealing with such high level of uncertainty is not easy and being able to. Learn from companies who have done steps in that direction and how they approached it and what steps they've actually taken in order to decide what to do is helpful for everyone. And this is exactly the conversation we're going to have today. Our guest today, Andrew Rabinovich, is the CTO and the head of ai. For Upwork. Those of you who do not know Upwork, it's a company that existed for a very long while, that I've used many times in the past that allows you to find contracted work on almost any topic you can imagine on the planet from anywhere in the world and resource talent from anywhere in the world. So it's a very helpful way to find people who are good at something. Find what exactly they're good at and hire them to help you on that specific thing. Now, a company like that is in a very interesting scenario right now because a lot of the work that gets offered are things like creating logos for people or writing. Content for people or creating web design for people and so on. And a lot of that work could be done by AI right now. I don't think it's there yet, but it's definitely something I'm sure there have been considering. on the flip side, there's a lot of other work that it cannot replace right now and that still require a lot of human. Intervention and efforts and capabilities in order to make it happen. But they're definitely in a very interesting junction as far as company, and that's why having that conversation with somebody from that company would be very interesting. So again, Andrew as the CTO and the head of ai. Was deeply involved in these kind of conversations, and hence, he's the perfect person to share with us the frameworks and the process that what they went through in order to decide how to approach AI from a company strategy perspective. Like where are we going to be as a company with AI in the next few years in order to best serve? Basically both sides of their audience because they have the people like me who need contractors, but there's a, I don't know how many, but probably millions of contractors on the other side that could use AI in order to make their offering more attractive to people like me. And so a very interesting conversation to have. And since they are in that unique situation and since they're already making step in that direction, I know this is gonna be a fascinating conversation. And so I'm really excited to welcome Andrew to leveraging ai. Andrew, welcome to the show.

Andrew R:

Thank you. Happy to be here.

Isar M:

so really let's, let's dive right in. You kind of know what I just said, right? there's a company like yours that provides services that some of them AI can do today, some of them might be able to do in a few years. Not nobody really knows. How did you even approach this from a concept perspective? Like what was the conversation? How was it initiated inside the company? what do you think about the future from a personal perspective? And then after we. Talk about this, we can dive into the actual process you guys took in order to address this.

Andrew R:

Sure. so to start, um, I've been working in, uh, machine learning, computer vision and AI for about 25 years. So none of these things are, binary to me as it didn't exist before and now it's here. It's a, an evolving process and, gradually improving one. I joined Upwork, exactly two years ago, when my, uh, startup called Head Headroom that was focused on video conferencing, collaboration platforms, was acquired, by Upwork. And the main reason why I. Chose to join Upwork is for the human talent that exists on the platform that you alluded to that really spans all of digital work, that exists in the world. I think there's around 18 million of active freelancers on the platform across, many, many categories, of digital work. the way. I think about AI is not, whether it's sort of shallow AI or general ai, vertical models or foundation models. I tend to think about AI from a human-centered perspective. So rather than thinking about building general intelligence, we think a lot about building human-centered ai. And I've done a lot of this at Google and at Magic Leap at headroom, and now. And Upwork where the, the big component of the experience comes from the humans themselves. the goal of human-centered AI is to not replace people, but to amplify them. And this comes in sort of two flavors. One, understanding that there's certain things that machines are good at and humans are not. And the second aspect of it is that when we look at. Certain skills and body of work that exists today, these are not all encompassing things that will be needed in the future, right? If in the past we needed people who write content to be able to know grammar and be able to spell well. Today, these things are not necessary at all because spell checkin and grammar corrections are at their best and you don't need to really worry about. So if you think about things like chat GPT, moving forward, you can basically, rather than having to write out entire paragraphs of content, you can list out a few ideas in bullet form and then say, go and translate that into pros. So to me that's a very natural extension of a spellchecker. This, to me, this is just a next generation of a tool that didn't exist or existed at with lower capability, and now has expanded its scope and does not replace the tool user, but allows the tool user to augment themselves in such a way that they can tackle much broader, concepts in the work that they're doing.

Isar M:

let me, let me ask you a question before you continue because I, I, I agree with you by the way, a hundred percent with everything you said so far in the courses that I teach, uh, in the lectures that I give, there's a lecture that's called, uh, my Five Rules for Success in the AI era, and one of them talks about, Uh, and it's all about like mindset changes. And one of them talks about, uh, what I call from profession to skill, basically stuff that used to be professions that over the years, just by the way, not just AI became skills like typing, right? So when I was a kid, my next door neighbor was a typist. I don't know any typist that lived today. Everybody knows how to type, so you don't need typists anymore. what I think that is happening though is that. Because of ai, more and more things that used to be profession are becoming skills that more or less anybody can acquire. So I, I, I'll use myself as a very good example. I understand software very well because I was a CEO of several different software companies, but I've never written a single line of code in my life. And now I'm generating really amazing applications for myself and for my clients. And so I don't have a scale of writing code. Anybody who knows how to write code will write code way better than me using these tools. But the fact that I have it as a scale right now allows me to do stuff that otherwise I would've hired somebody on Upwork to do. I'm literally now creating a really new, sophisticated website. And my previous website, I hired somebody on Upwork, which I'm not doing right now because AI is doing the work for me. So where do you see that trajectory from? I understand the human-centric aspect, and I agree with you. I wish that was. The solution that everything was going for. But from my perspective, if I can talk to a machine and not have to go through long iterations and wait for responses and so on, and the machine can do that aspect of the work for me, I'm not gonna hire a human. So I'm, I'm really interested because again, you, you see that at, I see me, you know, a size of one that's not a very good, uh, way to make decisions. You look at. Like you said, 18 million providers and probably 10 x that or a hundred x that. As far as the customers, uh, where do you see that trajectory right now? Where do you think it's going in the future?

Andrew R:

that's a great question. If you think about these skills, or trades. As a closed set, then I think we would be in a little bit of trouble. But as if we look through sort of the history of industrialization and evolution, these skills are evolving and they're, the set of these skills is expanding more than ever. So the assumptions. And the worldview that we have is that while certain skills are going to be replaced by machines, certain new and much more new skills will emerge. That will be much more complex that machines can solve, on their own right. If we think about. People driving cars. Back in the day, you had to like press the clutch, shift the gears, do all these complicated things. Today you sit in a Tesla, you barely have to like, touch the steering wheel. It kind of takes you there. And then, I don't know, uh, where you're based, uh, geographically, but in San Francisco you can like now take, Robo. Taxi. Taxi.

Isar M:

Yeah.

Andrew R:

Uh, so Waymo is, Waymo is cool because it's the most advanced and it's, it, uh, sort of, I've been closely related to it for a while, but there are now self-driving cars in the city that don't even have a steering wheel and they can go backwards and forwards like a train. It doesn't, there's no front door back to the car. Oh, interesting. So this whole scale of having to drive, I would be surprised if my kids like, or their kids. Have that skill. And I would argue that it's like a comp it's fun to be like a Formula One driver to drive as a skill. But today, like there's an a sport of equitarian, right? There are people who ride horses. For competition, but as a mode of transportation that doesn't exist because it's just so antiquated and obsolete. Yes. So the same thing here. I was talking to, um, a professor of computer science at a famous university yesterday, and we almost at the same time said that this concept of software engineering as we know it today. Is a skill and it won't exist in 20 years, right? The fact that you never wrote a line of code and now you're essentially writing code, but you're doing it in English, not in. Python or c plus or any other programming language, but English has become, the language of coding means that this whole construct of writing these weird symbols in a certain sequence that's very difficult to comprehend for an average person. All that is going away, right? And that's gonna be just become mainstream. Now as this happens, the ability of humans is only expanding because if in the past you couldn't do software engineering. Now through the use of ai, you actually can, so your abilities have expanded and we're seeing the same thing happening on Upwork, the talent or the freelancers that do the, all the 18 million that do all this work. They are very openly saying that they want certain parts of the workflows that they're involved in to be automated because these are very mechanical, repetitive. And basically boring parts of their jobs that they don't wanna do, but they have to do them because otherwise they don't get paid. They don't get paid for the fun and creative bits. But when it comes to like implementation, they're like, oh, hire someone else. No, you're, you are the someone else who has to do it. Yeah. Yeah. That's why I

Isar M:

came to you in the first place.

Andrew R:

Yeah. And that's why they're so happy that the possibility of replacing themselves in doing these laborious tasks can now actually be done now. There are two ways to go about this. Either the freelancers that you hire can go and try to replace essentially themselves through vibe, coding, coding agents, all these other systems. Or they can identify systems inside Upwork that can help them do that so that they're really working in a platform as part of a framework that. Systematically figures out which parts of the work requires this high level abstraction and guidance and, uh, evaluation and which parts of the job can be done in a fairly autonomous way. So that's one bit. The other bit is that there's a lot of talk about. Agents, or lar uh, large language models, taking over the world and, and like making a serious economic, disruption in the workforce. The reality is that work is actually very complicated. And if you can do a very small aspect of it in an automated way, does not mean that you can do a task end to end. we recently, published, and, uh, some findings and launched, an internal product in the form of a benchmark that evaluates these AI systems. where rather than focusing on stale and academic benchmarks, like solving SAT tests or even math olympiads, we take real jobs. From the Upworks marketplace that clients have paid real dollars for, and freelancers have delivered the outcomes for those jobs and tried to run them with the best in class, agents across different categories, and not to our surprise, but many to some others. These agents don't perform that well. Depending on a particular agent, you get performance, again, depending on the category, around 20 to 40% on the initial attempt of the agent to complete a task. Now, this is very consistent with the findings from other groups, in the industry and academia, but what's very unique about this evaluation is that we went a step further and we said, how would the agent perform? If it received multi-term guidance from a human expert, rather, so you, you give, I don't know, maybe you probably use cursor or something like that to, to build your website. you give it some prompt and it creates something. And 10, three out of 10 times it gets it right, but the seven out of 10 times it doesn't. So typically you don't stop there. Whether you just have a conversation with chat GPT or any other thing. You're like, you did this part correct, but this part needs a little bit more emphasis. Go and fix that. And through this multi turn interaction, it turns out that agents improve by some ridiculous amounts of like 70, 80%. But what's. That's very interesting'cause no one's measured that before. But what is very interest, not interesting, but surprising to see, is that this multiterm interaction between the human and the agent leads to a comparable output of a human doing the work alone, but it happens a thousand times faster. So it, it's as if, you know, in ancient Greece, you were a mathematician and you had to do some calculations and you will use like sticks and rocks to do it. And then I show up, I'm like, here's a calculator you would arrive at. Essentially the same answer. But it would happen instantly as opposed to taking you six months to figuring this out. Right? Yeah. So these agents are nothing but tools. These are statistical models that can't think, that don't have emotions. They don't have a goal, and they certainly don't understand how the world operates. There's no model of the world in these systems. It's just pattern matching systems. They're just. They match very, very large, patterns and they can't really think outside of distribution. So by providing these tools on the platform, we believe that it will make the job of a freelancer on Upwork much more effective and efficient as opposed to them having to use these tools off platform, which we know they already do.

Isar M:

Yeah,

Andrew R:

so this just streamlines their work.

Isar M:

So a, a few things. First of all, I agree with you a hundred percent. if, again, we'll use my example of developing a website. I've been working on it for, if I add all the time together, probably 24 hours by now. Uh, so it definitely wasn't a give it a prompt and, and you get a website. It's not what it is. There's the level, even, even in the website that I'm developing that is not crazy complicated. There's a lot of complexity. Uh, you know, there's different variations, uh, uh, break points. So you have a large desktop and then a tablet, and then a mobile device, and then you have different types of motions and they, like, there's, and you know. Engagements and stuff that needs to happen on the website there. You know, in backend and front end, there's definitely more than one problem. So I agree with you with that a hundred percent. And like I said before, somebody who knows how to code would've done this even faster than me because some of the stuff that I struggle with would've done better. The question that I have. And then, and then we're gonna dive into kind of like really the process that happened in inside of your company and what's the framework that you've used in order to decide what to do moving forward? The question that I have before that is because of the. Speed. This is moving right now? Yes. Right now, this one prompt. Now I had to spend 24 hours to develop this website and I'm not done yet, so it's probably gonna be 48 by the time I'm done. 48 hours of real work, meaning actual, you know, this will spread over a week, but actual work, work probably 48 hours, in. A year, two years from now, five years from now, I will be able to describe in more detail what I want and say, oh, pay attention to these kind of break points and these kind of devices, and make sure that this interaction happens and I want this to be available in the backend and it needs to connect to my CRM and all these kind of things, and it will just do it. And then my input goes down from 48 hours to maybe. 40 minutes. which means the fact that I have the knowledge and I've, I've been a CEO of several different software companies that kind of understand how software works, that won't be necessary because it will understand everything that I understand much better than I do. So I, I agree with you a hundred percent right now. Like there's, there's zero disagreement on where we are right now, on these agents. And if even if we look at, you know, GDPV, which is very similar, like it's the benchmark that OpenAI created, very similar to your idea. Like, here's 44 professions, 90 whatever, I don't remember how many tasks. Uh, and, and the best models right now performing around 40%, uh, the best top of the line models, which means they do shitty work because more than half the time, they don't get it done properly. But six months ago they did it at 20%. And so that's my question. Do you think that in a year, two years, five years, we get to a point where there's very little, we can do better than the AI can do on its own?

Andrew R:

so you draw parallels in improvement, between completing. Real world tasks and the performance of foundation models. You see that the jump between pre GPT two models, which sequence to sequence models existed for a very long time. This is not, this is not a new idea. Um. the change between GT two and GT three, three to four, and now four to five, you're starting to see it slow down. The reason why it's slowing down is twofold. A, because the algorithms are not improving. It's the same, transformer architecture. It has improved a little bit. When we added this concept of. F reinforcement learning from human back, and then the reinforcement learning from verifiable feedback. Um, but the algorithms are the same, and the amount of data that you feed these algorithms is not, is, is basically dried out. Right? Yeah. It's very, very difficult to generate data that doesn't exist already in such a way that it's net. New compared to the existing distribution, right? That requires discovery. And that's kind of the bottleneck of evolution anyways, right? We don't know what we don't know, and if we knew it, we wouldn't need these things to do it for us, right? Yeah. So that's, that's kind of the issue. so I do agree with you that there's a, all these agents are powered by, foundation models. Whether a general purpose or hyper verticalized ones, and I, I am, I'm a, I'm a believer in the vertical AI over this like general purpose stuff because mixture of experts is still the path to bring it all together. I do believe that the quality of the tools that use these even non improving foundation models will improve. I don't think it'll improve. A hundred x like you're suggesting, maybe it'll improve twice. So instead of using 40 hours or 45 hours, it'll come down to like five or 10 hours, uh uh, that you'll have to do. And that will not even necessarily change. From the performance of the models, it'll change from yours, your ability to figure out how to do it. So this whole concept of prompt tuning or prompt engineering expands beyond like one line of questioning to this whole thing. Like, I've built this app X many times, I know where the pitfalls are, so I'm just gonna tell it in advance so it doesn't like. Screw up on its own many times it'll just like follow my instructions and that. So that's how you get better at it. It's like playing a sport, or piano, whatever. You try it for the first time, it's very difficult, but then you have habits. the biggest benefit I think in platforms like Upwork is that tasks. That freelancers work on come from humans who have an ever evolving. Variation in need. Yeah. So people don't, if everybody kept asking for exactly the same website to be built, then it would be very, very difficult to leverage anything because we, they just have a distribution and new sample from that. The cool thing about Upwork is that people are coming up with very new ideas all the time. You know, you can. What used to be, as you pointed out, earlier people would come in and say, I need a copyright, or I need a logo, or I need like a very basic e-commerce website. Now people are coming in and they're saying, I need to build software for medical diagnosis. Or I can, I imagine in the next. when self-driving cars become abundant and everybody owns one, you'll be able to go to Upwork and say, I need help programming my self-driving car because I wanted to like, not take highways or do this like all kinds of things you can,'cause it's, it's basically a robot, right? And you can program your Roomba, you can program your self-driving car, like whatever. So the um, the set of. Asks or jobs that will be required to do on Upwork is going to evolve and it's evolving extremely rapidly. And the categories, we have internal discussions about how the categories of work are changing and there, and it used to be that. 10 years ago, somebody wrote down a list of categories and they broke them down in some hierarchy. This is like web development. This is engineering, and this is like data science. Now these things evolve so much that we actually build algorithms that reclassify categories of work. In dynamically so that we actually know what it is that people work on, and then we can draw similarities across jobs so that every task is not some bespoke idea, but is actually a subset of a meta con construct that we can think about and provide, useful feedback on.

Isar M:

Fantastic. I love it. By the way, I, I love this concept of. The reason this is not a finite or a zero sum game is because the requirements keep on changing, and hence the specialist keeps on improving what they know how to deliver because that's what people are asking for. yeah. Awesome. Let's, let's go into the actual process that happened at Upwork, right? So, so you, you look, you, I don't know, a year ago, two years ago when you started, you're looking into the future saying, okay, this thing is coming. It's gonna change a lot. How do you look at it from a company strategy perspective? Like what were the steps, and there was one thing you already mentioned that I think every company should do is really evaluate on the best level you can. Again, you are a very capable software company with lots of data, but on the best way you can, how that is actually going to impact your work. Right. You just said, you, you did the evaluation. You said, okay, here are the tasks that we are. Allowing people to get solved, how much AI can actually do to really evaluate the risk as it is right now, maybe as it's going forward. But break down, like when you sat down for the first time, what were the things you were looking at, and then what were the steps after that? What was the framework that allowed you to decide how to approach AI from a strategic perspective?

Andrew R:

when we started looking at the Upwork experience end to end, We realized two things, right away. First is that when people come to Upwork, they ultimately have an end goal that they want to achieve. um, and hiring someone to achieve that goal is really an intermediate step, right? You want to have a website that helps your business. You don't know how to build that website. So you come to Upwork, you hire someone, and then off platform, they help you build that website. This is very, very complicated. in machine learning, we call the inverse of that end-to-end learning as opposed to trying for, for instance, in self-driving cars, you don't. Train models to first detect stop signs and pedestrians. Then you train another model to take action. And then the fourth, the third model is to do planning. You do end-to-end learning where you're saying, I'm in point A, I need to get to point B, and I can't hit anything along the way. So that's like end-to-end learning. So in our case, we said, what is it that we need to do to make sure when a client comes into the platform and they say, I want to have an e-commerce website for my pastry shop. We don't say, you need to hire a Java developer. We say. Here's the e-commerce website that you need. So we go beyond matching and we actually deliver outcomes. later we can talk about how that actually happens, but that's like the first step in the process that we said. We have to extend the experience so that people don't just find someone who can do things for them, but they can just get what they want. That's the first bit. The second bit is much more. Practical. because traditionally clients on Upwork had to be very technical because when they came to the platform, they had to specify whom they wanna hire. And if you wanna do digital work, again, an example, you want to build a website. But you know how to bake cookies and you don't know anything about websites. The que the question is what do you ask for and how do you even know if you're hiring the right person? How can you evaluate if the candidate is correct? So in order to expand the set, uh. Of clients who can use these services. We decided to move beyond an ability for someone to technically describe what they want, and then being able to technically evaluate the results to basically turning this whole problem into a conversational, dialogue between a client. Uma, which is Upworks, uh, meta agent that drives all the experiences on the platform. Um, so rather than coming in and specifying exactly who you wanna hire and then looking at the results, clients now come in and they basically tell Uma what they wanna build and then Uma figures out who they need to hire, to get that done.

Isar M:

Interesting. Okay. So what I hear you saying is two different things. and I love both of them. One is, from the concept of a jobs to be done perspective, right? Don't look at the steps. what is it that you're trying to achieve or from a company perspective. What is the goal that you are serving for your clients? Right? If we try to generalize this, like, why do people come to my company? What are they trying to achieve? Uh, if, even if you look at, you know, at, at Salesforce, people don't care about Salesforce's software to, to be fair, it's really horrible and really hard to use. They want more paying clients. And that's a means to that end. Right? And the closer you can get them to that end in the most effective way, the better off you are. Going back, by the way, connecting very well to my five rules for success. One of these rules is stop thinking efficiency and start and start thinking outcome. We're so trained as humans, uh, especially business people, to think in steps like this because this is how. The world used to work. And now AI can circumvent a lot of these steps because it knows how to do some of these steps on its own. So you need to stop thinking about the steps you're doing today and go to, okay, but what do I actually need? Like, why am I doing this? And I, and I love the fact that this is how you approach this. Uh, the other thing that you're saying is how do you. Reduce friction in your ecosystem. And again, I'm trying to generalize what you said. I have people who have a need on one end, and I have the people who are, or if you want the inputs to my company, my, the rest of my ecosystem, my su, my suppliers, my inventory. Like all the stuff that makes what I deliver valuable to people. How do I use ai? To reduce the friction to the bare minimum, right? These are basically the two things you're saying, and I agree with both of these a hundred percent. Like if you look at the value that you provide today and why people come to you and you figure out how to help them identify the value and capture the value in a for more effective way, you are leveraging AI in a way too. Establish yourself to be more successful in the AI era. I have two follow up questions for that. One is, how do you project the change in what they will find valuable? And maybe in your case it's less of a thing because then your e the other side of your equation will figure out a way on how to solve that. But you, you said as an example, I want to give them the result versus having them walk through the steps. What does that mean?

Andrew R:

so let's start with the, with the second question. when clients come to Upwork, they have a varying degree of a conviction of what a good result looks like. and they traditionally. Have this expectation that once Upwork finds them the right talent to work on the problem, then through these interactions and brainstorming sections, they will converge on the outcome they want. and this, we, again, we have a lot of data that, that points to this. So we decided to flip the problem, upside down and have that discovery process happen between the client and Uma at the very, very first step of the process. So when client comes in, they describe, they de tell Uma what they want. Uma turns it into a project plan, and then they, they, the project plan is, is used as an atomic unit of work. That goes back and forth between the client and Uma until they converge on something that the client says, yes, this is exactly what I want. And then Uma can go back and say, well, what about a shopping cart? Did you think about like a checkout process that you wanna have on your, uh, bakery website? And the client may say, oh, I didn't think I need one, or whatever.'cause they just don't know. And they don't, they shouldn't have to. Um, so through this process, they solidify the project plan and then Uma goes and finds the relevant. Freelancers and in the future, AI agents who can collaborate to deliver the work product that, the client, is looking for. Once the work is complete, the next step is to provide some kind of confidence that what the client asked for is actually what's built. Because especially in the case of someone non-technical as a client, you give them a bunch of files or maybe even a URL and say, okay, great. This works. They say, this is wonderful. They use it for three days, and then in three days, like the domain expires or something. And it, uh, there's not, there's not much. And what do you do then? Like, you go back to Upwork, you go back to the freelancer that did this for you. It's not obvious. So the next logical step is for Uma to actually verify that what's been done is what's been asked for. And then it can provide the outcome to the freelancer, I mean to the client, in the, in this Amazon Prime guaranteed way. If you don't like it, you can return it because we're confident that what we built is what you want. this style of digital work delivery is fairly new, but unlike physical delivery, whether it's, GrubHub or Uber Eats, or, driving Uber cars or any of these. Delivery systems.

Isar M:

Yeah. Yeah.

Andrew R:

If you understand work from the concept of tasks, then even if something goes wrong, the amount of fixing that is required is actually quite minimal. And that's why this benchmark, the, uh, this, uh, happy index that we, uh, published is so powerful because chances are the success of an entire job. Depends on the sum of the successes of the tasks that comprise this job. And if you understand how to break down a job into tasks, if something is wrong, you'll help. You only have to maybe augment or modify one or few of these tasks out of the many that are necessary to comprise it, and that that's. A is very data driven based on all the many years of Upworks, work history, and b, it really leveraged this combinatorial and statistical nature of AI to be able to solve these things. Now, none of this takes into account the human ingenuity or creativity part, and that's why none of this can be done purely by machines. And that's why you need the human expertise in the loop along the way.

Isar M:

Fascinating. Okay. You said something that I was literally about to ask you next and you already hinted to it, so I'm gonna ask, I'm gonna go there and I'm gonna make it even more specific. You have, and again, I want to generalize this to other companies because what basically you're saying is saying, okay, I know a client comes to me for x. Maybe they don't exactly know what they come for me for. So if they buy, if they come to buy shoes, well, okay, they need shoes. If they're buying to buy a chemical, they need that chemical. But there's still steps in the process or many cases where that is not the case. And if you can help the client better define their needs, then they can get. To the outcome they're trying to reach in a much more effective way. And if you can help them get there faster, that's a very helpful thing to do. So I think that applies on almost any business, whether you're manufacturing, whether you're a middleman, whether you're a delivery company, it doesn't matter. You can help your clients better identify their needs and get their faster. You're gonna be the go-to company they want to go to. But what you said then is that, okay, now that I know all the different steps. So let's say there's 17 different steps to get to that outcome. I can identify the seven steps out of the 17 that AI can do right now. And because I've identified them very well, I can hand those seven to AI immediately and then hand the other 10 to humans. Or if it's a sequential thing, you know, whatever. Step one is ai, then it goes to a human and it comes back and so on. Do you see this as. The future of more or less anything with AI just getting injected into more and more steps and humans, as you mentioned, varying their expertise that been applied in different steps of whatever process.

Andrew R:

Ab, absolutely. I'm not, I don't have a very prescriptive or definitive answer whether it's like the first seven are done by machines and the last 10 are done by humans, or they're all done by human and machine collaboration, where in every step the machine does 90% of the work and the human does the last 10 of providing instruction and then verification like that, that split is very, task. Uh, specific, but I firmly believe that in all aspects of digital work, there's some components of the work that's generalizable across. Many, many, many jobs and there are some things that are unique. The unique parts will be done by humans, and the generic bits are always gonna be done by machines because they're just faster, cheaper, and much, much more reliable than humans. If you think about. Simple things that we experienced today and, and, and, and it's, now that I said it, it sounds kind of strange that because it's, it's not simple in the sense of, of the engineering, but like something that we routinely do. You go instead of. Asking. I was changing tires for my car yesterday and I told the shop which tires to put on. And they're like, oh, did you read consumer reports about the tires to get? I'm like, no, I just go to Chad GPT. And I asked them that. That's the, that's the simple bit. it gets the sort of baseline understanding of everything extremely quickly. You can trust it up to a certain amount. And then there's this concept of hallucination, which it, which some people consider it as like lying, but I view it as in, I think that's what the next logical thing is. And in machines, the notion of confidence just is, is about next word, prediction. So they don't know whether they. Hallucinate or not because the confidences are the same, but there's still this hallucination realm where they tell you things that don't exist. And that's where the human can come in because the human does have a representation of the world and they can take this, prediction, fit it into the realm of how things work, and then say if this actually makes sense or not. Or at least provide a notion of uncertainty, which machines have a very hard time dealing with.

Isar M:

Awesome. Interesting. So, uh, I, I wanna ask if there were any follow up steps that you did, and how, and kinda like what's the current status and where do you see the next steps, like the current status? there's Uma, you kind of explain what it does. Where do you see this is evolving in the next few years as a company?

Andrew R:

Sure. so again, when I started looking at this whole, Process from. Clients coming into the platform explaining what they want done. Then matching recommendation discovery happening. Then eventually client hires the freelancer and then extended it to delivering outcomes. We've taken every single one of these steps. And we've automated it with uma. So UMA is your personal assistant. Along the way, they can help you draft the proposals, they can evaluate, then they can find the right candidate match, like answer questions, all these kinds of things. So we've essentially, Uma has now all these skills that it can help a client through the journey on Upwork. The next path that we are going to is. This concept of capabilities something, and, and there, and luckily there's only three, so it doesn't, it's not that, time consuming to describe, but basically we want Uma to be able to reason. Reasoning models are kind of evolving. there there's some discussions of how well do they actually think and reason or is it just more pattern matching? So there's, there's a concentrated effort around that. Then there's effort about deep memory. Because there may be things that you say in session that may be consistent or inconsistent across sessions. It may be consistent or otherwise across different activities that you do on the platform. And they may certainly be different across different users on the platform. So, um, a is able, and that's, to me, that's kind of like this model. Of the world where Uma can make sense of what's being done and discussed and put it in the larger context of what's actually happening on the platform. And lastly, UMA needs to be able to provide guidance into solving problems that are difficult. So if we think about simpler tasks like board games or video games, we have seen. Very definitive examples where machines were able to come up with solutions that no human has ever seen in the history of the game. The reason why this is possible is because in re, the formulations they use revolve around reinforcement learning, and one of the critical components of reinforcement learning is providing reward. More gen generally having a value function that generates the reward in unknown circumstances. In board games and video games. The model of the world are basically the rules of the game. They dictate the reward. You make this move, you either win or you lose. it's fairly straightforward. And then you can try all the trillion combinations through self play to identify these unique opportunities like move 37. I think fortunately the human world is much less constrained. Then these video games and a lot of the solutions are very qualitative. And whether it's, you know, digital work on Upwork or drug discovery or philosophy or science in general, there's not a lot of things are mathematically verifiable, which makes reinforcement learning basically break because you don't know where to get these rewards. I think in an environment like Upwork. Provides a, albeit maybe a lower dimensional representation of the physical world, but at least an environment where agents can go and try novel solutions. And get feedback from human experts to determine if whether they provide solutions that are correct or not. And that allows you to do this self-exploration. So you give an agent a task and it goes and tries something. Maybe it's not a task that a human can achieve on their own, but an agent would try many, many, many combinations, and the goal role of a human would not come up with a solution, but to only verify if the agent provided solution is correct or not. So this has a lot of similarities with complexity theory because solving hard problems is exponentially hard, but verifying solutions. To exponentially heart problems is polynomial. Tractable.

Isar M:

Yeah.

Andrew R:

Which makes it a very, very big difference. So we sort of see the world where this environment of digital work allows both agents on Upwork and third party agents to improve themselves by doing, as opposed to training themselves on data that's been generated by humans. So in other words, rather than having humans. Label data for training these agents. The Human Feedback Pro is available at inference time as opposed to training time, and that's a huge paradigm shift to what's being done today.

Isar M:

Andrew, this was really fascinating. I could have kept talking to you for another two hours probably. I, I like the biggest question that I have in my head right now. And, and I, and I'm not gonna ask you that because we're short on time. Uh, but, but I, I want to keep it as an open question for other people. Those of you who've been listening to me for a while kinda like, know what I think. But, I, I love the way you framed these things and how you took us through this journey. But if really. Agents will do a lot of the work and they will figure it out because they can just do it faster, better than we can, and they will give it to us to decide, here are three options, which is the best one? Or, here's my suggestion. Do you think it's legitimate, uh, or not? Means we'll need less people. Doing work. Uh, so yes, we will need people. It's not like we won't need them. They will be a part of the process. I just think we'll need less of it. Which chito generates a problem when it comes to the workforce as it's gonna be, uh, in the future, but we maybe it will evolve in ways that we can. Anticipate yet. And that's a whole other thing that, again, just a big, uh, none of us has a crystal ball to figure that out.

Andrew R:

um, I'll, I'll, I'm not gonna go into details of answering this, but I just wanna leave you with this notion that the number of jobs has historically increased over time. Agree. And I think the same thing will happen, the jobs that we know today. Some of them will be replaced by machines, but the jobs that we don't know about will be, will require even more people that have jobs today.

Isar M:

Amen. Andrew, this was really fascinating. I really appreciate you coming and joining us. If people want to follow you, learn from you, uh, work with Upwork, like what are the best ways to connect with you and, and, and make the next steps.

Andrew R:

I publish, on various, uh, domains, um, but in my academic work can be seen on Google Scholar. but otherwise you can, uh, reach out to me on LinkedIn.

Isar M:

Awesome. Andrew, thank you so much. Really fascinating. I really appreciate your time and everything you shared with us today.

Andrew R:

Thank you. My pleasure.

Head Shot

Isar Meitis

Host