Behind The Tech with Kevin Scott - Neha Narkhede: Author, entrepreneur and cofounder of Confluent

🎁Amazon Prime 📖Kindle Unlimited 🎧Audible Plus 🎵Amazon Music Unlimited 🌿iHerb 💰Binance

NEHA NARKHEDE: So you know, why we wanted to open source it is this was an observation we made is, you know, this trend that you know technology companies are going to essentially be defined in software, you know, not just use more software, but literally all the actions and decisions in a company happen through software. It’s going to be a trend.


KEVIN SCOTT: Hi, everyone. Welcome to Behind the Tech. I’m your host, Kevin Scott, Chief Technology Officer for Microsoft.

In this podcast, we’re going to get behind the tech. We’ll talk with some of the people who have made our modern tech world possible and understand what motivated them to create what they did. So, join me to maybe learn a little bit about the history of computing and get a few behind-the-scenes insights into what’s happening today. Stick around.


CHRISTINA WARREN: Hello and welcome to Behind the Tech. I’m Christina Warren, Senior Cloud Advocate at Microsoft.

KEVIN SCOTT: And I’m Kevin Scott.

Today, our guest is Neha Narkhede, cofounder and CTO of Confluent, one of the initial authors of Apache Kafka and a former colleague of mine.

CHRISTINA WARREN: And, you know, Neha is on a Forbes list of the top 80 self-made women in America. And this is actually kind of an amazing list. It includes women in sports and TV and music. Beyoncé is on this list, Serena Williams and Oprah are on this list. But it also has a lot of women in tech and in business like Sheryl Sandberg, who’s the COO at Facebook, and Safra Catz from Oracle and Marissa Mayer and a number of other women in – in tech and business.

And I think it’s so cool that women in tech are counted among some of the most successful entrepreneurs in the country. And I’m really curious to hear about Neha and how she got to where she is today.

KEVIN SCOTT: Yeah, I mean, it’s no surprise to me at all that Neha is on this sort of list. It may actually be the closest I’m ever going to get to Beyoncé is my association with Neha.

But, you know, I got the privilege at LinkedIn of watching her develop as an engineer and as an engineering leader, and she’s absolutely spectacular. So, I’m just really excited to chat with her today and to have her share with everyone else like this really amazing and in some ways like an archetypical Silicon Valley tech industry story with everyone. She’s awesome.

CHRISTINA WARREN: All right, let’s hear from Neha.



KEVIN SCOTT: So, today, we’ll chat with Neha Narkhede. Neha is one of the initial authors of Apache Kafka, an open source software system that lets organizations process data in real time. Neha helped develop Kafka at LinkedIn where she was the lead of the streams infrastructure team. Neha is cofounder and CTO of Confluent, a company whose mission is to help other companies use Kafka and adopt modern streaming infrastructure. So welcome, Neha, thank you so much for being here today.

NEHA NARKHEDE: Thank you for having me, Kevin.

KEVIN SCOTT: Awesome. So, I want to start with your journey through computing. So, how did you get interested in computers and programming in the first place?

NEHA NARKHEDE: You know, Kevin, I was a very curious sort of self-learner (inaudible) as a child. And so when I was about eight, my parents bought me my first like personal computer. And it was sort of unique, you know, in those times back in India to have a computer.

And so somewhere deep down I was very appreciative, and then it sort of became like the tool that fueled my curiosity. So early days was just playing Prince all day and I was big into art, and so I just drew on it with MS Paint or something. And pretty basic. And then it taught me, you know, computering, programming and so that was sort of the start of the journey.

My dad used to tell me a lot about technology and how it’s going to have a huge impact on basically everything we do. And so that would be a good career bet for someone who wanted to learn so much and create new things. And so that sort of planted that seed in my head is – (Crosstalk.)

KEVIN SCOTT: And was your dad an engineer?

NEHA NARKHEDE: He was a mechanical engineer. And in our family, there were a lot of emerging technologists who moved to the U.S. And one of my uncles started Cirrus Logic and that went public. And sort of, you know, started from nothing, right? And so, I sort of saw that journey and thought, “Wow, you know, sitting here, I could actually dream about being a technologist.” And –

KEVIN SCOTT: So, role models were really important.

NEHA NARKHEDE: They were really important. And my parents did this thing where, you know, they told me stories about women who broke glass ceilings and were trailblazers. And they did that pretty early on. But they gave examples of every walk of life, so my dad had me read these books about Indira Gandhi, who was the first female prime minister of India, Indra Nooyi, who basically had origins in India and went on to become CEO of a multinational company. Kiran Bedi, you know, first female head of Indian police services.

And that sort of, you know, somewhere, you know, deep down, I think it sort of developed a sense of empowerment now that I look back on, hey, if they could do the impossible, maybe I could try, too.


NEHA NARKHEDE: You know, and so I think I had a bunch of different sort of role models, but my parents sort of, you know, made me believe that if they could do it, then I could do it, too.

KEVIN SCOTT: That’s really fantastic. So it must have been a big change leaving India and coming to the United States and going to graduate school at Georgia Tech.


KEVIN SCOTT: And like being in Atlanta, which –


KEVIN SCOTT: – I’m guessing is a very different sort of place –

NEHA NARKHEDE: It’s very different. (Laughter.)

KEVIN SCOTT: – than where you grew up.


KEVIN SCOTT: Like, how did you decide to do that? What was that decision-making process?

NEHA NARKHEDE: I think it happened over the course of time. So, my brother decided to do his bachelor’s here, so that was pretty early. He came here when he was 17. And back then, we had this, you know, family green card and it was very hard to get that, but my father got it and my brother got it.

And I said – I don’t want it. You know, I don’t want to move to the U.S. This was, you know, maybe I was in 10th grade or something. And so we gave it up. So, I actually gave that up – the golden ticket to the U.S. (Laughter.)


NEHA NARKHEDE: And then later on, you know, as I was studying computer science and got to sort of see its impact on, you know, basically every aspect of our lives. And, you know, back then, I was just sort of doing that by reading up on the Internet.

And so, you know, when I first thought about taking it seriously is when I said, “Well, now I want to go to the U.S.” And it is sort of a big deal, you know, from – if you come from the middle class to sort of go to an expensive university, that was going to be a pretty big leap.

And by then, I had earned scholarships to sort of fund my education, right? So, I just had free education up until my bachelor’s of computer engineering. So, what I told my parents is, “I don’t want you to pay, I want to try to get fully funded.” Right? So, I made a list of all the universities where it was possible to get either a research assistantship or maybe a teaching assistantship. And I applied there. And that was my journey, I think, I worked pretty hard. It was pretty hard to do that.

And so, when I spent the first few weeks, I spent it in Silicon Valley, right? That’s where my family lives. And from there I go to Atlanta and it was a huge shock. I mean, everything is different, you know? The way people interact with each other is different. Georgia Tech is in Midtown, and that’s right in the heart of Atlanta, and that’s very different from suburbia in the Valley.

KEVIN SCOTT: And Georgia Tech is like a really good computer science and engineering school.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And so, like not only was it this new city and new culture you’re getting accustomed to, but I’m sure that first year was –

NEHA NARKHEDE: It was pretty tough.

KEVIN SCOTT: – pretty hardcore. (Laughter.)

NEHA NARKHEDE: It was pretty tough. They sort of have these mandatory operating systems and theory classes, and I’m still wondering, like, what I actually learned in theory and if it was applicable to what I do right now. But it was pretty hard, right? And I was ambitious. I wanted to finish it within three semesters. Didn’t want to stay all four semesters in order to get the right H-1B path out. So, yeah, it was a tough I think period of struggle, I would say.

KEVIN SCOTT: What was your favorite class either undergrad or grad when you were studying computer science?

NEHA NARKHEDE: I want to say it was – no surprise that it was operating systems. Because it wasn’t just, you know, reading books and taking tests, it was actually reading a bunch of research papers and then in the second half, actually building something new.

Working in groups, and so I think that sort of hands-on experience and using the latest sort of research to learn what’s happening in the operating system world was the most interesting part.

KEVIN SCOTT: And was –

NEHA NARKHEDE: I almost flunked, I think. There was a point when I was like, I need to pull two-nighters. (Laughter.)

KEVIN SCOTT: Nice. Well, so you know it’s really interesting, when I was studying computer science, both undergrad and in grad school, the operating systems course was usually – it was either that or compilers, where it was the first place where you had to write a really big program for the first time.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And it was like complicated, it’s the first place where you’re really thinking about software architecture and like how all of this stuff fits together.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And like how you’re, you know, sort of dealing with all of the modules and decomposition of things. And so, was that like the first big code that you dealt with was in that OS class?

NEHA NARKHEDE: Pretty much, that’s why it was hard. And that’s what I had to pull like two-nighters. But that was – I think it was interesting and in some ways sort of built the foundation that helped me with Apache Kafka, because that – it sort of teaches you forced principles thinking, you know, right from the primitives, right, how the kernel works and how memory works and how multithreading works, right?

It sort of lays that foundation, and that program, the reason it was really hard is because it was the first time I had to actually work in a group where what we decided to do was each write different components and then integrate it at the very end. And that’s when I learned that integration is where most of the trouble happens. So, it took two all-nights, you know, to resolve that.

But I think the first principles, hands-on thinking and programming was really the magic.

KEVIN SCOTT: Yeah, well, I mean, it’s interesting that you had that experience and that it was so challenging.


KEVIN SCOTT: Like, everybody learns a lot when you go through sort of a gauntlet like that. But the more interesting thing is like you – the thing that sort of defines your career is like you really are a systems engineer.


KEVIN SCOTT: And so like you sort of face this big, daunting challenge. And then you decided at some point to lean into it.


KEVIN SCOTT: Like, this is what I’m going to spend all of my time doing is like building – building distributed systems.

NEHA NARKHEDE: Absolutely. That’s when I first started thinking about I want to be a systems, you know, and more importantly, distributed systems programmer.

KEVIN SCOTT: So, what did you do after you graduated?

NEHA NARKHEDE: You know, I sort of focused on databases more in my master’s. When I graduated, I actually – before I graduated, I interned at Oracle in the database group. And so, I decided to take my full-time job there because this was beginning of 2008. And so, we were facing a lot of challenges and –

KEVIN SCOTT: Okay, you’re a lot younger than I am. (Laughter.)

NEHA NARKHEDE: Yes. (Laughter.) Yeah. In some sense, it gives away our ages. (Laughter.)

But this was like beginning of 2008 and, you know, just before graduating, I’d taken this business economics class where we learned – we were putting together a case study about the top five banks. And so, we could see it coming, right?

And so I decided to take the job with Oracle rather than go to the startup world, where I really wanted to be, it’s because I really wanted to make sure I could stay in the country safely, right?


NEHA NARKHEDE: So, it was a very H-1B safe path into Oracle. So, I think Oracle was an interesting experience, I want to say. On the one hand, I got to learn what it takes to create high-quality enterprise software.

KEVIN SCOTT: Which team were you in at Oracle?

NEHA NARKHEDE: I was in the database search team. So, Oracle Text is what it was called.


NEHA NARKHEDE: Search within the database. And so it was interesting, I thought, but it was also, you know, slow – the velocity at which I wanted to learn new technologies, going back to my roots, I had given up a lot to get there.

And so, I wanted to make sure that I’m learning a lot. And so that’s when I spent one year at Oracle. I made up my mind that I want to take the risk anyway, and just venture into the startup world with a particular focus in open-source communities, because I thought that open source is going to allow me to learn more, and LinkedIn was on the short list because we had a nice page where we advertised our open-source projects and the teams that worked on it. So, it seemed very cool.

KEVIN SCOTT: Yeah. And I’m going to want to go back to that in just a minute because, like, I think that’s one of the – one of the very clever things that Igor – Igor did.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And, like, we’ll talk about who Igor is in a second.


KEVIN SCOTT: So, did you have like clarity in your head about how all the immigration stuff was going to work then? Or was it still like a big risk leaving stable Oracle? Like, when you joined LinkedIn were – like I forget, like, were we public yet or –

NEHA NARKHEDE: No, we were not public, we were –


KEVIN SCOTT: Yeah, so, like, you’re joining this like private company that you – like you don’t know when it’s going to go public, like, there’s still all of this uncertainty around it.


KEVIN SCOTT: So, like, how were you thinking about the H-1B and –

NEHA NARKHEDE: You know, what was going on in my mind is the H-1B thing was sort of the main driver for going into a big company, but behind that were years of struggle, right, to get to that point.

And so as I was struggling, you know, one year down the line to think about, hey, what should I optimize for, right? I’m here now and it’s a different country and you know, all my people are far away and I’ve given up a lot to learn so much.

I talked to my husband and what he said is, you know, do what you – you know, you want to minimize your regrets. And so, if you’re here, you want to think about, you know, what would minimize your regrets in the next five years? And so that’s when I said, “Well, I really need to go dabble in the startup world and just figure it out. And if it goes wrong, you know, I’m going to try and come back to a big company anyway.”


NEHA NARKHEDE: So back then, LinkedIn – when I told my friends, you know, this thing about LinkedIn is people just didn’t believe that it was going to be this big behemoth, right, of a company. They just like – it’s going to be bought by Google and it’s you know maybe going to be a midsized company.

And I was like, well, let me just interview there, right? And so when I interviewed people, the people really convinced me, you know, the – sort of the culture was of inclusion and when I interviewed at a couple other social network – networking companies, I didn’t sort of get that. It was a lot of like you know nerdy sort of individualistic, move fast, and LinkedIn I think really convinced me that it was about this community and about the open source contribution. And that’s kind of why I weighed the risks and said, “This is the right company to take the leap with.”

KEVIN SCOTT: Yeah, and so when you joined LinkedIn, you went into Igor Perisic’s group.


KEVIN SCOTT: So Igor is now LinkedIn’s Chief Data Officer and Igor worked for me for six years.


KEVIN SCOTT: Yeah, and Igor is – every time I – every time I see him, I tell him he’s my favorite Swiss PhD. He is. (Laughter.)


KEVIN SCOTT: And he’s also one of my favorite people in the – in the whole world. Like, Igor is a – is like a wonderful brilliant technologist and genuinely good human being. And he was running this group at the time that was called SNA –


KEVIN SCOTT: So, Search –


KEVIN SCOTT: – Network and Analytics.

NEHA NARKHEDE: That’s right.

KEVIN SCOTT: And so he owned the LinkedIn search engine, he owned all of the LinkedIn Graph stuff, which is, you know, sort of this foundational piece of infrastructure that made LinkedIn possible.

And then he owned all of the sort of analytics stuff, which were like all of the data systems that had to operate at scale. And so like it’s really interesting that you were attracted to LinkedIn because of the open-source stuff, because that’s how I was attracted to LinkedIn.


KEVIN SCOTT: So, at my startup, AdMob, we at one point needed to – we needed to deploy a key value store for our ad systems.


KEVIN SCOTT: And like we evaluated a whole bunch of stuff, including Voldemort, which is one of the open-source projects that you know your –

NEHA NARKHEDE: Co-Founder (inaudible)

KEVIN SCOTT: Your co-founder Bill – Jay Kreps in Igor’s SNA group and like it was the best-performing key value store for our particular use case at the time.

And like we used it and like – and also because it was open source, like you not only got to use the software, but we interacted a bunch with some of the engineers at LinkedIn. So, you saw like it’s great code, it’s like super –


KEVIN SCOTT: – good piece of design.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And the engineers were great. And so, when I was thinking about where do I go, like, I knew some folks at LinkedIn.


KEVIN SCOTT: But like the important thing to me was like, oh, there’s some really great engineers there.

NEHA NARKHEDE: That’s right.

KEVIN SCOTT: Like – and whose mindset about open source and like how to think about like actually doing software development with that open-source ethos was aligned with my own.


KEVIN SCOTT: And that’s how I chose – one of the big, big factors in me choosing to go to LinkedIn.

NEHA NARKHEDE: That is amazing. So, you might have joined around the same time I joined.

KEVIN SCOTT: Yeah, I was –

NEHA NARKHEDE: Or before that.

KEVIN SCOTT: I started in the – February of 2011.

NEHA NARKHEDE: Got it, yeah. I started end of 2009 –

KEVIN SCOTT: So, you were –


NEHA NARKHEDE: Maybe a year, yes.

KEVIN SCOTT: You substantially preceded me. (Laughter.) But yeah, it’s sort of fascinating. So, like, tell me a little bit about what it was like working in that group, because one of the things that Igor always encouraged folks to do, and it sort of set part of like what is now a core value in LinkedIn engineering is like it’s open source first.


KEVIN SCOTT: Like, you actually have to have an excuse to say no to open sourcing something. So, what was that like as an engineer?

NEHA NARKHEDE: It was fascinating. So, Igor was super supportive, very technical, and I was just basically extremely fortunate to land in his group. So, I started on the search team, because I had a search background and I thought, you know, why not? And you know, when I joined on the search team what I was doing more of was ETL and data pipeline stuff.


NEHA NARKHEDE: And how to get data into all the search systems. And I was doing less of, you know, deep distributed systems challenges, which is what I thought I would do on the team. And so that’s when I realized that our big problem as a company back then was actually data pipelines, right? And that was the one thing that was at least slowing or stopping us from building these data-driven products at scale.

So, I started digging around, you know, I started talking to Chris Riccomini who was on the analytics side. I talked to Jay, who was thinking about, you know, Hadoop adoption at LinkedIn. And I asked him, like, hey, Jay, like, what is – what is the biggest challenge you think we need to solve?

And he walked me through this data pipeline problem that he was thinking through and he said that, actually, you know, the main stuff is this Hadoop stuff and the analytics thing and then the side thing is this data pipeline thing. And I’m just tinkering around, but no one really wants to work on it. So, I’m just going to slate it for completion, but no one really wants to work on it.

And my experience was the opposite, is that that’s all that was important to search and analytics. And so, I said, you know, how about I join on that side of the team? And so, I talked to Igor and I said, “Igor, I’ve been doing this and I think this is the biggest problem. And no one is working on such an important problem for the business. And even though it’s sort of dirty work, it seems like I want to do that.” Right?

And he was very supportive. So, I thought like a bunch about talking to Igor, like I thought for maybe an hour and what I’m going to say and like why I’m going to justify this. Because I’m, like, a new engineer. I’m only six months in at that time. So, I absolutely haven’t totally proven myself out. That’s what’s going on in my head.

But I somehow conjured up the courage to go talk to him, and he made it super easy for me. He’s, like, “Yeah, you know, let’s talk to John when he’s back.” John was the tech lead for the search side, and let’s get it going.

And that part about LinkedIn’s culture really amazed me and that was super inspiring to be part of, you know, because you don’t have to go through a lot of bureaucracy, right? I could just talk to Chris and Jay on one side and Igor on the other side and John and, boom, like in two or three weeks, I’m working on Kafka.

KEVIN SCOTT: Yeah. And so, for the audience, who are not all engineers and not all necessarily Silicon Valley folks, like, what is Kafka?

NEHA NARKHEDE: Kafka started off as a high-throughput distributed, super-scalable messaging system. And has transformed into what’s called an event streaming platform. So I’m going to take some time to talk about what an event streaming platform is, but if you think about the database and you sort of unbundle it, you take the transaction log out and you take the thing that creates the tables and the query processing out, and if you apply it to the scope of an entire company and all its software, then that’s basically what Kafka is today. And we’re calling that as an event streaming platform.

It has the logs, it has stream processing, which is the ability to create tables on that log, and it has the ability to connect to all the different systems because of that abstraction.

KEVIN SCOTT: And so the – you sort of described one of the first obvious things that Kafka got used for is so – in a modern, large-scale software business, you have lots and lots of things that are producing lots and lots of data.

And some of the data is log data, and like log data basically is a record of like here’s something that happens –


KEVIN SCOTT: – that you may want to pay attention to at some point. Or you may not.


KEVIN SCOTT: Or it could be, you know, sort of events – you know, like some sort of transactional data that is being written into the system, so like this is the database log.


KEVIN SCOTT: For instance. So like here’s the thing, and like you know we – it’s a sale that we made or a click that happened or something, and like we need to update some other state based on this thing that happened, like, somewhere else.

And like the – you know, what – you described working on ETL, and so ETL is this process where you take a bunch of the stuff that is flowing out of all of these systems and you gather it all up and you sort of say, “Oh, I’m going to add up all of the clicks that happened on this particular ad.”

And I’m going to like store that in a database somewhere that you know sort of counts all the clicks so that I can pay the – you know, the advertiser – or like charge the advertiser for –

NEHA NARKHEDE: Pretty much.

KEVIN SCOTT: Yeah, and it’s like a gazillion different things, right?


KEVIN SCOTT: So, you need ETL to work so that you can basically reason over what’s happened in your system. And so like that’s sort of where Kafka started. It was – like it turns out that like getting all of this data from all of these places where it’s being produced and like getting it to all of the things that need to consume it –

NEHA NARKHEDE: Consume it.

KEVIN SCOTT: – like ETL systems is actually a pretty hard technical problem.

NEHA NARKHEDE: It turns out, it seemed very simple when Jay drew it on the board, which is what got me excited about it. But as I started to look into it, it was pretty complex. You know, back then, LinkedIn was going through its own micro services transition, right? So we were creating a lot of tiny applications. And then we had all these analytical systems and data systems that needed access to the data.

And if you think about, you know, what the source of truth was, there actually wasn’t one. You know, the applications sort of were talking through an MQ and because MQs didn’t scale, not a lot of the micro services were talking to the MQs, and we had all this custom tooling and data warehouse-centric tooling that was copying all that data and scraping it at the end of the day.

And so the motivation for solving this problem was just like a simple observation is if you think about LinkedIn, we produce data 24 hours a day, you know, by processes and applications that never stop, that users that were global, never stop.

But if you think about all the infrastructure that we had to harness that data and put it to use, it was sort of stuck in this bi-modal world where you (inaudible) simple lookups on a relational or NoSQL database, or you could wait for the end of day, where you can get like these slow, big, batch dumps and query them.

And if you think about a global business, what is even end of day? And at the same time, our use of data was changing. It wasn’t just the reports that our employees and our exec team needed to look at, it was the problem that you mentioned, which is all this data that was being generated on the side, we wanted a way to take it and supply it to all the software that basically ran the business.

You know, every user action we wanted to make a decision on, every user experience, we wanted to customize instantly, not a couple times a week, which is what we were doing.

We started by just looking at existing systems, if you remember. It was trying out MQs and looking into why they didn’t scale. We invited folks from Scribe, you know, all these different log – logging systems, and trying to see if we could turn that into a real-time thing. And what we realized is it’s a fundamental architectural flaw at the heart of the system.

KEVIN SCOTT: And one of the really interesting bits of genius I think about Kafka, like one of the things you all did especially well is – and I think the reason that it’s caught on so broadly and gets used for so many things is that you tried really hard to make as few assumptions as humanly possible about what was going to be producing data and what was going to be consuming it. Because a lot of the failure of these other systems like Active MQ and like we had this thing inside the company called Data Bus –


KEVIN SCOTT: – is that they made a bunch of assumptions about what it is that was producing the data and like all sorts of things like event ordering and –

NEHA NARKHEDE: Absolutely. The GMS API –

KEVIN SCOTT: The more constraints that you added, like, the less utility you could get out of the messaging system as a whole, and it sort of restricted how it could be used and it made it cumbersome to reason about and to scale and to manage.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: And like Kafka is really this brilliant piece of design because it’s very, very easy to just take something that makes data and like write it into Kafka and then it’s there for – the producer doesn’t have to think anything at all about the –


KEVIN SCOTT: – about the consumers.


KEVIN SCOTT: So, anybody can consume the data. I mean, at LinkedIn now, like, there are thousands of unique things that produce data – write it into Kafka, and there are thousands of unique things that consume data.

NEHA NARKHEDE: Consume. Yeah.

KEVIN SCOTT: And like they don’t have to know anything –


KEVIN SCOTT: – about each other.

NEHA NARKHEDE: Yeah. And that was the heart of the, you know, design thinking that went in is, A, it needed to be, you know, it needed to serve decoupling, right? So, decoupling of the schemas from the producers and consumers, decoupling of producers from consumers, and the third thing, which is why we really created and made it a log is this realization that if we wanted to serve this use case where the whole company is going to need to publish and the whole companies need to subscribe to it, is it needed to be a multi-subscriber retentive model for messaging.


NEHA NARKHEDE: And that was really the heart of that change is simple APIs, retention, leave it to the user to decide, you know, how to configure that system appropriately, and then made it super simple and scalable.

KEVIN SCOTT: Yeah, I mean, I think another one of the genius design things about Kafka and like one of the things that you all at Confluent are talking a lot about right now is that thinking about systems as like data producers and data consumers with like a messaging and like stream processing mechanism connecting them, really is like a way to sort of disaggregate data systems.

NEHA NARKHEDE: Pretty much.

KEVIN SCOTT: So, like, you know, intellectually, you could solve a whole bunch of these data transport problems by just writing all the data to a database. And like you write them in with SQL, you read them out with SQL. But the problem with that is then you’ve got this sort of monolithic database thing and –

NEHA NARKHEDE: It’s couple tightly.

KEVIN SCOTT: It’s super complicated, and like you just – like all the different pieces of the database system scale differently. And so like pulling the message and stream processing substrate out lets you sort of scale the system in a natural way. It’s almost like how you broke up search engines into like different components and it’s like how we –

NEHA NARKHEDE: Absolutely. It is the – that materialized view is not just in the form of a relational table, it could be in the form of a search index, it could be in the form of a graph engine, it could be in the form of a simple log or Hadoop file or something.

It was that sort of, you know, decoupling to a point where you can actually just create these streaming, materialized views in whatever fashion and downstream system that you could conceivably think about.

KEVIN SCOTT: Yeah. And it also sort of separates out the failure domains of this thing, so it’s easier to operate. So, like, you have all of these independent systems and like when one fails, it doesn’t necessarily mean that everything fails –

NEHA NARKHEDE: Cascade it to the other.

KEVIN SCOTT: – and it’s like easier to debug when like the things that are failing independently of one another are like simpler themselves.

NEHA NARKHEDE: Absolutely. And it’s this form – you know, it’s this idea of accounting applied to software systems where if only you could remember all the events that happened rather than just the right answer at a point in time, you could actually troubleshoot much better.

So, if you remember, the way we built Kafka and the way applications used it is you just sort of like – you know, a log that keeps track of all the events. So if you make a mistake in today’s deployment and you realize it two days later, you can actually go back and rewind and replay.

And I think the rewind and replay was really the favorite feature, you know, that led to Kafka’s broad adoption is, oh, wait, I could actually make a mistake and actually go back and it would allow me to go back to a point in time and offset in the log and replay the source of truth of what happened, not my interpretation of what happened, but actually what happened.

KEVIN SCOTT: So, you all built Kafka. Its initial uses were inside of SNA, which is great because like that’s where –

NEHA NARKHEDE: Yeah, we had to prove it out. (Laughter.)

KEVIN SCOTT: Yeah. That’s where like you know some of the most complicated distributed systems were in the company, it worked brilliantly. And so, at some point, like, we started using it for everything inside of the company, like the site reliability engineers, like, built all the monitoring.

NEHA NARKHEDE: Monitoring.

KEVIN SCOTT: And alerting stuff on top of it. It’s just basically everywhere.

NEHA NARKHEDE: Everything.

KEVIN SCOTT: And we decided to open source it. So, like, talk about, like, why did you all want to open source it? And like what did you get from open sourcing it?

NEHA NARKHEDE: Wow. So, we – you know, why we wanted to open source it is this was an observation we made is, you know, this trend that you know technology companies are going to essentially be defined in software, you know, not just use more software, but literally all the actions and decisions in a company happen through software. It’s going to be a trend. It likely is not limited to LinkedIn.

LinkedIn is not the only data driven company. It’s that every company in the world is going to become a data-driven company. And so, we wanted to sort of get validation for that trend and decided to open source Kafka as a result of that.

And I think LinkedIn made it, you know, very easy to open source Kafka. We were in the group that did that, but we did one more thing, which is we donated it to the Apache Software Foundation, and that was uncommon for LinkedIn is to give up the control of the software that was being built and give it to an independent entity. I think that really led to its broad adoption, so –

KEVIN SCOTT: Yeah. Although, that wasn’t a hard decision. Like, Igor – you all ask Igor, Igor said, “I think it’s reasonable.” Igor asked me, and I’m like, “It’s fine.”


KEVIN SCOTT: And it was done. (Laughter.)

NEHA NARKHEDE: And it was done, right? And in 2011 is when it was donated, I think –


NEHA NARKHEDE: – to the Apache Software Foundation. So, it was very, very simple to do that. And I think that was the turning point for the community to get built around Kafka because they sort of saw Kafka as not just a LinkedIn thing as like an independent project that could develop a whole open source community around it.

I think that did wonders to Kafka’s adoption in the world is it reached in, you know, beyond Silicon Valley in about two years since we open sourced it to a point where I was on a call with June, our other – my other co-founder – and helping this big Fortune 500 just for free about their Kafka problems.

And as I sat there, I realized that, wow, you know, Kafka has gone mainstream. And in order to really fully realize the next level of Kafka’s adoption and potential there will be a company around it. And I just sort of thought that, hey, if there is a company around Kafka and it’s not the three of us, that would be a shame. (Laughter.)

So, I pushed that idea to both of my then teammate, now co-founders, and it turns out that they were also interested. So that’s when it – you know, we sort of began this, you know, brainstorming sessions to start Confluent.


NEHA NARKHEDE: This was five years ago.

KEVIN SCOTT: Yeah. So, even though you had this traction with Kafka, it’s still a big decision to go start a company.

NEHA NARKHEDE: Yes. (Laughter.)

KEVIN SCOTT: In fact, like, I was in three companies in a row that were, you know, sort of private and like ramping up and it’s an intense, intense experience.


KEVIN SCOTT: So how did you all get the courage to do this?

NEHA NARKHEDE: You know, I don’t know – I can’t speak about my co-founders, but for – but what ran through my mind when I decided to take the leap, and I’m so thankful that they joined me in that endeavor, because it’s going to be much harder as a single co-founder to do this, is the same thing that ran through my head when I decided to take the leap to LinkedIn.

Is, basically, I think FOMO, you know? There was just this fear of missing out that, hey, there’s a company around the thing we created, that’s just not going to work for me.


NEHA NARKHEDE: You know, it just – we had to do it. That was one. And the other thing was it was very clear that it had turned into a phenomenon, right? And it was clear that it needed to have a first-class, you know, place in every company’s data architecture. And that sort of investment, it really requires like a business vehicle around it, that was really the second thing that went through my head is, hey, it’s like – just practically speaking, we’re going to need to devote all our time, you know, not just partially, but think about all the companies, you know, beyond LinkedIn that could use this, right?


NEHA NARKHEDE: So, I think those two things ran through my head is fear of missing out and impact. (Laughter.)

KEVIN SCOTT: Yeah, and I – look, I think you’ve said two things that are like – people should just sort of pay close attention to, because it’s phenomenally good advice. So, there’s this fear of missing out or regret minimization.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: Like, minimize the number of regrets –


KEVIN SCOTT: – that you’re going to have over a period of time. So, like, that is really good advice. And the other thing is, like, you know, put yourself in situations where you can maximize the amount of stuff that you’re learning.


KEVIN SCOTT: Like, if you do those two things, like, you can accomplish a lot –

NEHA NARKHEDE: A lot of things. You can accomplish a lot of things. And at the same time, you are uncomfortable almost most of the time, I would want to say. (Laughter.)

KEVIN SCOTT: Yeah, actually, I find the discomfort a sign that you’re doing stuff right.


KEVIN SCOTT: Like, a lot of people try to avoid discomfort.

NEHA NARKHEDE: Discomfort.

KEVIN SCOTT: And like of course there are flavors of discomfort that you should avoid.


KEVIN SCOTT: But the discomfort that comes from trying something really, really hard and not actually being perfect at it is good.

NEHA NARKHEDE: Yes, it is.

KEVIN SCOTT: Because that is the thing that sort of drives progress and that builds skill and that like actually makes change happen.

NEHA NARKHEDE: Absolutely. I think like this comfort – I realized that through my master’s even is just there was a ton of discomfort in coming to a different country and learning about computer science from first principles, very different style of teaching.

And that’s when I realized that I’m just going to have to be comfortable with that discomfort if I wanted to learn things, which is what I really wanted to do.


NEHA NARKHEDE: And so I think that sort of training almost helps a lot in starting a company because you’re just going through one obstacle after the other after the other and it just literally never ends, right?

KEVIN SCOTT: Yeah. And so, like let’s talk about Confluent. So like, I still remember the day that Jay walked into my office and –

NEHA NARKHEDE: To tell you that we’re leaving? (Laughter.)

KEVIN SCOTT: – he had this – I think he thought he was going to get a different reaction out of me than he did. He came in –

NEHA NARKHEDE: Pretty much.

KEVIN SCOTT: – like almost worried it’s like, okay, I’m going to – I’m delivering bad news. And I told him, I’m like, oh, this is actually pretty cool.

NEHA NARKHEDE: Yeah, that was the other unique thing about LinkedIn’s culture, we were out in a month with no drama, a lot of support from everyone –

KEVIN SCOTT: Yeah, and we invested in the company.

NEHA NARKHEDE: – from you, from Jeff, from Reid. And you invested in the company. And that is unheard of, you know. There were a lot of – there were a couple of other stories that did not look like that back then.

KEVIN SCOTT: Yeah, but what you were doing made sense. Like, we were never going to – like LinkedIn is a social network. It’s very important when you have a company that you have – you have a mission and vision and you focus on the mission and vision. And like LinkedIn’s mission and vision wasn’t building distributed streaming infrastructure –


KEVIN SCOTT: – for the rest of the world. And so like I was completely supportive of you all going out and like finding your own mission and vision that, you know, took this thing that you helped build at LinkedIn and like now you’re going to make it useful for other folks, great.

NEHA NARKHEDE: I still get asked that question is how – exactly how did that play out internally? I’m like, it was very simple. And that’s just – it speaks volumes about LinkedIn’s culture, I think.

KEVIN SCOTT: Yeah. And like the thing that I wanted to make sure that we had and like I think by and large has worked really well is that there’s still a team of people at LinkedIn – like a pretty big team of people.

NEHA NARKHEDE: Pretty big team, now, 40 people I think.

KEVIN SCOTT: Who work on Kafka because Kafka is a very important piece of infrastructure for LinkedIn. I just wanted to make sure that like you guys and they were going to be able to –

NEHA NARKHEDE: Collaborate.

KEVIN SCOTT: – continue to collaborate with one another.


KEVIN SCOTT: And that’s worked pretty well I think.

NEHA NARKHEDE: That’s worked pretty well.

KEVIN SCOTT: I mean, it’s not always perfect, but it’s like worked –


NEHA NARKHEDE: Yes. (Laughter.)

KEVIN SCOTT: It’s worked really well.

NEHA NARKHEDE: It has. It has.

KEVIN SCOTT: So, how has it been? You co-founded a company, like, you’re start – there’s nothing. Like, you’ve got to figure out the business model, you’ve got to figure out like how are we going to build software? How am I going to hire engineers? How do we, you know, like, where’s the office going to be?

NEHA NARKHEDE: Yes. (Laughter.)

KEVIN SCOTT: Like, so talk about this, like, what were the big challenges in the first days?

NEHA NARKHEDE: I think the big challenges were the – there was a time when we spent before even we started the company on what the business model was going to be. And the big question is, you know, do you want to have a software business model? We’re going to sell subscription on software packages? Or are you going to have a fully managed service?

And back then, five years ago, it actually was not that clear. And it wasn’t as simple of a decision as it is now for data systems. You just start with a fully managed service, that’s it.

And so we thought a lot about that and realized that Kafka is a data integration system, after all. It needs to be where most of the data is, and we were told that most of the money is in the enterprise and enterprise’s data is still on premises.

And so that one was – it turned out to be a very important decision that we made is to start with a software package so we could address the – you know, the needs of all these enterprises, and then in time – and this is something we did I think well – is in time also transition to the fully managed service part as they’re moving to the cloud.

But in the early days, hiring engineers was the focus. And I want to say that it was a tad bit easier for us than it is for startups that just start with nothing is our big pitch was Kafka is everywhere. And so, do you want to come work on the thing that’s everywhere and help us build, you know, the best product around it?

And so that was – you know, those were the two sort of pivotal things that we did in I think year one, and that’s how the early customers came. And what I was surprised about is that is how the latter customers also came and that is how the cohort after that also came is just it’s this crazy inbound vehicle which was another thing that – it just doesn’t work like that in enterprise companies. You have to build demand for the product that you’re producing enough so you can commercialize it. And for us, it was sort of the opposite. And that was a great experience and it still is.

KEVIN SCOTT: Well, you know, look, this is – I don’t think every software design project is going to turn into a business quite as successful as Confluent. But, like, everyone should be thinking about this, like, part of the reason I think that you all are as successful that you have this crazy inbound is because of like honestly software architecture decisions that got made years –


KEVIN SCOTT: – and years and years ago.


KEVIN SCOTT: Like, Kafka’s just easy to use. Like, it’s no more complicated than it needs to be. And like that is a really fantastic thing.

NEHA NARKHEDE: Pretty much.

KEVIN SCOTT: You know, in a way, it’s like MapReduce, right? It sort of abstracts away a bunch of like super complicated like hard distributed systems stuff and like lets you just sort of pay attention to like here’s my data, here’s my applications, like, it solves real problems.

NEHA NARKHEDE: And it just scales. You throw more data at it, a couple more servers, and it just works. And that sort of – it just scales and works was sort of the, you know, real magic behind that adoption. And we didn’t realize this years ago when that adoption was happening is how important it would be to make the company building part easier, not easy. (Laughter.) But it sort of turned into a big machine for us to then harvest and commercialize.

KEVIN SCOTT: So, there have obviously been a bunch of technical things that you all had to go sort out but like what are some of the non-technical things that have been interesting challenges for the company?

NEHA NARKHEDE: So the non-technical things are, you know, the first lesson we had to learn is the role of marketing in forming Confluent and Confluent’s demand-generation engine is just sort of – we had to go through our own learning experience of like just what – the kind of DNA we need and what was its role in the organization and how technical it needed to be because our end user persona is still the engineer, the person who first makes the decision to adopt it.

And that was a pretty big learning sort of you know moment for us is this whole inbound motion, it takes first-class investment in – luckily we were good at, which is evangelism and path leadership is, believe it or not, that is really the investment that actually works with these distributed data systems. No matter how big you get, is that’s the way you generate demand for your system.

And that was a whole different way to do enterprise marketing than what you imagine you know enterprise companies need to do. That was sort of one. And then the other one was the role of sales in enterprise companies, which is you know, imagine sort of sales have to go through this big outbound qualification and calling to companies.

And for us, it was sort of the reverse, which is how do we best harvest all this demand that existed around Kafka and convert it into the best possible vehicle for Confluent and for the company? Those were the two sort of big go-to-market learning lessons for us and it’s sort of unique about – I want to say open-source-based enterprise companies, not so much just about any enterprise company, which just looks very, very different from Confluent.

And then the role of product in particular was another learning lesson. It’s a very engineering-driven product, and a very engineering-centric adoption cycle. And so, the role of what product management should do two or three years in, we hadn’t figured it out. And it was so hard and we tripped and fell, so I decided to actually move from engineering and technology into just product is product management and building out the right product team and doing product marketing. And that whole transformation alone was hard, but deeply satisfying.

KEVIN SCOTT: Yeah, and what were some of the challenges there? So, I can – I can imagine what some of them might have been, but I’d love to hear how you made the transition.

NEHA NARKHEDE: So, the first challenge is just learning for me was just learning about product management from first principles. So, I followed my playbook, went and read every book there was on product management, and recruited a couple of mentors. And that’s how I quickly sort of got myself up to speed.

What was easy is I was running and I had built the engineering team by then, so what I understood is what engineers needed from product managers, right? So, we needed to retain the engineering-centric culture and supplement and complement it with product to actually help us prioritize and get the customer feedback in. Because we were not – you know, if you just build for what you think is right technically, you’re not really going to end up with the right product that your customers are going to buy.

And so that was one. The other one is the DNA of product managers that we needed, like, uniquely Confluent needed. And what I realized is, you know, if you go after the enterprise product managers, they’re actually not going to succeed because they come in with a certain way of doing things and with not much of a foundation in technology and systems.

And so the second thing I did was make sure we hired product managers who were engineers just by training, who had done computer science, who had done some software programming, and had then learned product management so it was a much easier transition. I think that was the second.

And the other thing was how did these teams work together? And that was really, really important to establish is we are not going to come up with plans in a room full of product managers, that’s just like not how planning happens at Confluent, is like every step of the way, engineers are involved no matter what we do. Even if it’s prioritization, even if it’s like how we’re selling, they’re there.

KEVIN SCOTT: Yeah, and I – look, I think it’s really good to hear you say that you were thinking so much about culture, because I think culture effectively, turns into the playbook for collaboration inside of organizations. And it’s just at some point when the scale of what it is that you’re trying to accomplish gets big enough that you can’t – you can’t sort of fake good culture.

NEHA NARKHEDE: Pretty much, yeah. (Laughter.) It happens in your first 100 people I think.

KEVIN SCOTT: Yeah, I mean, I think it – you know, as soon as you get past Dunbar’s number, it becomes like existentially important. Like, if you don’t have a good culture, if you don’t have a culture that encourages the types of collaboration that you’re going to need to build the things that you need to build, you are screwed.

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: So that’s awesome to hear that like that’s how – I mean, and this, again, I think is another lesson for deep technology people – like at some point, like, this is just sort of the necessary work that you have to go do, even though it’s not like figuring out the locking algorithm on this leader election thing that you’re trying to do.


KEVIN SCOTT: It’s, you know, at certain points in time in the development of the company, like, getting that culture right and like figuring out like how you’re going to get people to like play the right roles and to like collaborate effectively with people in other roles is like –

NEHA NARKHEDE: Absolutely.

KEVIN SCOTT: – the most important thing.

NEHA NARKHEDE: Culture is like the shared set of values, right, that you want your people to sort of stand on top of and respect. And I just sort of think of it as you know the values that we appreciate that you know gets you sort of – that just allows you to progress in an organization and the values that don’t work, right, that just will not work in an organization.

And those sort of boundaries is the way to think about culture and just like the way to scale a company is by getting it right in the first 100.

KEVIN SCOTT: Cool. Well, so, I want to shift away from tech and business for a second. Let’s talk about you, like, what do you – what do you do for fun? Like, you’ve got this super high-pressure job, like, the company’s growing like crazy, like, it’s – it’s, I’m guessing, if your experience is anything like mine, it is probably intense, so –

NEHA NARKHEDE: It is very intense.

KEVIN SCOTT: So how do you have any sort of balance in your life?

NEHA NARKHEDE: I strike a balance, and my favorite activity to strike a balance with is to travel to new countries and experience new cultures and that’s what me and my husband do, and our crazy hobby together is to go SCUBA diving.


NEHA NARKHEDE: And some of the, you know, crazy locations. Oftentimes, to see different varieties of sharks if I can’t – I can’t – I’ve been in one of those cages where great whites are on the outside and you’re on the inside, I can’t confirm if I was scared or not. (Laughter.) But I would say I survived it, and it turned out to be fun in a very weird kind of way.

KEVIN SCOTT: Yeah, you might be a little bit braver than I am. (Laughter.) And you mentioned before we started recording that you’ve got a trip coming up where you’re going to do some high-altitude stuff in the Himalayas?

NEHA NARKHEDE: Absolutely. I’m going back to India for a couple weeks, mostly for work, but we’re going to go up to the Himalayas – hopefully to get a very good view of the Everest from (inaudible).

KEVIN SCOTT: Awesome, well, definitely – definitely take some pictures. I – and be safe.

NEHA NARKHEDE: Thank you. Absolutely.

KEVIN SCOTT: Awesome, well, thank you – thank you so much for spending some time with us today, this was fantastic.

NEHA NARKHEDE: Thank you for having me, Kevin. It was great.

KEVIN SCOTT: Awesome, all right.


CHRISTINA WARREN: Well, we hope you enjoyed Kevin’s interview with Neha Narkhede. That was such a great conversation. Like you said, she kind of has that archetypical Silicon Valley story. I thought it was really interesting how she read biographies of famous women and seemed almost to kind of prepare herself for what she wanted. She’s had this vision in her mind for a long time.

KEVIN SCOTT: Yeah, I think it’s one of the really consistent things that we see with some of the folks who’ve accomplished big things that we’ve chatted with on the podcast is that at some point, you actually do have to have a vision of where you want to go. It doesn’t necessarily need to be like a super clear vision where you’ve got every last little, you know, waypoint on the journey that you’re about to take mapped out, but like you do have to know approximately where it is that you want to go.

And like one of the really impressive things about Neha is like I think she over and over and over again throughout the course of her career, just like we heard in this conversation, has had just this very clear notion of where it is that she wanted to go.

CHRISTINA WARREN: Yeah. I love that, I love that vision and that drive. What I also think was really interesting is, you know, the story of Kafka, you know, she works on this and this story of how it goes from being this internal tool to being something that’s contributed to the community, that’s open source, that used all over the place, but then it’s also, you know, she’s able to kind of, you know, spin it out and make her own company based on that. That’s in some ways kind of the archetypical modern Silicon Valley story when it comes to open source.

KEVIN SCOTT: Yeah, it is, and, you know, it’s one of those things where I’m really glad that we – that we open sourced the technology. Like this is the way open source really should actually work. Like LinkedIn and now Microsoft at a pretty large scale are investing heavily and contributing back to the – to the open source community. Like we consume a lot of open source software, so –


KEVIN SCOTT: – and so like, you know, it’s the right karmic balance to contribute back. And there’s more than one way to contribute. Like one way is like obviously sort of the way that most contributions happen, which is like you sort of open source a project and you do your development work, partially or completely out in the open, which has a huge number of benefits, some of which we chatted about in this conversation with Neha.

But like another way to contribute that is like I think, you know, as you pointed out that’s really powerful is like allowing companies to form around these open source ecosystems that are getting a lot of traction.

And, you know, if you look at Kafka’s traction inside of LinkedIn, it is staggering. Like there’s a pre-Kafka world for LinkedIn infrastructure and then there is a post-Kafka world for LinkedIn infrastructure, and like it is starkly different.

Kafka in a sense lets you sort of think about how you build infrastructure and how you build infrastructure teams and their culture in a different way. It sort of decouples everything and just sort of like lets a lot of creativity happen, because it removes like one of these control points that exist in many sorts of infrastructure stacks.

You know, this separation of concerns that sort of let people who have data publish it without having to imagine every possible use and to allow folks who need the data to sort of consume it for the application or use case that they’re building like is a super powerful notion. And so, it’s awesome to me to sort of see more people benefitting from that, like more companies, and like I’m glad that Confluent exists to help make that happen.

CHRISTINA WARREN: No, definitely. I love – like you said, it’s one thing, A) it’s amazing when these internal tools are open sourced, but then when it goes to the next level and you are able to see companies built off of that technology and are able to make real businesses for themselves, that’s kind of the goal, right, because the – for many years that was sort of the unspoken rule about open source is, oh, there’s not – there’s not a way to make money, and you had a few examples, you know, the Red Hats of the world would kind of prove that wrong, but it was less common.

And now we’re seeing I think in this – this kind of I guess maybe third wave of open source that I would kind of include, you know, like the Kafkas of the world, the Reacts of the world, things like that, Node certainly where you can start to see these real businesses form around the technology.

KEVIN SCOTT: Yeah, and I think it’s really – it’s really exciting. Because sometimes open source is hard, sometimes it – you can have a really, really great idea, and you can even have a really involved community, and you still need a little bit more than that to like help tip everything over into like more widespread adoption.


KEVIN SCOTT: And so like these companies are fulfilling a very, very important role in getting more technology in the hands of more people, which in my opinion, and I’m biased, is like a really good thing.

CHRISTINA WARREN: I’m biased, too. I – that’s one of the things I advocate for in my day job. So, I’m glad to see more of this.

All right, well, we are at the end of our show today. Thanks to Neha. And last show, we asked you to let us know what technologies you think are inspiring kids in the next generation.

KEVIN SCOTT: And survey says?

CHRISTINA WARREN: Well, we got some really interesting responses. The most prevalent one centered around augmented reality, which is kind of not a surprise, and that is due to all the gaming with Fortnite and Minecraft.

KEVIN SCOTT: Awesome. We shall see.

CHRISTINA WARREN: I sure – I certainly hope so. And as always, we would love to hear from you about ideas for future guests or anything on your mind. So, you can e-mail us at [email protected]. And be sure to tell all of your friends and colleagues and fellow Beyoncé fans about Behind the Tech so that you can geek out together. Thank you for listening.

KEVIN SCOTT: Indeed, thanks, and see you next time.