High-performance Hiring & Interviewing (fmr. Amazon GM, Prime Membership, Anna Collins)

Ben: Hello Acquired LPs and welcome to another episode of The LP Show. David and I are looking at each other across a wonderful squad cast, David in his San Francisco home and me up in Seattle, staying safe and staying healthy. I think as Governor Inslee’s message up here, we hope all of you are safe and healthy as well.

Today's episode is one that has been requested a few times in the Slack and a topic of great discussion. When you dive deeper into these company-building topics and stop talking about just the news of the day of who got funded, what companies going through trouble, or the day-to-day of running a company, one of the most important things is how do you choose who to bring into your company, and what process do you put them through to see if they're the right person. We have never discussed interviewing on The LP Show.

As David and I were talking about how we should do an LP episode on interviewing, I was telling David about this story of this awesome entrepreneur in residence, Anna Collins, who's been working with us at PSL, and she hopped on this interview loop to help us out. Just had in our post-interview discussion, the most thoughtful feedback, the best questions, all of us were joking, looking at each other, and saying why did any of us even do this interview loop, and can Anna just do the whole thing for us? 

Here we are today to do an LP episode on the very first time on how to conduct the best interview you can, a key part of building a high-performance team with the best person I can imagine to help us do it. Anna, thank you for being with us today.

Anna: Thanks, Ben. It's great to be here.

Ben: To give folks a sense of Anna's background. I mentioned I personally know her because she's been working on new startup ideas with us. Before that, Anna had a prolific career as a leader of larger organizations and starting things from scratch. She is a fantastic leader and human, having started and scaled new businesses, multi-billion dollar companies in healthcare, EdTech, advertising, gaming, CPG, retail, and of course technology businesses.

She was the worldwide general manager of Amazon Prime. Before that, she was recruited by Microsoft to build out and scale the global search advertising business, from concept all the way through growth, and ultimately, to $1.6 billion in revenue. At CVS Health, she was responsible for leading the internet channel strategy and the acquisition, culminating in the launch of cvs.com.

She holds an AB in economics and an MBA from Harvard University. While not changing the world, Anna coaches her sons, Henry and Cooper, in basketball and keeps up with her wonderful wife, Debbie. A quote that closes out, Anna’s bio reads, “Family first, love wins.” I just wanted to include that because I saw it in a bio here somewhere, and thought it'd be a great way to close that out.

Anna: Thanks, Ben. I appreciate it.

Ben: You have such a prolific background. Why don't we dive into some of the early history of that before we get to this nitty-gritty of tactical interviewing questions? One of your earliest experiences as a three-sport athlete was on the Harvard basketball team. I wanted you to share it with us because I know a little bit about the story. What that was like for you and why that was a special time in your life?

Anna: It was a special time because when I went to Harvard as a freshman and joined, we were in last place in the Ivy League. In the first year, we won 3 games and lost 21. We were losing basically all the time. We weren't just losing, we were getting crushed by the teams, 20 points, 50 points. It was the first year of a new coach that came, Kathy Delaney Smith. She had a vision of winning and that was part of why I wanted to be on that team.

Turning the program around, we went from worst to first in four years. In my senior year, we won the Ivy League and reversed the freshman year record. We were 21:3 and it was the first time Harvard basketball ever won an Ivy League Championship, men or women's. That was a big journey going from worst to first. A lot of heartache, sorrow, and a lot of learning, how to experiment, be a team together, and it was really fortunate to be part of that period of Harvard basketball history.

Ben: I have to imagine going from worst to first those early years of your life on something like that. You had to be a pretty competitive spirit, had to be this “win at all cost” mentality and learn to trust people that you didn't know before. Has that always been a part of your personality or did you develop that as part of that experience?

Anna: I think I came out of the womb as a competitive spirit, so that's been part of competing in sports. Actually in music, I played the flute in a symphony, and also did state competitions during that, so I loved competing in music as well as different sports.

I think that competitive spirit showed up in different places, and I found different arenas to do it, but one thing I would say the other piece is I always pick team sports. In high school, I played volleyball, softball, and basketball. In college, I played basketball, volleyball, and lacrosse. I played team handball at the Olympic festival at the national team level.

I could go on about that, but it's always team sports. People were like, “Oh, you should have been a swimmer,” and I'm like, “Well, I can swim, but I never wanted to do the individual sports.” During that journey in basketball, I had a practice that if we did not win a game, I did not look at my individual stats, and that's because it didn't matter what I did. It didn't matter how many rebounds or how many points. If we didn't win the game, it didn't matter, so I had a practice of only looking at my stats if we won the game. That gives you a notion of how I feel about team versus individual, and what matters.

Ben: Speaking of team versus individual, I know you were in the Air Force. I'm curious to hear a little bit about your path of being in the Air Force, and how that ended up leading you into the world of technology.

Anna: I could share a little bit about the Air Force. Basketball had more to do with the lead into technology with our friend Mike Galligan. The Air Force, I had the opportunity. I was picked to go to an engineering and science seminar at the Naval Academy based on escorts. When I was a junior in high school, I got exposure to the opportunity of the service academies, as well as then the ROTC (Reserve Officers' Training Corps) programs and scholarships. I had a choice when I was graduating, I had both scholarships and appointments to the Naval Academy and the Air Force Academy, and you can take those scholarships anywhere.

I also had other options with the academic and athletic scholarship, so super fortunate, but I wanted the experience of serving. Serving mattered to me. Growing up in Michigan, I had that service component and value as a family. Then second, the opportunity to lead, to manage people, and be responsible for resources and people at an early age. Really at 21, you're thrown in as an officer and you get that opportunity. I knew that I wanted that. Those components altogether had me really go after the ROTC scholarship and I did that at MIT, cross-enrolled at MIT as I went to Harvard.

Ben: At some point, you made this choice, “Hey, I'm going to go into the world of business. I don't think the Air Force has been this amazing experience, but that isn't my career.”

Anna: In college, I studied economics, and that was the closest thing to business at Harvard. I thought I'm going to go to business school or law school. I'm not sure, but the Air Force. I didn't know how long I would stay but as I was in the air force, I figured out I was gay in my early- to mid-20s, and that was when it was before Don't Ask, Don't Tell. It's just your criminal if you're gay.

I was in finance. That's where the Air Force matched me as a career, which ended up being a good career for me and background. I wanted to get out. When I figured that out after my four years of active service and I had an inactive time that I reserved that I owed, but I didn't feel comfortable staying in. For me it was integrity to not be my full self, so I was just scared and I got out after that first four years. Otherwise, I may have stayed in a little bit longer, potentially have gone to graduate school, and then served again longer in the service, but for me, it was the urgency to get out based on being a criminal.

I didn't want to go directly to business school out of the military. I wanted private sector experience, so I picked a great opportunity and went to Johnson & Johnson Orthopedics, had a great experience there and then went to business school.

The technology back, so the Internet was happening when I was at CVS, and I was doing corporate business development. That was one of the opportunities/problems and challenges that the company had.

Ben: What year was that?

Anna: I graduated from Business School in 1995. That was 1996–1999. I was at CVS and working on that as part of that time.

Ben: Okay, so the run-up had begun.

Anna: The run-up had begun. At the same time, Mike Galgon had come out to found Avenue aQuantive and we'd become friends back in Boston through playing pick-up basketball. We did peer mentoring, which I recommend. Peer mentoring is one of the most powerful mechanisms there. It's not always that fun, but it's actually one of the most beneficial and powerful mechanisms to grow as a professional.

We were doing that every month talking, and it ended up as I was getting more excited about what I was doing for CVS and the technology space, and he was growing a startup. That's how I came out to Seattle to join him and the crew at Avenue aQuantive.

Ben: What is peer mentoring? How do you structure that if somebody wants to do that among a small group? What should they do?

Anna: There are different ways to do it. I think for me, the critical piece is trust. Mike and I had a trusting relationship where we could share what was happening, what was working, what wasn't working, and what problems we had. It was (again) informal in that way. We just agreed to talk and catch up, then share what was happening, and be able to noodle with each other about what was going on in our different businesses.

I think (again) you can do more formal things than that. It's been true across where you have your trusted people, and you're able to talk with them about what matters and what's working, what isn't working.

Ben: Let's zoom in on that period for a moment of you decided to join Mike at aQuantive. That was a heck of a journey (aQuantive) especially in those late 1990s and early 2000s. For folks that don't know, this is a company that was one of the very earliest pioneers in the internet advertising space that IPOed, and then sold to Microsoft as a multi-billion dollar acquisition. There's a story in there that we'll dive into this show at some point, but Anna, I want to zoom in on the leadership component. What did you learn being an executive during that time period and how fast things were moving?

Anna: It was the quintessential chaos stepping into chaos. It was pre-IPO when I joined in August 1999. One thing is being flexible and adaptable. Even in the role, I interviewed and was recruited as VP of Business Development, but the week before I joined Mike (who was acting president at the time), said, “Hey, we really need a leader or this media group that has 30 plus individuals in it, and a manager to help figure out how to scale that portion of the business because we're growing like crazy and it's not working basically. Would you instead, come into that operating role, and be VP in Media as your starting position?” This is a week before I started and I said, “Sure, that'd be great,” because I preferred an operating role.

When I came in, (again) a lot of great talent doing whatever they could to serve as clients and to work. They were basically doing media buying, planning, and trafficking of the ads as the ad serving and trafficking of the ads. We were negotiating with Yahoo for every client separately, so there's a media buyer for every client, for every publisher on the Internet, like Yahoo, for example, and they were all negotiating separately.

One of the things I did was reorganizing structure to scale and to bring the negotiating power of leverage of Avenue aQuantive to Yahoo. We only negotiated once we said, “Hey, this buyer represents all of Avenue A's clients, and now we're going to represent that much money. Now, we're going to negotiate with you (Yahoo) for rates,” and that, as you can imagine, one of today's examples of one of the things. But with that, you have to move quickly, also bring people along with you, include them, understand, and work backward from the customer.

Those earning trust, but moving quickly and balancing those, is really a judgment call. I would say that's one of the things to really say, “You know what? You have to move quickly. Be flexible, adaptable, but also still earn trust with people, work backward, be customer-focused (if you will) and understand what that means for what the customer experience is as you're changing it, and scaling the business as an example.

Ben: I know we're moving quickly through your background here, but you've just done so much, and I think it's a really relevant context for our discussion on interviewing. Speaking of customer-focused, you spent some time in Amazon and you are the worldwide general manager of Amazon Prime. Amazon is famous for not only the interview process but what do they call them? The principles, the 18...

Anna: Leadership principles, 14 leadership principles. To be clear, I was one of the leaders in Amazon Prime, and I was the global leader for Amazon Prime membership, which did include retention engagement of the first few Prime days, Prime member spends, pricing. That's super fun to do. Again, a big team of people to be thankful and grateful for that experience. Amazon was a tremendous place and is a tremendous company. I learned a lot and got the opportunity to work with a lot of amazing people there.

You skipped over the Microsoft portion, which I spent about seven years of [00:14:31], which (again) I just skipped back to Microsoft in building out that search ad business from scratch. I hired with my team of people, over 500 people across the globe in 18 months, and created brand new roles to do that. That was actually the most phenomenal hiring practice that I've ever done both in scale and in speed. I can talk about that, but going back to Amazon, Amazon's leadership principles, and how they operate, Amazon is like a mash-up of Harvard Business School and the military because of how standardized they operate, with great adherence to process and the way things are done.

That's one of the reasons this applies across the business, whether it's creating a new business, working backward. We do in the working backward process with a customer, press release, and frequently asked questions or ERFAQ, or it's the hiring process, what that process is, how it's done, what the components are, and how the preparation is for that starting with the job description. What are the most important seven leadership principles (for example) for that particular role, and who should be on a loop? How do you do the pre-brief before the loop? How do you assign roles?

What you're basically doing is you're giving everyone a different, probably two leadership principles to interview by, and then everybody, you're basically getting a portion, everyone's doing a portion of the interview data collection, and you get a full picture of the person. This is exactly the same process, by the way, that we did at Microsoft, so it was not different at Amazon and Microsoft.

David: Amazon has done a better job of branding it.

Anna: Yeah, potentially, as far as the communication on that, but I think also in the way that it's practiced. In other words, there might be more variation of how well the process is followed in different divisions or groups at Microsoft. Again, that's changed since I've been there, it may have changed since I've done it, but during the period I was there.

Microsoft has their 7 values, at the time 33 competencies. It’s a similar thing and we did the same. So, how I interviewed at Amazon was not different from how I did it at Microsoft, with my teams as an example. It was different. We use different values/leadership principles, but the actual method was the same.

There's a bar raiser at Amazon, which is the person or the role on the loop that is there to do two things. One is to assess the long-term potential of that candidate, so are they a good fit for Amazon or the company? Do they match those values? Do they have the potential to grow? If you're hiring at X level, can they grow two or three levels beyond that? So, are they good long-term fit?

Number two, are they just a good cultural fit for the company? The hiring manager always has the immediate pressure of needing to fill that role to get the job done, the business done, so this bar raiser role is this long-term potential. And then also calibration. Is this person at that level if you're hiring them as (for example) a frontline manager? Do they have the right experience, competency, capability, and proficiency to operate at that level? So, you're calibrating for that level.

Microsoft has a role called an as-appropriate. It’s the same role.

Ben: Yeah, the as-ap.

Anna: As-ap, exactly. That would be an example of how those practices are really the same, but the adherence and standardization of those are greater at Amazon.

Ben: And with the as-appropriate, my understanding of the place where that term comes from, is they're the last person on the loop, and they only joined as-appropriate, if the rest of the group is trending up and they need that one person to come in and weigh in. Was that your experience? Also, the bar raiser, is that the same way that the bar raiser would be the last person on the loop?

Anna: Two things. One at Microsoft, (again) this is where we always had it. We had loops and the experience because you're going to construct a loop and say you want to do it to make it convenient and fast for the candidate as well as the company. To a degree possible, you would have them all stacked together, so you start at the beginning of the day and at the end of the day. Based on schedules, it's too complex to say the as-appropriate [00:19:27] yes, or the bar raisers are at the end. If you could do that, that's great. It doesn't have to be that way, though. In a scan with scheduling, it's virtually impossible to always do that. What's more important is that that role was on there.

Second, you don't get to know the other person's feedback until after you've done your feedback as an interviewer. At Microsoft, you could see the other interview feedback as an as-appropriate, and then be able to use that to probe on concerns. That would be one of the things that, as an  as-appropriate, as the rest of the loop happens, you could not only have whatever you're assigned values or leadership principles in the Amazon world, but you could also have other concerns that you want to probe on, that have come up through the loop today. That's the other piece of it.

David: You said Amazon being a cross of Harvard Business School and the military, how did you think about creating such a process-driven culture? What you have to do when you're hiring thousands and thousands of people, you need a standardized process. Making hiring decisions are human decisions. It's not like there's a right answer. What was the appropriate balance of a rigid process or a rigorous process and still having room for understanding that human element?

Anna: A couple things. Having a process is like if you ask someone about boundaries and creativity. You actually need structure for creativity. Basketball is played in a court that is 90 feet long, a free throw is 15 feet. It has a lot of rules and structure to it, but yet, you could have Steph Curry, LeBron James, Larry Bird, Michael Jordan, different styles of basketball within that structure.

In music, the same thing. You have the staff, the same notes, different 4/4 times, and you have Mozart to Madonna to Lady Gaga. I believe that structure is what is required for creativity, and for humans to actually connect in. If you have chaos, that's the opposite of being able to have an ability to connect in a good candidate experience. It's all about being human—you’re right—you need the structure to help facilitate that and to make it productive.

David: It's like, “Oh, let's just go out to lunch together.”

Anna: Right and at the end of it, you don't know. It's similar actually to PSLs structure for assessing a business idea. You have a way to do a voice of a customer interview. If everyone just, “Oh, let's just go talk to a bunch of customers,” but you don't know what data you're looking for, you don't have a hypothesis, then you could conduct 10 voice of customer interviews and have nothing at the end.

Ben: I've certainly made that mistake, especially earlier in my career. I was trying to do interviews without a clear plan, then I ended up spending an hour of both of our time and have no data at the end. I think that's very common for people early in their careers.

Anna: The other thing is it leads to a bunch of subjective bias when you're just saying, “Oh, it's this person,” because (again) we all have these biases around. Did they like things I like? Are they interesting? Are they engaging? Someone might be introverted versus extroverted. You start judging based on style and other factors, then what are the capabilities?

The interview is about assessing fit, fit for the role, fit for the company, fit for the potential to grow, fit for that level, so you're assessing fit. What are the skills, functional knowledge, experience, capability competencies? Those competencies include the EQ stuff and business judgment. “Right a lot” is a leadership principle at Amazon that is about judgment. Are you right a lot? Judgment is both quantitative and qualitative, getting back to the subjective. How do you assess someone's judgment? That's important, whether they're frontline first-level developers or they're a director leading a big business.

What I think is important about that structure is part of that also is reminding people that this is a two-way candidate experience, too. They're assessing, is this role, is this company, is this opportunity right for me? So again, you want to show up with the right game, intention, and organization so that they get their questions answered.

David: I love that analogy of the process is like the basketball court. With no basketball court, no rules, it's impossible to have creativity and greatness, but you can still have creativity and greatness within the process.

Anna: Absolutely, and being authentic to your point. To me, that’s actually showing up, having a plan, then being able to connect with the person while you're engaging in a dialogue around their experience. You're also sharing about what the opportunity and the role in the company is like, is important.

Ben: There's so much more we could touch on in your background, including the fact that you were the President and COO of Bulletproof, which I'm sure lots of people are familiar with disruptive consumer health and wellness brands. Let's weave that into the meat of the episode here. Talking about the interview itself, whatever is most useful for you, whether you want to talk about it like Anna Collins is starting a startup right now and how you would do this, or if you were at a bigger company, how you would do this. I want to talk about, you're about to hire some people and you need to figure out how to do that. What do you do first? What's your playbook? How should listeners think about this?

Anna: A couple things. One is we started to talk about it with the leadership principles and competencies. The first thing is interviewing or bringing people onto the bus is one part of the people process. There's, what people are on the bus? How do you bring the new people onto the bus? What seat on the bus? How do you develop people and move them from one seat to the next seat on the bus? Then, when is it time for someone to leave the bus, get off the bus?

All of that, that larger people process is rooted and grounded. Like the basketball court and rules for the basketball game, there are structures and guidelines for the people process. For me, those start with the vision, mission, and values of the company. For example, at Bulletproof when I got there, Dave (the founder and CEO) had done a phenomenal job of growing the business to a very large business at the time I joined, but it was still very bootstrapped and chaotic from how it was operating.

When I asked 10 people what’s the vision, what's the mission, what are the values, you'd get different answers. There were some essential things about it that showed up like it was disruptive. It was a bleeding edge, innovative brand. It's better for you, a brand that is about nutrition, it provides content and products to help people be healthier.

But how that was being communicated to both customers and employees was different and it wasn't (again) clear. So, one of the things I did in the first 90 days was start a process to define the vision, mission, and values with the employee input, customer input, feedback, then ran a process with Dave and the leadership team to define the vision, mission, and values.

It's clear the mission of Bulletproof is to create products and provide information that radically improves lives. Everybody can say that, know that. Values like gratitude, innovation, and customer-focus (for example) are critical. So, you say, “Okay, here are the values, here is the vision and mission.” Bake those into an interview process that says, “Okay, now we're going to interview for these things. We're going to interview.” Those are critical in addition to skills and functional knowledge for whatever role it is. Those are critical things to weave in, just like for Amazon is with defined leadership principles.

Those same vision, mission, and values get woven into the people processes for developing and assessing employees as well. With that, the skills and functional knowledge are really being assessed when the candidate is in that front end part of the selection process for candidates, before (I should say) phone interviews, the initial interviews by the hiring manager and the recruiter. You're really in your testing.

The actual final loop is about confirming those skills and functional knowledge, then assessing overall final assessment on that as well as the experience, capabilities, and competencies in full, but you're 80% sure the person has the skills and functional knowledge by the time they walk in the door for the full loop. You're still pressing and assessing for those, but your front end of the process has already done a lot of both in the interviews, those pre-interviews done assessing for that. Now you're going for this deeper, knowing about what their capability is, their proficiency level at other things, and they're fit for operating within that company according to the values.

Getting back to actually defining the role, it's important. When you're saying, “Oh gosh. What is this job description of the role?” In fact, it needs to be thoughtful in saying, “What are the key capabilities and leadership principles that are important for this role, and not just for this role, individually or standalone, but with the team?”

For example, when I was at Microsoft and I was the only person as I came in, they're like, “Create the business plan for the search advertisement.” There was nothing. It was me in a room by myself. As I created the plan and I started figuring out what are the different teams I'm going to need, what kind of leaders, then I was going to have to recruit industry experience because there was no search advertising experience in the company for me.

I also needed Microsoft DNA in the team and the current display advertising business. I knew I needed a mix and I started to say, “Okay. What do I need for experience both in industry experience, the different levels of leaders, and kinds of roles?”

One of the first people I recruited externally into the company was a former Avenue aQuantive, a mentee of mine, Randy Wooten, and I hired him as a senior director of this premium service operations team to build out. I was recruiting him back from an international assignment in London where he was running the Atlas business for aQuantive.

What was important is I was picking him because I need an international experience that I didn't have at the time. That was something I wanted in that role as an example. You're also looking at and saying what do I need for experience that the team doesn't have, that I don't have, that's complimentary, as well as if you [00:31:25] of capability for that specific functional knowledge? So, you're looking at broader. You're also clear about what's critical, what's not negotiable for that role and what's not because there are no unicorns. People are imperfect—we all are—so you have to say what am I willing to give up and trade-off in a candidate for this role?

Ben: You do that in the job description then. You have enough forethought when you're writing the JD to know what the critical ones are.

Anna: That's right. Some of it is literally in the job description for the capabilities or experience and some of it again is required. Some of it is preferred next to what's non-negotiable. As you're also learning, as you’re seeing candidates and you're talking to people, you also learn and get a clearer picture of what's absolutely critical and also what kind of talent is in the marketplace that you have available in a timeframe.

Ben: I have yet to make a hire at PSL where I didn't have to iterate the JD after meeting candidates. I've always wondered, is this a failure on me to define the role or is that actually just part of the process? Particularly when you have very few roles per year, where you need to understand what people are like to inform of what the actual job description can be, rather than perfectly birthing the JD and then having that go out to the marketplace. Is that necessary to iterate?

Anna: It depends, especially in the circumstance that you described. I think that makes sense, but if you go the other way and say we needed to build teams and we had to launch the ad business in France—which is the first country we picked after Singapore to keep the wheels on—we had to have these capabilities across these different roles. We also had more than one for it. Search media specialists and search media analysts are two roles that we made up.

We made over 14 different roles for these teams and created new roles. We had to create them, hire at scale, 30 not one, and do it in eight weeks. We didn't have the luxury of saying we're gonna do what you did for the one or two that are very unique kinds of roles. In some cases, you can do that.

For example, at Bulletproof when I went I didn't have any GPG (Consumer Packaged Goods) experience. I wanted for sure when I was hiring my VP of Supply Chain, my VP of Sales, my CFO, they had to have CPG experience, especially as we were going into retail distribution, which again, I didn't have. I had a lot of eCommerce in retail, but I didn't have natural food, Better For You products, and mass retail distribution in a mass way. Knowing what you don't have just like I did at Microsoft, there’s a difference between is there only one VP of Supply Chain or are there 30 media specialists or search analysts?

David: I was thinking when you recruited Randy into your new business unit for search ads at Microsoft, that was a case (I think) a lot of startups, particularly founders, find themselves in this position where you have a need and you have a former colleague who you've worked with, who you know would be perfect for it. How did you think about that moment where on the one hand that's so overwhelming? You've worked together before. You know that this person is great. You know they're gonna be great. Let's get them in there, get them working. How do you balance that with maybe there might be somebody better out there? Should I run a broader process for this or is it better to really wait, though? I know this person is going to do great.

Anna: There's no free pass. It wasn't like I thought of Randy and then he got the job. That wasn't what happened. In fact, he'd never worked for me directly in the Avenue aQuantive Business. He wasn't on one of my team, so I wasn’t intimate with his work, although I knew what he had accomplished and I saw him as a high-performing individual and leader. That was clear to me. He had the experience that I wanted. I knew him but I was picking him more by what his results, accomplishments, and reputation were.

I also looked at other people. He wasn't the only person. It wasn't that I said here's a guy. He was the best guy in the field of people. I worked really hard to recruit him which wasn't easy. He had said, “Hey, if you ever have something, I would love to work with you.” I said, “Hey, I'm playing that card.” By the way, he went through a rigorous process (like we're talking about) at Microsoft to get hired in at a senior level there. Of course, he did super well, was a key leader on the team, and continued to excel at Microsoft and beyond.

Ben: This question of having worked with someone before is an interesting one because it's related to a thing that I rely really heavily on in interviewing, which has references. I always find that what you can learn in an hour in an interview completely pales in comparison to a trusted reference. I also know that that's dangerous because there are lots of circumstances including bias that could cause that person's perceived performance in their last job to not be relevant at all to what they could do at your company. How do you weigh that?

Anna: A couple of things. One is constructing the loop of five people in your company to collect their own data independently in this targeted way across the capabilities, competencies, leadership principles that matter for the role to your company and for your company. It's the essence of what you're going to use. Also, if you have prior work experience.

I've hired another individual, Jeff Hall. I hired him three times, twice at Amazon and once at Bulletproof. He's back at Amazon now after the Bulletproof experience. Still he went through. He was sweating the interviews at Bulletproof. He had passed because he had worked for me before. He still hadn't nailed those interviews and be like is he right for this team, for this company, for this role?

The other thing is what you said, Ben, is really important, which is what's the fit for this particular opportunity position? Just because they were a great fit at that other company for that particular thing doesn't mean they're going to be a great fit. It could but that's what you need to suss out. What are those key capabilities and cultural aspects that make or don't make a good fit for that candidate in this role or position? The more someone is "athlete or general athlete" and has demonstrated flexibility and adaptability, looking for those kinds of signals, those evidence points where they've demonstrated these things that matter to you, that's what you really want to look for, whether it's in a reference. 

The same thing when you're having a reference. You're like, “Hey, is this a great guy or gal?” No, it's what specifically do they do? You're also looking for contra evidence that says what are the challenges in development and growth areas? We all have strengths and opportunities. Do the strengths match what you need and the opportunities aren't going to be barriers in that particular position or company?

There's no panacea. Just to be clear, I've made a lot of mistakes. I'm not 100%. I've made bad hires and I've been part of making bad hires both direct reports to me as well as extended on teams. There's no 100% on this. If you're right majority of the time and get anywhere close to 80%, then you're phenomenal.

David: It always strikes me in hiring how similar the process is to investing, really any type of investing. People obviously do talk about this, but I don't think enough. It's the type of decision and the type of impact that it can have and your ability. Nobody is ever going to be right 100% of the time. Warren Buffett isn't right 100% of the time. I'm sure Jeff Bezos in hiring isn't right 100% of the time. I really love what you said earlier about the basketball court analogy. I think that makes so much sense for both of these.

Anna: I appreciate that. It just reminds me that I have some rules of thumb that I use. One of those rules of thumb is when in doubt, throw them out. As I was getting trained as an as-appropriate at Microsoft, Michael [00:40:19] is the leader there who wasn't as-appropriate who put me through. It's not just you get trained, but you also do an apprenticeship.

This is similar for a bar raiser at Amazon. You're doing shadowing, apprenticing, and really through a process for a while until they are getting shadowed by someone else. They're like, “You're at the bar where now you can fly on your own and do this. We trust you as a company to go do this thing and basically be a barrier to coming in. If it’s not, you're holding the line basically for the company for both the calibration and the potential long-term hire for that company.”

When in doubt, throw them out. That means there's no on the fence. Everybody on a loop should say hire or no hire based on the competencies and values or at least some principles that I'm hiring on. In other words, you could say you're a hire based on what you interviewed for, but I'm a no-hire based on what I interview for.

That's job one for every interview on the loop, to go after and assess with the key questions, what they're looking for. They can also—if they have time—get other data, if you will, or say, “My gut was off,” or, “My spy sense,” or, “I had this question. I don't understand why.” You can add that as extra flavor, but you're making a decision based on what you assess based on your questions going after your signed competencies and values.

Hire or no hire, there's maybe a hire or maybe no hire. Remember that you're assessing [00:42:01]. You're also creating a candidate experience they're assessing. Your job one again is to get that data because otherwise there's no match, but they're also collecting data and you want to give them information about the opportunity and the company.

Ben: Do you ever go into cell mode in an interview?

Anna: Absolutely, the faster I get my data. I think yes, this is it then the faster I can go. Even if I know in my interview or think that they're not fit for that role, I'll be looking for, are they a fit somewhere else? If I think they're not a fit for the role and they’re not fit for somewhere else in the company, I’ll do that and search for that. If I think they’re not a fit for the role or the company, I'll still want it to be a good candidate experience because they are out in the world. They could be a customer or they could be a future. I want that to be a good experience regardless. I'm not saying I always do a good job of that or that I can't be intimidating during the interviews. I've heard that feedback, too, but I work to be connected, authentic, and also create a good candidate experience where I can.

Ben: This is something that Mike brought up to me. He was saying it's such a fallacy when people think about employer brand, that it's created by the people that they hired. I think his example was, we hired 1000 people or something at aQuantive but we interviewed another 50,000 that we didn't hire. Our brand is actually the sum of what they think of us, not what the small number of people who work at our company think of us in terms of talking about our company out in the world. 

I think it's an oft-overlooked thing, especially in startup land where people are just obsessed with efficiency, making the hire, moving on, building, and selling. A lot of people I think miss the fact that all those rejections that you're making out there, every one of those people has an anecdote of the way that they interacted with your company.

Anna: Yes. This also gets to the timeliness of scheduling the interview, communication back and forth, before and after, collecting information, how was the experience. That's the other you don't know. Certainly asking, getting a survey back, getting feedback back on why it wasn't that, and being able to use that to improve the process, that's an important part as well.

One of the things we didn't talk about specifically is the kind of interview questions, behavioral kinds of questions, and the way that Amazon and Microsoft that I've been trained, and even before that to do interviews is this behavioral questioning or behavioral interviewing which says what is the past? It's basically what you're doing is getting an example of past experience or behavior where they demonstrated this skill, knowledge, capability, critical thinking, or judgment, and you're asking a question that will demonstrate where they're showing or they're telling you about when they demonstrated this capability or skill set before.

A lot of those questions start with telling me about a time when blank, blank, blank. It's not yes or no. Typically again in an interview, if you're doing it, you only have a chance to ask one, two, or maybe three questions, and then you're probing within that question a lot to really get to specifics. That's the kind or type of interview method if you will.

David: Have you learned to handle those, the starting out, jumping-off point of that question?

Ben: Actually, no. I want to hear it as what's your favorite question.

Anna: I have a portfolio of questions. I have hundreds of questions. One of the things again, the benefit of having been around the block a few times is I do have this data bank of questions and answers where I've asked and I have my data set of what kinds of answers and what I think are good answers or not. It's just starting out with how I start out interviews. I do say who I am, my role, and I'm really happy to meet you. We're here to talk about XYZ opportunities. Thanks for making time to come in today or talk with us today.

I let them know who I am, what we're doing, setting it up, and that starts. Usually, I'm typing. I said I'm going to take notes and hopefully it is not disruptive. I'll try and make eye contact. I'm usually pretty good at it. You can also take notes if you want. I tried to set up an interview like that. Setting that up, I think it's a nice thing to do. Also I'll make time in the end to answer your questions but I'll start out with some questions so that they know what to expect.

I usually start out with a couple of what I call softball questions that also give me some information about how they've prepared, how interested they are. Again, I've asked these questions for decades now at all companies and all different levels whether they're college grads or senior executives. Those questions are, one, on a scale of 0–100, what percent fit are you for this opportunity and why? What are the three to five key factors why do you say you're X percent fit? From that question, some of them will say 80%, 75%, 90%, 100%.

First of all, I say opportunity. I don't say role. I don't say job. When I say opportunity, I mean opportunity in the broadest sense because that's including the specific role. It's talking about that business, that sector because it could be a division of Microsoft or it could be a startup company but it's talking about the role, the company, and the people. It's everything, how they match or what they know about the role, how they reflected that role to their own experience, how they reflected the company, the business, and product service, whatever it is, all of that, and how enthusiastic are they. That will show up as well. How prepared they are will show up to that.

The next question I ask is, on a scale of 1–10, what level of interest do you have, the opportunity and why? What are the 3–5 key factors in your answer? I say in 10 is you pay me money to have this job and one is I can pay you the money. Also when people answer the question, will they give me a number in both those examples? It's another piece of feedback to get that.

David: Have you gotten answers from 1–10?

Anna: Yes, I have, mostly on the 5–10.

David: Have you ever hired anybody who gave you a less than five number?

Anna: The number is the least important. Although if someone gives you a 10 or 100, I usually coach them to give me something out which I have received both those answers before more than once, but it's really the thinking that goes. That's the most important thing, not the actual answer.

Ben: I'd imagine that guides you to a place where they're talking about why they're interested in the roles then you have context of their perception of the company and the opportunity. I would assume your next set of questions no matter what value you're interviewing for gets shaped from there.

Anna: Actually, no. I have my questions that I'm diving specifically now if I will collect information. I'll use it and I might ask additional questions, but I have a job to do. If I'm interviewing, does someone have good judgment, their critical thinking or management leadership, then I have a set of questions or two that I have to ask to get that information in my mind.

Now again, I could use another one. I could segue. It's true. I could segue potentially usually the most, but then mostly I'll say we're going to move on. Thanks for that. Sometimes they'll share great stuff why their prior experiences are a super great match for the show. I'm like, “Gosh, that's great,” or, “That was a super accomplishment.” I also tried to really let the candidate know that I'm listening in an authentic way. People share what they've done and it's impressive. You're like, “Oh, wow. That's great. That's awesome. Congratulations. Thanks for sharing and keep going.” I'm not perfect at that but I pay attention. I try to do that as I'm going along.

Ben: Let's jump into one of these questions where you have a job to do. It's a question that you've premeditated. You've said I'm mapping the fact that I'm interviewing for this characteristic to this question. Let's say that characteristic has good judgment. What one or a few questions do you ask to tease that out without being obvious and direct and asking the person to, “Tell me about a time where you exhibited good judgment?”

Anna: A couple of things. One is it doesn't matter if the person knows that I am actually assessing their judgment.

David: It's not a trick, right?

Anna: I'm not trying to trick. Yeah, that’s certainly it. I'm not actually trying to trick, be devious, or deceptive. That's not my intent during all those. At the same time, I also don't want to be like, “Oh, yeah. What judgment do you have that you're getting?” That makes people uncomfortable. It's not actually the best way to get to it.

Those are battery questions, but one of my favorite questions for assessing judgment is, “Tell me about a significant decision that you made in the last year or two. One of the biggest decisions,” I say. “We've made small decisions every day at the business, around different things, but I want you to just give me a significant business decision that you made in the last year or two that you would change if you could. Why do you pick that decision? Tell me specifically about the decision. What you would change and what you would do differently?”

David: What are you looking for in answers to that?

Anna: When you're assessing a question, there's the star method. It's this typical behavioral interviewing method for what's the situation that they're starting with describing the situation. They're telling you what actions or what they actually did, how they actually did it, the why behind how they did, what they did, and what kind of results they got after they took those actions. The situation or task, the actual action, why behind the action, and the actual results that they got in a very qualitative or quantitative kind of results that they actually got.

For me, I'm looking at the judgment. First of all, what scale and scope of the decision. One of the things about assessing level—whether it’s an entry-level position or a senior-level executive—is scope and scale, capability and impact, and what they have demonstrated. When you ask for one of the most significant decisions, if they pick a very small scope kind of decision, you're like that was not a significant decision. That's not in good judgment. You picked that decision. Or, that's the biggest scope and scale you've ever had, so then you pick that decision.

David: Right. One or two things that’s wrong. Either you don't have experience making big decisions or you do and you just exhibit a bad judgment by not telling me about it.

Anna: And again, you're not trying to trick someone. You're really trying to set them up. Sometimes I'll give people do-overs if they pick something. I'm like, “It's not really what I'm looking for. Is there something else?” I want the candidate. They're there in the process at this point. I'm trying to help them be successful. I'm not trying to trick them. I actually want them to be a fit, but I'm trying to figure out. I'm trying to get the information that helps me know that or not to the best of my ability.

That's what I'm doing through that process. I'll say, “Is there another one?” I'll give them an example and say, “When I was building a search business at Microsoft,” or, “When I decided whatever at the summit,” whatever is appropriate, I will pick something out of mind to even help them give an example of this is what I'm looking for if they're struggling. My point is you're looking for what they demonstrate the capability, what level of capability is it? Does it fit for the role that you're assessing or looking for?

Another one is it's like teamwork, a collaboration, or earning trust in Amazon leadership principles is—Microsoft has one; similar when it's for teamwork—adverse. I call it an adverse situation question, which is, “Tell me about a time when you had a problem, a big disagreement with a peer. Not like we have small disagreements all the time, but a big work disagreement. Tell me about one of your biggest ones that you've ever had in your whole career. Tell me what it was, what happened, what did you do, what did they do.” I have them walk me through specifically in that.

Again, you'll see how they worked through that, what was their point of view, did they understand the other party's point of view, were they able to articulate it or not, how did it resolve, what did they learn, what would they do differently. You start to get to self-awareness, vocally self-critical capability, as well as your ability to team, understand, and work with other points of view.

Ben: Anna, what didn't we ask you about the world of interviewing that would be helpful for folks to know? The only thing that comes to mind for me is we didn't really touch on the debrief, how do you get all the data in a room and then make a decision. 

Anna: If there's a pre-brief, pre-guidance, the hiring manager owns the pre-brief. That's the other thing I think accountability for hiring. Maybe that's something we didn't talk about, the work and effort they go into preparing for, first of all, the role, the interview, the loop, the resume, certainly the process upfront and sourcing, and the ownership of the process. Recruiting does not own hiring or they are part of it, the hiring managers and the team that the hiring manager picks to partner on that process, but the hiring manager owns the process. The hiring manager is the one who's responsible, accountable, and needs to be the quarterback also of the process.

It's another role of the as-appropriate at Microsoft or bar raiser at Amazon is to make sure that hiring managers are doing what they're supposed to be doing, give that feedback directly to hiring managers if they're not owning it, being the quarterback for the process, and running a process that is effective as well as efficient. That does have to do with pre-brief. You're gonna send out and assign roles. You're going to prepare the loop ahead of time.

When there's a new role, having a conversation or pre-brief about the role, what you're looking for, and everyone's part on the loop is important. Getting an email and an actual meeting could be a short meeting like a 30-minute meeting again to do that, and then the debrief after where everyone has to get their feedback within 24 hours. You want to be efficient. You schedule a debrief with everybody.

David: In playing both pre-brief and debrief, the hiring manager should be leading both of those?

Anna: Yes.

Ben: When you walk into the room for the debrief, everyone's weighed in with their information. Presumably, the hiring manager has looked at everyone's feedback but has anyone else seen it?

Anna: Yes. After you have written your feedback, after someone has submitted their feedback, then everybody should be able to read whatever feedback is in. It's actually another great process point that you really do want everybody (if they can) to read the feedback before they get in the room for the debrief. Again, it goes much more efficiently. If they can't, they can do it real-time and that's where they're going to share their point of view with their area where they’re assigned.

David: I think there's even another point in here that you're probably assuming from being in high-performing recruiting organizations for a while, but I can tell you, it's definitely not the case for a lot of folks, which is that everybody who is on the loop writes down their feedback after the loop, right?

Anna: Yes. This is a practice. It's one of those things where it's extra work. It's also one of the things that I loved when as a senior leader in any organization, that people were like, “Oh, I can't wait to have you. You aren’t doing anything. You just think.” I'm like, “Are you kidding me? I'm doing stuff all the time.” Yes, I think some of the time, but there are no jobs that are just thinking jobs. All jobs have doing and thinking in them. That's one of my favorite 2x2s to do especially new college grads.

That's a little digression, but it gets to this point of, yeah it's a lot of work doing and selecting people. Guess what? it's the most important thing. I actually think it's one of the most important things a business does. You started out the podcast that way, guys. I do. Who do you bring on the bus and what seat on the bus as part of your team? That's it. It's usually the biggest expense companies have. All of this, it’s people.

It's super important and it's worth putting in the right effort upfront, to get the right people on the bus, and the right seat on the bus. Yes, writing down the feedback, submit it. It doesn't need to be a narrative novel. It can be shorthand but it needs to be understood, which is why at the high-performing interviewing teams that do this—big and small companies—just write the feedback, edit it right away, and send it in when they're done with the interview.

If it's not clear and it's the end of the day, you're going from meeting to meeting or interview to a meeting, you'll say, “Yeah, I'm gonna do it at night or the next morning,” and let it settle a little bit to say,” Here's my hire no hire based on what I heard.”

Ben: That's as good a place as any to leave it. Anna, before we wrap here, is there anything you want to leave listeners with? This could be just something that you want super entrepreneurial folks to know, something to plug, an opportunity to follow you somewhere, or anything like that.

Anna: One thing is, as I'm working on as an entrepreneur in residence, I do have the opportunity to do some consulting contract type of projects now if people want in this people area or other scaling businesses, et cetera. I'm available for some projects right now. You can find me on LinkedIn, message me there, and I'll be able to follow-up with you.

Ben: Awesome. Anna, thank you so much for your time.

David: Thank you so much. This is good.

Anna: All right. Thanks, guys. Super fun. Take care.

Ben: Likewise.

David: Stay safe.