Evan Wong (00:00):

Welcome to go with the workflow where we interviewworld-class legal leaders and experts in workflow automation to learn fromtheir hard earned experiences in making work more efficient and moremeaningful. Today my guest is Erin Crumb. Erin is a legal manager too atPinterest where she championed the first implementation of generative AI forher legal department before becoming a lawyer. Erin spent seven years as anentrepreneur and continues to flex her general skillset, managing a team,optimising legal operations whilst being senior legal counsel for technologywithin Pinterest's commercial legal team. On this episode, we did a deep diveon using a shared email inbox for legal intake, what the legal request workflowlooked like at Pinterest, the challenges with that and how that changed whenthey implemented Gen AI to monitor and respond to incoming emails. We talkedabout what it took to implement genai, the lessons learned and fielded a fewlive audience questions on handling edge cases, maintenance and changemanagement. I was so glad to capture such a practical and candid discussion ofan early to market live application of genai. So I hope you enjoy this episode.

(01:23):

Alright, we're going to go ahead and get started here.Thank you to everyone who has joined us here live today. If you haven't joinedone of these sessions before, this is called Go With the Workflow. We are aseries that invites legal operations professionals to come on board and toshare their knowledge around process improvement, efficiency, and automation.My name is Evan Wong. I am the co-founder and CEO of checkbox, and I have herea really special guest with me today, Erin Crumb. One thing that I learnedabout Erin, I think she likes having plants around like I do. I remember thefirst time I met you, Erin, you were in a beautiful sunhat outdoors surroundedby plants, and the only difference I think between Erin and myself is thatwhilst her plants are real, mine are always fake and artificial because I can'ttrust myself to take care of plants. I'm not as good as a plant mom like youare. So with that, Erin, welcome to the show.

Erin Crum (02:26):

Thank you for having me, Evan. Thank you so much and wewill work on your green thumb. I can help.

Evan Wong (02:31):

Yeah, I'm going to ask you for some tips because all ofmine are, I'm not kidding. Mine are artificial. That's pretty

Erin Crum (02:39):

Artificial.

Evan Wong (02:40):

Yeah, great. Awesome. But otherwise, let's start off.Erin, I have an opening question for you. You didn't start your career in a lawfirm as a lawyer like most. Instead you ended up building businesses, so tellus a bit about that.

Erin Crum (02:57):

Yeah, I definitely have a non-traditional background. Iwaited until I was in my late twenties to even decide to go to law school. Iwent to law school at 29, so before that I was a makeup artist. I worked inreal estate, I did corporate recruiting, so I had kind of some pretty good jobexperience before. And then when I graduated from law school, I went intocompliance at macy's dot com, which was a really interesting intersectionbetween technology and legal. It was like a primer on all things engineeringand product. It really helped me understand those fields without actually beingan engineer or a product manager. And then I had the opportunity to purchase a,I have a business partner. We purchased a small franchising company fromretiring, baby boomers ran that for about six, seven years. I was also thelawyer, I was a baby lawyer at the time, but I was handling commercial leases,financial audits, all the stuff, kind of trying to figure it out as I went,leaning on law school friends and people I knew throughout the trades and thenwe decided franchising wasn't the best fit after a few years.

(04:11):

We just didn't love doing that, but we learned so much. Wekept our corporate owned location, which we still run today. And then I wentinto various small startup size companies and worked in-house on contracts andthen landed at Pinterest about three years ago.

Evan Wong (04:28):

Very cool. I can see being an entrepreneur myself, so manyof the skills and challenges you face for the first time, and so naturally youhave to expand yourself into becoming a generalist. So I can see that in yourbackground come through. I'm sure that is actually a strength of yours now inyour role, which actually I want to ask a bit more about. You mentioned you'renow legal, I think it's legal manager two at Pinterest. Most people by the wayI assume would know of Pinterest, but just to get everyone on the same page,how would you describe the Pinterest business and more importantly, how doeslegal's role fit into enabling the business goals?

Erin Crum (05:08):

Yeah, so Pinterest is basically, it's a visual discoveryplatform, so anything that you're looking at, if you want to have inspirationon a home design of how to redo your living room, if you're looking forrecipes, if you're looking for travel destination ideas, it's kind of the go-todiscovery platform for someone seeking inspiration and to discover new thingsfrom a legal perspective. Our team runs the gamut. I mean, we look atintellectual property and we have a team that deals with trademarks and patents.We have a privacy team that deals with privacy laws. Those are always changing.Right now it's a big time in that industry. We have folks on the product legalside, which means they're embedded with engineering and product teams, helpingthem stay compliant with various laws and internal company policies. We have acompliance team, we have an employment team, we have a trust and safety teamthat's a very important team for Pinterest that keeps Pinterest safe online.And then we have the commercial team, which is my team that I work under. Myboss is the A GC of commercial and she oversees the entire thing. I just havemy section where I do probably 70% of the time I'm doing contracts and thenabout 30% I'm overseeing the legal ops team and that's the team that supportsthe commercial attorneys in the commercial contracting process at Pinterest.

Evan Wong (06:37):

Very cool. I want to ask you a bit more about that kind ofdual part of your role, but before I do that, I want to make a comment.Actually, I've used Pinterest very recently. I recently moved to New York City,a big change in my life and I wanted to decorate my room and I wanted to go fora Japan look. So that's literally, yeah, I use Pinterest as an inspiration forwhat to buy, and that's the little fake plant that I just held up was part ofthat decoration that I ended up doing, but it was inspired by Pinterest. So onthe dual side of your role, you mentioned 70% legal counsels, 30% legaloperations. I know that on the legal operations side you are responsible ofcourse for many things. I think mainly under the umbrella of contracts, butbroadly under the, I guess the concept of efficiency. So how do you think aboutefficiency in your role at Pinterest and specifically for the legal department?

Erin Crum (07:41):

Efficiency is a really top priority for our team. It'skind of evolved from being smaller pre IPO to then IPO and now kind of thismedium to growing size company and we're trying to keep up with that scale. Andso there's a lot of legacy processes that we just have to look at and say whatdo we need to keep doing and what is just something that we can streamline andget rid of? So that is a big part of our conversation currently within our teamis what still makes sense to be doing and what are we trying to solve for? Isit a company risk? Is it just like we want to document stuff? Is it a customerservice thing? And then thinking about is there a better, more efficient way ofhandling that or can we just stop doing it because there's, we have this bigpush right now on efficiency, very important and it's kind of like the people,the processes and the technology, trying to make those three things kind of fittogether.

Evan Wong (08:41):

Yeah, that's really interesting. I was speaking to a roomfull of legal department leaders the other day at an event, and a big part ofthe topic that I was leading was the point you made, which is there's a lot ofwork that ends up coming onto the desks of our lawyers and our legal team. AndI think the default because we want to be such great business partners is tokind of just get through it and respond to everything and try to do everything.But what I picked up on your language is what other things that we don't wantto be doing or should not be doing. And so it's actually a prioritisationexercise and almost putting types of works and classifying them into differentbuckets and then right, sourcing them. This is the things that probably isn'teven a legal question. Why is it coming to us? Is there a more efficient way toredirect it to finance or people and culture? And for the things that do cometo us, should a lawyer actually be working on this or should it be, I dunno,self-service or today's topic of course is around ai. That's definitely a veryefficient way to think about efficiency.

Erin Crum (09:44):

Absolutely. Absolutely.

Evan Wong (09:47):

When we met, by the way, one of the problems I think thatyou were looking to solve for was the idea of the shared mail inbox and you'venow solved that or solving that with ai and I think that's what a lot of peoplewho are joining us today are very much interested in is seeing how we'veimplemented or are looking at AI together. What I find, if not almost all legaldepartments have this shared inbox, right, legal@companyxyz.com and it's agreat way to centralise requests, but there's still I think a lot of challengesthat legal departments tend to face with this method. Things like someone stillhas to monitor that inbox. It's a real person reading, triaging, mayberesponding sometimes, but definitely escalating to the right people to addressthat request. I've heard from other legal departments that there's a double upin work that sometimes happens because it is a shared alias inbox.

(10:50):

People don't know which requests have been responded toalready and people are working on. And so there's a double up of work confusionwhen the advice might be slightly different. There's a lack of data of courseas well. When it's in email, you don't get the reporting and the analytics. AndI think the challenge that probably you were highlighting to me a few monthsago was the fact that you keep getting very similar questions and we werespending time answering them and that whilst it's important to get the businessgoing, took us away from obviously more interesting high value work. So I thinksome of those challenges you also saw at Pinterest. Can you walk us throughwhat the workflow was before when a request came into your legal at Pinterestinbox, take us from start to finish, what did that look like before?

Erin Crum (11:42):

So before we still have it a 70 page FAQ document thatjust one of our contract manager lead took the time to start capturing all ofthese same FAQs that keep coming in over and over and over again. She had overtime learned, what do we need to say to answer these questions? She kept arunning document ended up being 70 pages long, and so it would be someone is incharge of answering the inbox. We had a whole system of, okay, if you're out ofthe office, who's covering for you? So there was always someone in charge of,we knew that needed to answer requests would come in and then the contractmanager team would utilise that document to copy the answer to that questionthat came in and paste it into the response. And then we're still all copied. Imean, so everyone on the alias is getting, the answer comes in, we see itanswered, we see a request back.

(12:43):

Sometimes they're looping in accounting or they're loopingin another team. We're all getting copied on every email. Sometimes people aregood and they take us off the thread, but sometimes that just doesn't happenand we're just five days into it and they're still talking about it and it'snot even a commercial legal question. And then sometimes there's more escalatedquestions, so edge cases or attorney needs to respond to that or it's veryspecific to someone has to go find the agreement, download it and send it overor send a link to it over to someone. So there was kind of two types, like thefaq, the bulk of the inbound requests, and then those kind of more edge casesthat we had to get more granular when we answered. So that was the initial, theway it was done is so manual and just the same questions over and over and overagain because people think of commercial legal, we do have a lot of theanswers.

(13:42):

We work with accounting, we work with purchasing, we workwith a bunch of different teams that we touch, I think more than a lot of otherlegal teams. The vast majority of our work is just involving other, and thenpeople from across the company asking us to help them do their deals so we knoweveryone. And so everyone just comes to commercial legal to ask us questionswhen they could be going to the wiki or they could be going and finding thatelsewhere. It's just in their minds over time, it's like you train people, oh,you just ask, oh, anything legal, just ask commercial legal, they'll tell youwhere to go from there. So we became this directory of information and we hadto think about how do we put boundaries in place? What are we not going toanswer? Where do we send folks In the nicest way, we still want to besupportive and help people find the information. It's important, but we've alsogot to have boundaries for what our team is staffed to be able to handle andwhat really doesn't make sense for us to handle.

Evan Wong (14:39):

Yeah, totally. And although as you describe it out, it mayseem hyper manual and you would think that maybe a SaaS company like Pinterestwould be operating in a much more digitised way. It's actually completelynormal in the sense of I see so many legal departments operate exactly the sameway, which is they're still working out of the shed inbox and it is a verymanual process and there is a very key person risk in a person monitoring it.And when they're in a meeting, then who's actually then getting these requestsmoved along and there's a whole bottleneck process and it's actually a verycommon problem that I see. So you touched on a good point around therelationship piece. I think a lot of people are sometimes scared to implementtechnology when so much of perhaps the relationship with the business clientsare built on being the go-to person. So how do you think about the balance ofthat in on one hand it's good that the business comes to you, it means theytrust you and you're helpful generally, but on the other side, it's like thatreally, really takes you away from actually adding true business value. Andperhaps the paradox is that if you had more time, you can build more meaningfulrelationships on the things that actually do matter. Anyway, that's my littleopinion, but what's yours? How do you think about it?

Erin Crum (16:06):

I agree. Yeah, it's definitely a balance. We never wantpeople to be frustrated. Everyone's tried to call and change their flight ortalk to an airline and it's hard to get someone to answer your question. Youhave to go through a bot that doesn't really understand what you're asking. Youhave to answer a bunch of questions, it's not really hitting what you're tryingto get. You're already frustrated because of it. And so in a workplace setting,you're just trying to do your job, you just need the information. This is whereI usually went to find it. What do I do now? Now you're sending me away. So itis a balance of our company has the resources and the q and a informationavailable. People just might not know about it. So part of it is just findingthe actual policies and finding the actual URL where these policies are hostedinternally on our wikis and sending folks there so that now they have any, it'supdated in real time.

(17:05):

So now it's like, okay, I can bookmark this and this is myanswer now the purchasing policy I know or the signature policy, I know if Ihave a certain agreement over a certain amount of money, this person has to bethe one. I don't have to now ask the team every time. I can actually just copyit, copy this to my bookmarks and find the answer. So those kind of smallerlower hanging fruit as they say, trying to make those, just make theinformation more easily available to them, but in a way where we're not turningthem away, we're just saying it's faster if you just go right there, you don'teven have to wait. And a lot of times now as a globally scaling company,sometimes the US is closed, we're mostly US based, so sometimes we're off forMemorial Day or some US holiday, but they're working abroad somewhere else andso they now have to wait another 24 business hours to get a response to maybesomething that's a critical question. So it was like, what can we do? We havethis bounce back email for people when they email our alias and it's like, ifyou need this, go here. If you need this, go here. If you need this, go here.If you need access to this team. And it's just a bounce back that hopefullyresponds to most of it, but a lot of times folks still have to wait a littlebit to get a response if we happen to be closed for a holiday or something.

Evan Wong (18:26):

Very insightful. And obviously that was where you were,but now very excitingly your shed inbox is powered by Gen ai, which by the wayI think is gen ai. So many legal departments have been talking about it for thelast maybe year and a half since Chachi PT blew it up. I think a lot of peopleare in exploration phase and dabbling in it, but you've sort of been a bit of afront runner in that you are truly implementing it. So I think there's a lot ofinteresting insights we'll unpack for the rest of this session. It's now run bygen. Can you just explain maybe to start with what that actually means? Whatdoes it mean to have a gen AI powered shared inbox?

Erin Crum (19:12):

Yeah, so we took that 70 page document that I talked aboutearlier that just had the canned questions that FAQs that come in, and then thecanned responses that we would usually just copy someone would physically copyand paste and we took that and genai is basically it needs a training documentand in this case we're using checkbox and so checkbox is already trained as abot on GPT, so it already has the ability to interact, but now it needs to knowour specific use case. It has to be trained, so it's like hiring a new internor a new employee to come in and help us. So this employee has context fromliving in the world and having experiences in life, but it's like trainingsomeone to understand the nuances in our specific department and how we answerquestions and where we send people.

(20:00):

So we took that 70 page document and basically edited itslightly to make sure that it answered the questions for the bot in the waythat the bot couldn't read and understand. And there's user acceptance testing.You have to go through a lot of testing to make sure, why is it saying thisstrange thing? Sometimes it'll misunderstand what you're trying to tell it tosay, and so it'll respond to something totally weird. And so you just have togo back and adjust some things and figure out where in the document it'sgetting that information and then you just kind of get to the point where it'slike training a new person. It's like having an employee where it's not a oneand done set it and forget it type situation. It needs to be updated all thetime. There might be questions that we're going to get in that we didn'tanticipate.

(20:48):

It's not on the FAQ document and the bot for some reasondidn't realise that this is an edge case. We have a response that it shouldsay. If it doesn't know, we want it to specifically say, I don't know. We don'twant it to make up stuff, but sometimes it will try to make something up in aminor way and so we'll have to go in and email the person and get thatadjusted. But over time, that's just going to get, it's just like having a newperson who answered it wrong and we have to help with training for that person.It's a continuous process, a lot more effort on the front end for more of along-term, more efficient process down the line.

Evan Wong (21:28):

I love that analogy. I think it makes it really easy tounderstand and get you in the right head space for how to use ai because I feellike usually with new technology, and I've seen this cycle over the years beingin this space is people expect that somehow technology is going to magically asyou say, one and done solve the problems with very little effort. And that'srarely the case. There's always going to be some readiness work in your case,you've already done it in terms of having that 70 page document, that was asort of a low effort knowledge management tool that you use for the emailinbox. Pre ai, probably a lot of legal departments don't have that 70 pagedocument and they might need to sift through their scent inbox to even getthere, but there's readiness work and then there's working with the AI and thenunderstanding and having some patience around finessing it.

(22:24):

You touched on some really important concepts that I thinkpeople today are generally thinking about when they think about AI aroundaccuracy and hallucination. I love your approach of the importance of having atesting period and that it's okay and it's sort of a process that you walkthrough very similar to someone new. Starting on the technology side, we thinka lot about hallucination and accuracy as well, and there's a lot of technologycontrols. What else have you thought about in terms of trying to ensure thisbot is go live ready? So besides testing, was there anything else that you didto give comfort to not just yourself who's obviously a big champion of change,but maybe some of the lawyers and your boss, how did you bring them on side aswell?

Erin Crum (23:18):

There's always this apprehension on what type of data isthis AI getting? And even though we have really good knowledge that we're notusing, using our data to train its models, there's still that risk. And so Ithink there's a twofold approach that we took, which was what can we minimisegiving the bot so that it can answer things in a well read way, a helpful way,but let's not overdo it then if we're giving it so much information, sometimesthat gets scrambled. So what can we link out? So instead of what's the policyon purchasing, we don't need it to spit out the policy on purchasing this, wecan just say, here's the link to the wiki that has our policy on purchasing.And so we're not giving it a tonne of information. It's also not having to beupdated every time the purchasing policy is updated, it'll just be updated onthe URL, which we're sending to them to access in real time.

(24:16):

So yeah, we're not putting any sensitive information inthere, we're just linking out any internal policies. And so that gave a lot ofcomfort to the fact that everyone's really worried about it telling the wronginformation or having information it shouldn't have. And it's just the onlylinks it's getting that it's sending to the person is you have to be logged inthrough our Pinterest systems. It's only going to respond to a Pinterest emailaddress is the other thing that was really comfortable for folks that it's notjust random someone off the street, it's only going to know to respond if it'san at Pinterest email. So those two things really help mitigate the anxiety Ithink for security and then also our management team. But it definitely takes alot of building trust within your team in order to get the buy-in because itdoes sound scary and there is a lot of bad PR about AI and what it can do andhow it can go wrong.

(25:18):

But if you think through some of those issues, learn aboutthe tools, you're not technical, there's lots of ways to learn about it still,even if you're not technical. And so that way you can really think throughthese issues and partner with your IT team to get it into a place that feelscomfortable. That's not too risky because that is a concern and we got a lot ofquestions about that. Our security team, our privacy team was very interestedto know what kind of information was getting passed through and even ourcorporate legal team and different legal teams, please make sure it's notanswering legal questions. We don't want it to be giving legal advice. We don'twant it to be giving policy advice. We want it to be linking out to everythingthat it needs to instead of doing it everything itself.

Evan Wong (26:05):

Yeah, that's really good advice. I feel like the themehere is think about limiting the scope in 2 cents, the scope of the users whocan actually use the tool, so internal Pinterest only, and then limiting thescope of even what the bot is designed to do or answer non-legal work,nonsensitive not high complexity, high risk type of use cases. And so I thinkyou're right. A lot of the times when we use, let's say chat jt, we're notworried about putting in the ingredients of our fridge into the tool and askingwhat can I create me a recipe because that's not actually sensitive informationthat someone now knows in my fridge. And so when you extrapolate that to thebusiness level, there are definitely use cases of AI where it is a lot lower inrisk profile where you can start to build that muscle and create theseefficiencies today rather than waiting or standing back and being scared toengage. So that's really good advice. We've got a question here, by the way,from Bree, from the audience, she says, hi there. If there is an edge case one,what does the response read? And two, what is the next step of the legaloperational process? Meaning how does the team then route that edge case? It'sa great question. So over to you, Aaron.

Erin Crum (27:26):

Yeah, so right now we just started with the bot. There's alot of functionality within checkbox that we're looking into. We wanted tostart small and then kind of add, so right now it's taking care of most of ourFAQs, the edge cases go to. It tells it to email our other alias. So in certainsituations if it doesn't know the answer, it says, I'm so sorry, I can't answerthis question. Please email this other alias, our secret alias that we don'tadvertise on any internal wikis or anything. It's only in certain cases whereit can't answer the question or it knows it shouldn't, that it sends over for ahuman to answer and then it has the person email us again at a different inbox.Ideally, what we would like to do is hook up the ticketing component so that itautomatically would just go and get assigned to someone.

(28:18):

And that's another component of checkbox technology thatwe haven't implemented. We wanted to start, we didn't want to boil the ocean,let's start small, see how it goes, and then kind of add things as needed. Buteventually we will do more of a Jira type ticketing system where it getsassigned out and then there's a way, a better way of tracking it. Right nowit's a little bit more manual. It is cutting out a lot of just the FAQ stuff,like 70% of what comes in there, is that still 30% still that's going to remainfor a little bit longer where we're going to have to manually figure out who'sgoing to respond.

Evan Wong (28:53):

That's great. And again, I think you've embedded somegreat learnings there as well. For people. It's like with technology, there'salways going to be so much functionality that can go broad and deep, but youreally want to pick what is the smallest low hanging fruit to really drivesuccess as early as possible so you get those wins on the board to thenimplement more change and expand from there. So yeah,

Erin Crum (29:20):

That's awesome. Yeah, it's a lot to try to do everythingall at one time for the team from a just training process of like, okay, here'sa new way we're going to be doing things. Just doing it little by little Ithink is a little bit more digestible. Once we have the wins, like you said, wecan get through and then be like, okay, so since this is working, here'sanother way we could add to the efficiency that typically in my experience, hasgone a little bit more smoothly from an adoption perspective and a technicalperspective. Things as you're trying things out that you're realising, oh, I'mglad we didn't implement it that way, that wouldn't have worked or based on theway our setup is, that actually isn't ideal. So getting in there, getting thequestions kind of like a build measure, learn kind of mentality about it andthen to try to replicate the product people, we try to do that over on our sideas much as we can.

Evan Wong (30:16):

Yeah, very cool. That's a lean startup methodology thatyou've just dropped there. That was I learned at the very beginning of mystartup career, so that's really cool to hear that. Again, we have anotherfollow up question actually, which is how do you adjust these enterprise wideteams to change? So what was the change management approach that you used toget these teams to now engage with the bot?

Erin Crum (30:46):

I think we just kind of turned it on. We didn't tellanyone we were doing it because I think once, and I think we asked you actuallyEvan about your opinion about this and you said, and I think it was right, thatif you tell people about it, they're automatically going to be a little freakedout about it. Just like, oh, now I'm going to have to talk to a bot. Or if youjust turn it on and just let it, you're monitoring it. So we're looking atwho's getting, you can go into your dashboard and see what inquiries are comingin, how is it responding, and you can go in on a daily basis and follow up withsomeone if you're feeling like, oh, that wasn't right or correct things as itcomes through manually because there is that front end work that you have to doto refine it.

(31:30):

Just looking over the shoulder of an intern. But wespecifically didn't really do a lot on the communications for outside of ourteam because it's conversational. You give it a persona when you set it up. Soyou say you're a world-class legal assistant at Pinterest, and you are going torespond by in a friendly, professional way. And so you tell the bot how to act.You can also give it a funny persona like Dumbledore I've heard too to make itsound different and fun based on the time of year or something, but we haven'tdone that. But you give it a persona, it sounds like a person, it'sconversational, it's not bot sounding. And so we thought that for most of thequestions, they're just going to get the response back and they won'tnecessarily even be thinking that it's not a person on the other end. And thenwe'll be monitoring in times it has gone wrong. We'll be monitoring that andfollowing up with folks as that happens.

Evan Wong (32:32):

And if it wasn't clear in the way, I know that peoplecan't visualise it on this particular session, but if it wasn't clear thepeople interacting with your shared inbox is interacting with it, they, there'sactually no change in that sense. So the question around how have we conductedchange management? The answer is, but we didn't really have to because wehaven't had to change the behaviour of how people came to legal. Sure. The waythat we now respond to the questions is through ai, but it wasn't traditionaltechnology implementations where you now had to say, oh, log into this newsystem and click this button and now fill in this form on top of all theseother systems that you probably have to work with. No, it's you're alreadysending us emails, just keep doing it the way you used to and now we will justin the background, gain the efficiencies on the legal side without having tocreate a whole bunch of change for the business.

(33:29):

Right. It's interesting, I, it's almost counterintuitivewhat you mentioned around not communicating that change, at least on thebackend of the business. I think it made me think about, I think people usespellcheck on word sometimes nowadays as an example of AI that we've alwaysbeen using for a while. And I feel like if you told lawyers that Spellcheck hadAI when it came out, the adoption of that would be significantly lower becausethey would be scared of AI reading documents and things like that. So it'salmost a little bit of like you've got to find the right balance of what helpsdrive change versus maybe impedes it right in this case. Very cool. Okay, soit's still early days. I know it's still early days for you, but what impact doyou think this will have? What do you anticipate to be the difference now forthe legal team or the business? We talked about the challenges I think of the,but if you can make it explicit, what are you anticipating in terms of theimpact of this change here?

Erin Crum (34:40):

Well, I think from an employee perspective, it is kind ofa bummer if a lot of your day is just copy and pasting the same thing over andover again and hitting send on an email, that's not stimulating, that's not avery fun thing to do for your job. And so one of the big things that I thoughtwas a benefit is just removing this repetitive, less meaningful work out of aperson's day and giving them the opportunity to get involved in more meaningfulprojects and more meaningful work in other ways. So now they're freed up timeto help us analyse the history of a deal a little bit more, or now they canspend time researching a special project or pulling information or helping usfind information better. And that helps build their skillset and it helps themfeel like they're actually contributing in a more meaningful way to the companyand to their own professional career growth.

(35:36):

So that was number one for me. I know I've done jobs likethat. It's a real bummer. So what can we do to just get rid of stuff like thatfor people? And then also just from an efficiency standpoint, one of our KPIsis efficiency. Now we can show we are able to reduce manual intake and responseto a lot of our inbound questions, and I think that's a big win for managementto see. And it's also going to, the changing AI landscape and the technologyavailable to legal teams is changing at such a rapid pace. So jumping on boardin even a small way, this tees us up to adopt more technology in the futurebecause now our team understands user acceptance testing and they understandhow to project manage implementing a new technology into the team when theyhaven't really had to do that a lot in their jobs.

(36:32):

They're more like contract manager type folks. And sothey're expanding their skillset and our team is now being more ready to adoptfuture technology in a way that seems less scary for us. Now we've done it, weknow what's what's entailed and we've have a process for it, and we have now anice refined source document that our AI was trained on. We can now use that inother ways, and we can use the learnings from that to adopt other types ofthings like a contract analysis AI that pulls out contractual language based onyour playbook and all those technologies that are coming out for legal thingsthat we can't even anticipate in the long term. This is taking a step towardskeeping up with this rapid change the technology is presenting to us at thiskind of new era for the legal industry.

Evan Wong (37:28):

Yeah, that's a great attitude and a great culture. I'm sohappy for you and your team. And you just touched on really interesting, a lotof even the learnings you just mentioned is translatable. It is not necessarilyspecific to implementing ai. It is a lot of the learnings are aroundimplementing technology, and I think we have another question that's just comein from the audience, which is related to that. So it ties nicely. The questionreads, where does the bot sit within the various systems of the company? Whatdoes the technical implementation process look like? And I'm going to dovetailthat into a question I have as well, which is there is always the technicalimplementation piece. I know for you, part of that is obviously setting up theemail itself with IT to set up the rules for forwarding and the IT email stuff.And then there's the second part, which is more novel to do with the actual genAI component, like the actual gen AI setup. And I think you touched a littlebit on that, but yeah, it maybe answered the audience question, my question,which is what does the whole setup or the implementation process actually looklike? And maybe you can sprinkle in some learnings as well having gone throughit.

Erin Crum (38:44):

Yeah, great question because I just thought you just turnit on and hook it up to your email. Turns out it's a little bit morecomplicated. And luckily we have an IT team that really was supportive of this,but it just integrates into your Outlook or your Gmail or whatever your companyuses for an email server and it integrates, so you have it so that you have tobe signed in through Okta to access the checkbox platform, the dashboard sothat the administrators can kind of see all the questions that are coming inand do testing and do all of the configuring through the dashboard that youhave to, and it's in your Okta, like a tile on your Okta homepage. But as faras there's just this connection between, and it's magic because I don't reallyknow how it works, but it's like email forwarding between your Outlook or yourGmail that connects into checkbox bot and it knows how to forward the questionappropriately from the bot to the requester and back and forth. So yeah, it'sjust backend IT kind of stuff where they talk about email forwarding and allbunch of acronyms that I still don't know what they are talking about. Andthere's this backend setup where it knows to connect, and then somehow they'reable to tell it only answer at Pinterest email. And so there's a couple ofcomponents. There's the Okta login for the administrator piece, and then justfrom a user perspective, they're just emailing it and that's going directly tothe bot and back and forth.

Evan Wong (40:22):

And then that's helpful. And then to summarise the gen AIpart, you're saying before you've got the source document that you had beforeanyway, and I think different organisations might have a wiki page that theymight have their knowledge sitting in or policy documents. And so that's sortof like your corpus of responses or knowledge and you've taken that, you'vecleaned it up a little bit to make it more AI friendly. And I think some ofthe, just to share with the audience, maybe some examples are instead of adiagram, describe the diagram in words or instead of using language, I know youused yours is email reply templates. So it might be like, oh, thanks for yourresponse. I hope you're well that the AI doesn't need, so we can take out theflower really language, but you're really doing the tweak to the sourcedocuments to make it more efficient for the ai. And then you are clicking anupload button. You just uploading it.

Erin Crum (41:18):

You just upload. Yeah, you just upload it into thedashboard for your bot, and you can have several different bots that are hookedup to several different emails. And then for each one you have a sourcedocument. It's just like you click on the button, you upload it from yourcomputer, the newest version, especially at the beginning, you're changing itand you're doing different versions of it. Click the most recent version andupload that in, and now it's working off of that. So just like a pretty quick,now it's just now training on this new source document. Whenever you want toupdate it, you just go in and get your word pulled up or your Google doc pulledup, make your adjustments, and then upload that into checkbox, and then that'show it gets updated with the bot.

Evan Wong (42:04):

Yeah, I'm excited for the next phase of yourimplementation where, like you said, you kind of start to connect it withticketing. Some people don't like the word ticketing, so I'll use the wordmatter management. So I'm excited for that. And also the ability for theresponses to also link out to actual workflows to get actual things done aswell. Because I think with the bot, it doesn't just consume a single sourcedocument, it can consume multiple along with different types of workflows. Andso at some point it's intelligent enough to say, well, based on this request,I'm going to reach into this source document and pull out this answer. And Oh,we have this workflow that's also relevant and I'm going to serve thosetogether so that the user gets the response and is able to finish off whateverself-service or process with a lawyer. So I'm excited for that next phase, andmaybe we will have you come back on again to discuss that. I've got anotherquestion from the audience from Kelly this time. She asks, how are you makingsure that the source documents remain fresh, especially if the content ismanaged and owned by multiple different functional teams? It's a good question,Kelly. So Erin, how do you do it?

Erin Crum (43:16):

Yeah, we try not to answer every single question withinthe response into the email. We try to link out as much as we can to wiki pagesbecause those are where things are getting updated. So if you're asking what'sthe policy on some type of compliance question, we're not going to answer thatin the email. We're going to say, thank you for your question. It sounds likeyou have a question for the compliance team. I think your question was relatedto X, Y, Z. Please find the link to the compliance wiki page here. And so it'lljust link out. It'll acknowledge their question, it'll tell them what it thinksthey asked, and then it'll provide a response and a place to go to navigate.But it's not in the body of the email saying The policy on that is because thosechange. And yeah, it would be so hard for us to have to remember, oh,compliance updated that policy.

(44:13):

Oh, purchasing updated the purchasing policy. Oh,accounting now updated that policy. It's too much for us to try to always beupdating at all times. So each of those teams has a wiki page within Pinterest,and so we're just linking out to whichever document or wiki page, that's wherethey're keeping this archived or latest version of whatever policy or proceduretheir team wants you to know about. We have Google Docs and there's a wikipages that we link to. So yeah, we just want to avoid trying to have to do thatbecause it's way too manual.

Evan Wong (44:54):

Absolutely. I've seen teams that have a hybrid approach aswell, depending again on the use case because I think Aaron, what you'vedesigned here makes complete sense from a low maintenance perspective. Linkingout is a fantastic way to ensure that we're always referencing the latest andgreatest. I know that teams do that approach together with policies and rulesthat either one don't get changed at all very often, and so the bot can kind ofconsume directly from it. And then if it then goes to more volatile content, itlinks out. But also there's a second use case. I've seen teams use this AI forthings like delegation of authority, and in that case, their actual policydocument is the problem in the sense that there is just so much complexity inthe tables that they have and all the factors that must be considered and theseexception rules.

(45:51):

And a human gets completely lost in trying to read that.And so even if you link out to that policy, people are still confused. And sothere'll be some use cases where the bot is consuming directly that documentand then giving the answer in a very kind of clear cut human response. But Ithink the hybrid approach, and back to what you were talking about before, it'scarving out the right use cases with the right design under the right scope isa very intellectual decision. And I think there's so many ways to use ai, and Ithink we're all still kind of new to it, but it's really good to share theselearnings given that I think we've begun to do some real stuff here and there'smore questions coming through. Sorry. So I think Kelly has a follow up. Oh,okay. This is a good one from Kelly. She says, if the content overlaps, how areyou making sure that the bot is targeting the responses to the applicablecontent?

Erin Crum (47:00):

I'm trying to think of a case where that's the case.Usually it'll just know by the keyword, like a policy on X, it'll know this iswhere I send you, it'll send you maybe to an overall wiki page, maybe not justthe policy page of under that wiki page, but just the general page, the landingpage of the team. But yeah, I don't think we've run into that as much as anissue. And if it doesn't know, it'll just, that's when we have to manuallyanswer. If it's more of a complicated, they want a bunch of different stuff andit's hard for it to figure out which answer it should give, it'll just defaultto having us manually answer. We haven't really come, I haven't seen thosetypes of questions come through for us often, but yeah, I think that wouldprobably be one that has to get manually answered if it's too confusing for thebot.

Evan Wong (48:00):

Yeah, I always say it's a real concern. It's a realconsideration, but I always say, if you gave conflicting information frommultiple different policy documents to a human being and ask them what theyshould do in this scenario in terms of to interact with the policy, the humanwould not know what to do. They would get confused too. They'd be like, oh,what they don't know. This is all conflicting information. So it comes backdown to actually knowledge management and governance at the root of it, right?Technology is an enabling layer. It's not going to suddenly figure outinconsistencies because a human can't do that, right? It needs to be a humandecision and a governance to then enable tools like the one we're talkingabout. So hopefully that helps. Kelly. I think Kelly also suggested or askedwhether checkbox has a way to automate the reformatting of content to make itmore AI consumable. The answer is, I wish we did. That'd be a really coolfeature to upload something. And then the AI turns it to be more AI friendly.That's like meta ai, but the answer is no. At the moment, our customer successteam works with our customers like Aaron, and we pass on best practises andhelp shape up your documents for you. But it's a great suggestion. I'll send itto the product team.

(49:23):

Let's see. We've got one last question in the audience Ithink that's sitting here. That is, can you explain which inquiries aresuccessfully answered versus not? And so how do you address the ones that areleft open? Are there parameters set in place such as you can only go back andforth a certain amount of time before it gets forwarded to the team? Cool. Sohow do you know when something doesn't get answered correctly or it can't inyour tool? Aaron?

Erin Crum (49:53):

We are doing a lot of just monitoring the dashboard. Sowe're on a daily basis looking through the questions that are coming in andmaking sure the responses are accurate, and if they're not, we're following updirectly. And so it's just unfortunately kind of a manual, especially at thefront end of the process. It's very manual. We are hoping it learns over time.It'll be better about saying when to email our alias, but sometimes we haven'trealised, oh, we need to tell it to do that. So as we learn what kind ofquestions are coming in, we're editing the document to match some of thesenuanced questions that are coming in. So over time, it'll get smarter aboutbeing able to know when it can't respond and what to do in that situation. Butespecially at the beginning, we're still in that phase of trying to figure itout. So it's definitely more of a manual process right now and then goingforward, it still will be. I mean, there's going to be just someone who checksin with it consistently to make sure that folks are getting what they need andthat we're not leaving anyone hanging.

Evan Wong (51:04):

And Kelly has just asked to follow up around, so does thebot have a memory of these chats? And the answer is yes, memory is a reallycool word. We don't use that. But like an audit trail, you can actually see arecord of all of the questions and responses that has occurred between the botand people asking it questions. And that's what you mean by dashboard, right?Erin? When you are monitoring what's coming through, you're actually looking atthat transcript or the record of everything that's come through. There's also afilter, by the way, Kelly, that you can set to show only the responses wherethe bot could not answer. And so you can also see very quickly where maybethere's some gaps in the policy or knowledge that the body's trained on. But Ilike that the human in the loop is another very important concept to AInowadays that we're all coming to learn with.

(51:53):

It's like trust, but validate is kind of the motto. Youneed to trust AI to do AI's job because otherwise, if you end up, I've heard ofpeople who use an AI contract review tool and then they use it, but then theyspend just as long checking the AI's work and then redoing it to the pointwhere the ROI is now negative for implementing the technology. So you kind ofwant to trust it but still have the option to verify. So human in loop isimportant. But let me finish with one wrapping question to you, Erin, which isthinking kind of broader, bigger picture, you're one of the first to dabble andimplement gen AI in legal departments. What are your thoughts about gen AIgenerally for the legal industry? And then I don't even want to say in the nextX years because I feel like this is moving so quickly. So I'm just going to capthe question at that. What are your views for the impact of AI on the legalindustry and in the near future?

Erin Crum (52:59):

Yeah, I don't think it's going to replace, but I thinkit's going to very much make our lives a lot easier when it comes to justcomposing a quick response to something, to summarising an issue. Sometimesit's really hard to contextualise for a non-legal person when I'm trying toexplain when I'm doing my other piece of my job, which is commercial counsel,and I'm trying to explain to a business stakeholder some of the issues that I'mseeing involved in the negotiations and having a tool that I could just type instream of consciousness thoughts about this and say, please write this in aconcise way to a non-legal stakeholder, and it can summarise my thoughts in amore succinct, approachable way. So there's things like that. I think there'sjust going to change the pace and the ability for us to communicate better overtime and faster.

(53:51):

There's of course, ethical and regulatory consideration.So a lot of it is just staying on top of webinars and conferences and subscribeto some substack and stay on top of it. Find some people on LinkedIn to followthat are thought leaders in this space when it comes to legal and ai, and justcontinue to stay on top of it and try to participate with it as much as youcan. Because one thing to be kind of a sideline watcher and understand it fromthat perspective, but it really helped me adjust my understanding of what goesinto this technology when I was actually in it and working on it. So there's somuch future opportunity for AI to really make our lives much easier, like acentral place. We can ask it a question and it knows everything about everydocument in our organisation, and we can ask it and it'll tell us immediately.There's no having to go find it. Just things like that, it's going to happenfast and we just have to stay up incrementally. It's harder to jump from A to Cif you haven't gotten to B yet. So yeah, that's where I think it's justengaging with it, trying to stay curious and not being afraid of it, becausethe more you know about it, the less scary it can get.

Evan Wong (55:13):

Wow, that is an amazing wrap. I could not agree more.There is so much that you learn by doing rather than just reading theoreticallyas well. Both are important, but that is great advice. But otherwise, I'm goingto wrap up by saying, Erin, thank you, thank you, thank you. This was anincredible conversation. You've been super gracious with the candour and yourexperience. I think there is a lot to be learned in the industry around all ofthis, and these conversations really help move the needle. So thank you againfor being so generous with your time and also your advice today.

Erin Crum (55:45):

Glad to be here. Thanks for having me.

Evan Wong (55:46):

Awesome. All right, Erin. I'll see you soon enough, I'msure, and I do want those tips on how to raise my little plant baby. So we'llhave a separate session on that together. Alright, have a lovely rest of yourday. And for everyone else, thanks for tuning in.

(56:04):

Thank you so much for tuning in to this episode of Go Withthe Workflow. If you found a valuable, you can subscribe to the show on yourfavourite podcast app like Spotify or Apple Podcast. Also, please considergiving us a rating or leave a review as that really helps other listeners findthe podcast too. That's all for now, so we'll see you at the next episode.

‍