No Perimeter, No VPN, No Problem

Mark Loveless: Oh, hello there. My name is Mark Loveless. I am a senior security researcher with GitLab, and I'm going to be presenting today. I'm going to be talking a lot about Zero Trust. The talk is entitled, No Perimeter, No VPN, No Problem, and that will become, hopefully, readily apparent what we're talking about here, really, really quick. For the agenda, I'll talk a little bit about GitLab itself and then I will also get into GitLab's Zero Trust methodology. And then from there, we'll move on to how we actually implemented Okta at GitLab and then from there also cover a few tips and tricks that we picked up along the way, some success stories, if you will.

Mark Loveless: So, first off, about GitLab. We are a web-based, DevOps, life-cycle tool, and what that means is that when people are writing code, they need to be able to manage code and, as a SaaS company, we provide a way to do that. We also have an on-prem product as well, if someone is so inclined. But for the most part, this is what we have. We eat our own dog food. We actually use this ourselves to put out the GitLab product. We are an extremely fast-growing global company and that's going to become readily apparent when I get into some of the details as we move forward.

Mark Loveless: We are fairly unique as a company goes. It may sound cool, but as a member of the security team, all I can say is that it is anything but cool. It is challenging. I mean, I do have fun at my job, don't get me wrong. It's very challenging, very rewarding, but it is certainly not an easy task to be able to pull this off. So let's go ahead and move forward here, talk a little bit about Zero Trust. I think everyone's pretty much familiar with Zero Trust and whatnot. And I wanted to talk about how that kind of plays into our decisions in some of the vendors that we've chosen to work with over time, including at GitLab, or including Okta.

Mark Loveless: Okay, so traditionally, what you end up having with most companies is the solution for doing Zero Trust, they talk about removing the perimeter. We're going to, because your perimeter's diminishing, you're getting, you know, a lot more emphasis on not having a perimeter, per se. Everyone talks about dismantling the VPN. You're not going to need that VPN anymore and most companies, they just have a few remote workers and so a lot of the vendors that we've talked with, they say, "Now you may have a few remote workers. We need to make sure we are able to transition and handle them."

Mark Loveless: And most companies, at this point, there's a mix of on prem and cloud. There are a few companies out there that are cloud only, but most of them have some semblance of on prem still. And asset management is just assumed. It's just naturally assumed. "Oh well, you guys probably already have all of your employees' computers all taken care of and everything." That's them. For us, there's a very harsh reality, and that is that we have no perimeter. We've never had a perimeter, ever. So therefore, we've never had a corporate VPN. There's been no need for one. Everything has been just without the perimeter, working in the cloud.

Mark Loveless: We are all remote. We are the largest all-remote company in the world, just nothing but remote workers. That takes people aback, particularly if they're, you know, a vendor trying to pitch a solution of, "Hey, we're going to help out and we might be able to take care of those remote workers." Well, you know, we're nothing but remote, and like I mentioned, we're all cloud, so we're completely cloud-based with all the various applications and stuff we use. And, believe it or not, we have limited asset management. We have a pretty good handle on our infrastructure, as far as what we have running in the cloud.

Mark Loveless: But, as far as GitLab and GitLab.com, for example, what we have there, where we store our customers' data, we have a pretty good handle on that, but as far as users' laptops and whatnot, that's a completely different story. That's a little bit different. We're working on it, but we're not there yet. There are some other issues and these involve culture. We're open core with our product and what that means is that the core of the GitLab product is an open-source product. And so that means that anybody out there can contribute code to it and whatnot. It is completely open source.

Mark Loveless: Now we do, as a company, for added features and stuff like that, we do have things that you pay for and get with the product, but the core of it is open source and so we refer to that as open core. We are more than open core with our culture as a company. Like I said, we allow people from outside the company to, if they find a bug, they can contribute and say, "Hey, I found a bug. I want to fix it. Here you go. Here's a code fix for you." So we have people checking code from all over the place. We've kind of adapted this to everything, so all of our documentation is also very open source.

Mark Loveless: Our corporate manual, for example, is up and online for anyone and everyone to see and to add to it. If there's someone who doesn't even work here that says, "Hey, you should've included this." They can submit a manual request and we'll go through the whole process and see about getting it approved. It's happened. It'll happen again. And that's what I mean by, you know, more inclusive than just our employees. It just, it does come down to everyone we encounter out there.

Mark Loveless: We also have some corporate issues we have to keep in mind when we’ve been kind of looking at this whole Zero-Trust thing. We've got a pending IPO. As an open company, they announced several years ago, "Hey, in November, 2020, we're going to do an IPO." And that's documented in our manual, and that is the direction we are still, at this time, heading. And so with that pending, there's like a lot of things that come up as a result: compliance issues, there are auditing issues, as well. These are huge issues for us to overcome, particularly because of the amount of growth that we've had as a company.

Mark Loveless: The growth has just been absolutely insane. We've grown in revenue, we've grown in employees, everything. And we're, as a result of growing employees and being all remote, this complicates the compliance issues, since we're in other jurisdictions. We're all over the planet. On top of that, as a company, we're going to have the normal security objectives that are needed to, just to make things happen, to make things work as a corporation. So those are in play.

Mark Loveless: Essentially, we're a bleeding-edge company. And bleeding edge, again, it's just like, "Oh, you're very unique. You've got this bleeding-edge thing going on." And so, as a result, with this whole bleeding edge, it's not like we can go somewhere and ask someone, "Hey, can you help me with this? What did you guys do when you encountered this issue or this situation with your company?" We end up finding out that the answer to that is, "You're doing what? I had no idea." And so you get this real interesting world you end up living in as a result of that.

Mark Loveless: I’ll talk about a few pain points here. Now, classifying our data is, we do have a process for it, but the thing that's interesting is that our data classification is fluid. We have four different levels of data classification, green being public, and then there's yellow, then orange, and then red, red being the most sensitive data that we might have. So some really sensitive employee data that's pertinent to HR and whatnot, or customer data that, you know, we want to protect customer data at all costs. And so there are times when, from the time when one logs in in the morning until the time when they log out, that the classification of that data can change.

Mark Loveless: And so there's this fluidity associated with that data classification, where you can say, "Well, okay, in the morning when I logged in, data was green and then later in the day it became red, and now it's split and it's like part of it is orange and part of it is now green again." And this happens a lot within the company and that's just the nature of us trying to map out our data to make sure we get classification going. This is something that we've encountered. Provisioning and deprovisioning of users is hard, simply because when you have a ton of SaaS applications out there and you're trying to create accounts for all these users, it becomes a real chore to track down who's responsible for what SaaS application.

Mark Loveless: And while they're not doing their regular job, can you step back and create this new account, et cetera, et cetera. So that's kind of a mess. And like I said, we just barely have end-user asset management. To give you an idea, at this point when I'm recording this, I think they started buying employee laptops maybe two-and-a-half years ago, and before that it was BYOD. And so we have employees that are here at the company now that are still using a BYOD laptop. That's part of that culture, that's where that comes from. So just even buying our employees laptops to use is a somewhat new concept for this company. So it's interesting, very interesting.

Mark Loveless: Now Zero Trust in itself, it sounds awesome when you hear the elevator pitch on Zero Trust. But there doesn't seem to be any agreement on the various vendors as to what Zero Trust actually is, and everyone keeps referring to Google's BeyondCorp model, and that's fine. They can go ahead and refer to that, but that doesn't necessarily mean that's what's going to work for us. We like the idea of it, but the BeyondCorp model was intended for somebody else, not for us. No vendor has a complete solution. That's the one thing right there that we immediately noticed, because we had needs in a whole bunch of different areas and there's just not a single vendor that covers every single instance of stuff that we need. As I get into it here, you'll see what I'm talking about.

Mark Loveless: Basically what we've encountered a lot is, we've seen a vendor that isn't Google and they're trying to sell an incomplete version of an internal Google solution to GitLab and that's just not going to cut it. That's just not going to work. For our definition of Zero Trust, I'll go through this one at a time. User identification: we want to ensure that the user is who they say they are. That's number one there. Number two, we want to ensure the device that they are authenticating with, that we recognize that device and we know that that device is approved to contact GitLab.

Mark Loveless: Data classification: we need to have all of our data classified so we can make decisions when it comes to whether we're going to allow access. So we're going to make sure that, you know, we have data classification, we have access-controlled systems where we can actually go through and say, "Okay, these systems have these controls in place and we can do this and that via policies to define whether they actually gain access to that data.

Mark Loveless: There is a whole grab bag of things that are kind of extra that I wanted to talk about. We have a need to make sure that all data that's in transit is encrypted. That's a big thing for us. We want to have very full, very robust logging, because we want to collect all this logging data from all these different systems, not just from the vendor solutions, but our own systems, and be able to pull these together into one cohesive log that we can look at.

Mark Loveless: We need to make sure that we also have support for a CDN front end, so that we're able to make sure that when we have CDN out there and they have all these different parameters and whatnot associated with their communication with us, that it actually is going to work. And also, while some of these things are not necessarily Zero Trust related, if we're going to come up with a solution for some of these things, we've got to make sure that the solution doesn't eliminate our ability to do something Zero Trust later on. And so that's kind of an important point there.

Mark Loveless: Our rough plan is to not focus so much on Zero Trust networking as a whole, as much as, just say, if it's Zero Trust-ish, then that's probably good enough for us to consider. So if something's kind of in the realm of Zero Trust where we can kind of make it work out mentally, then we'll go ahead and go for that. We have to make sure that we're going to reduce everything to basics. All these different things that we have out there, if we reduce these down to basics, it's going to be easier.

Mark Loveless: There'll be chunks that we can actually deal with and then we just basically go through and then solve each one. So every little chunk of thing that we have, we just go through and just solve each one of those one at a time. That also allows us to do things like find budget for some of these individual things, and then we have to make sure that everything all still talks to each other, that everything's all still tied together. So that's pretty important, as well.

Mark Loveless: All right, so now we're getting to why we ended up going with Okta, with the relationship there, how that came about. There were a few reasons why we chose Okta.  I forget what we had at the time when we first approached Okta. It was something like 70, 80 SaaS applications. It's over 100 now. But that was a kind of thing because just even cataloging what we had was a problem. And the other thing, too, is that we needed something to be able to support different identity methods so that we could actually talk to these different systems with whatever identity solution we had. That was pretty important to us.

Mark Loveless: Like I said, the growth rate has been absolutely tremendous. So we were, I think, around the time we started this, we were around 350 employees, and we are over 1,200 and still growing. So this, in the past 12 months, that's basically what it's looked like. It's just been absolutely crazy. As a result, onboarding and offboarding has been an absolute massive time sink, just getting accounts built and making sure you find all these accounts when they leave. You don't want to end up saying, "Hey, why is Fred's account on this system? Who's Fred?" And someone says, "Oh, he worked here a year ago."

Mark Loveless: You don't want that scenario to crop up, particularly if Fred had administrative access to something. Just the growth rate there is incredible. And, like I said earlier, we're leading up to an IPO. With all this growth and expansion into new areas, it has really put a huge burden upon making sure that, you know, for compliance and various audit requirements we have going forward. So SOC 2, PCI, ISO, there are things that we have to make sure we're ready for, and we need some semblance of control over our environment and ability to kind of prove our environment, as it were.

Mark Loveless: So how do we do this? This was kind of an interesting thing here.  In our tradition of being an open company, so to speak, we did what we called an open beta. So we did have some, basically, there was staff that had vested interest in what we were doing, and we had vested interests in making sure that their department was covered. We just put together stuff in beta and basically, anyone could, you know, we put it out to the whole company, just said, “Anyone can join up for this and give us your opinion. We welcome your feedback.” 

Mark Loveless: And I think the entire security team, and there were some other parts of engineering that also went ahead and opted in to participate in the open beta. We were able to, when you had those people opt in for it voluntarily, they had concerns and they were able to bring a lot of those up. Being able to present to them, just to give you an example of this different multi-factor options that played a big part, because we had people that just said, "What is this Okta Verify app? I don't want to load an app on my phone. This is my personal phone. I don't want work apps on it." And there'd be a lot of passion against doing that.

Mark Loveless: It's like, "Okay, well, that's fine. How do you feel about using U2F or just this YubiKey and using that?" And we also allowed for TOTP. If they're using, say, Google Authenticator or something like that on their phone, then we gave them another option where they could do that. And having a product that supported all of that really played a big part in addressing a lot of the concerns that people had. And that really helped with the beta. Then we did in May, June of last year, we started our live deployment.

Mark Loveless: That was interesting, too. We did start with what we refer to as non- critical apps, but we also included GitLab.com itself. So that made for an interesting challenge right there. So that was good. And we did learn some of the common pitfalls and whatnot. Not only were we building up trust with the users, but we were building up our own trust in the product itself. It's just like, "Okay, this is really working. I think this is really going to work well." And that led us to the next step, which was July through October, which was critical app deployment.

Mark Loveless: And these were the apps that were, you know, absolutely had, there were some critical things associated with those. To get through that, there was a lot of overcommunication where we had a dedicated Slack channel, just for people that had concerns about stuff. We actually had that during the open beta, as well, come to think of it. We did allow for a lot of ways to get feedback from it, but just the main thing was  constant communication: “This app here is now moving into Okta. This app over there is now moving into Okta.” And kind of moving forward. And just really, we had a lot of different scenarios as a result of that. We had to test ... I'll talk about that here in just a second.

Mark Loveless: Like I said, our key challenges, and the obstacles, it was getting the users to adopt and change over to this. Even though we're the security department, coming in and saying, "Hey, the security team wants you to do this security stuff because of security," that wasn't a good way to get users to just jump on board. What we were doing, we were saying, "There is a user experience here that is going to be easier for you, and you're just going to have to sign in once, and then you have this dashboard and you can click on whatever apps you need to go to." And that  really kind of helped by just saying, "Look at how we're going to improve things for you."

Mark Loveless: So that made a huge difference right there. Another thing that was kind of a hard thing to deal with was, a lot of this was production-first implementation. And by that I mean, just like you have a SaaS company, they have an application that you're using. They may not have a full and working dev environment or test environment. You just go with it. That's it. You don't have much of a choice. You either implement it or you don't. So for some of this, it was kind of, “Hold your breath, we think this is going to work.” And then we would just go for it. And with some applications, we had challenges where it was an all-or-nothing. It was either all Okta or no Okta.

Mark Loveless: And others, we could do Okta, but we could also configure it to where we can, you know, not have Okta and kind of stage through the whole process, and that kind of helped right there. But, still, when you're doing these cutovers, a lot of these SaaS providers, they weren't used to this kind of thing. And so they wouldn't necessarily have support on speed dial to say, "Hey, we’ve got to reconfigure this entire SaaS instance here and make it work. And in the meantime, we can't use it and this was critical to the company."

Mark Loveless: That part was pretty challenging, but we did get through it. SaaS application debt, now basically what I'm talking about here is that you had application owners that didn't understand how their SaaS application worked. So we may have someone, let's say, for example, for some SaaS application that was used by sales. They're experts in sales. They're not experts in IT. Even if they do have plenty of experience with IT and IT issues, they may not be as well-versed with security elements of that, in Infosec. When you go to someone and say, "Hey, this SaaS application that you're in charge of, does it handle SAML?"

Mark Loveless: And they look at you and say, "Who's SAML? We don't know who this guy is." Then you realize, "Okay, well, we’ve got a little bit of a thing here." That's what I'm talking about with the SaaS application debt, is that we found ourselves going in and doing this, the just cutting edge, bleeding-edge kind of approach to this stuff. We uncovered a lot of things that we didn't expect to uncover. So, going through the process, when that kind of thing happens, you just deal with it. But that was quite a thing just to have to be able to take that on.

Mark Loveless: There were also some new help-desk challenges that came out of this, as well. ‘I changed my phone’ became the new password reset. That was the big one, particularly, you know, because either they’ve got their authenticator out for TOTP, or they're using Okta Verify, something like that, and it became kind of complicated. There were some new processes and procedures that we had to put together for the help desk. That was interesting.

Mark Loveless: Now on to some success stories. Getting our text stack actually cataloged was great and we had kind of a rule that we're now able to enforce. We would say, "If it's not in Okta, it doesn't get approved." So it's got to be able to either be in Okta, talk to Okta, interact with Okta, and before it's going to get approved  out there in it. And I think the tipping point really came for this, we started having app owners come to us and they would be just like, "Hey, we're running this SaaS application, you know, X, Y, Z." And we're like, "We've never heard of that. We don't even know that that app existed, let alone that we own it and we're using it as a company.”

Mark Loveless: They would come to us because they wanted it in the dashboard and they also heard from their co-workers that, "Hey, just if you're running an app, you put it in Okta and then you don't have to create all the accounts anymore." We had people hunting us down. So we realized, okay, this is documented enough that we're actually making some real progress. And so that kind of helps.

Mark Loveless: Onboarding and offboarding. Now, when we onboard people, when you onboard people into any company, you usually have this huge template, this list of items that you're going to go through, that are going to just, you know, make sure they're signed up for benefits, make sure that their payroll stuff is in place, and all that. There's this huge list of things. Well, a lot of it is creating accounts in various systems, so there's this base level of accounts that need to be created. And then also, based upon their particular job description, there may be other SaaS applications that they need access to.

Mark Loveless: Someone in accounting may need something different than, say, someone in sales or someone in engineering. So that onboarding template, we were able to reduce that by over a third, simply by going with Okta, because all of a sudden ... To give you an example: when I first started, I think it was about three weeks, typically, for a new user (this is like in February of 2019 before we started doing the Okta stuff), it could be three weeks before I’d get access to all the different systems that I need to get access to, and that was ridiculous.

Mark Loveless: But the thing is, it just took time. That portion of it, creating all those accounts, because well, right now we use BambooHR for our HR product. A new user is created in BambooHR, and then that is immediately exported into Okta, and then from Okta that is immediately, well, we're creating accounts in all these other systems. And that's pretty much all automated at this point. So, there are some exceptions because there are a few SaaS applications that aren't perfect and they don't follow the exact rules we'd like them to follow and whatnot. 

Mark Loveless: It took it down from three weeks down to just, you know, a minute. And it was just insane, so it really reduced the onboarding stuff. The other thing with that is that it allowed for nearly real-time offboarding with automation to be able to ... Because if we can create all those accounts, we can also disable them at the same time, when it came time for offboarding, so that was absolutely phenomenal to be able to do that. So that was a huge success story for us.

Mark Loveless: The whole application legacy, that situation. This provisioning thing, it just allowed 3 to 10% of human error accounts that were not deprovisioned to be actually caught. So we're talking about, you know, Fred having that account from a year ago. You're going through this and you're in the process, you can see where the mistakes happen, see where they occurred, and we're catching some of these human-error things as a result. And so this helps with security. It certainly helps with an audit, that kind of thing where they're looking to see what accounts you have there, and to know that they're right and those people are supposed to be there.

Mark Loveless: It's phenomenal. It's a wonderful thing to be able to do. It's also allowed us to eliminate a lot of shared accounts. That's always something that plagues companies, is shared accounts. And so, that's fantastic. We're just grateful that we can be able to just put in a policy, a group within a policy, and these people have access to do this. So you can actually see who's performing that action or whatnot via some specialized account. So that's wonderful.

Mark Loveless: The compliance controls, that's all been dramatically simplified, and that's really improved the speed on some of the things. Some of the verification and stuff that we had to go through, with auditing, that went from weeks to hours. And that's just phenomenal, simply because you can get to everything and be able to export stuff out and say, "Hey, look. Here's this, here's that." It's wonderful.

Mark Loveless: The other thing that helped with compliance is just being able to have password standardization, and then just have these policies that just simplify the entire compliance process for us. Because they're just across the board. They just worked everywhere. As a company, since we're global, we're all over the place. We're in something like, I think, 65, 66 countries at this point. And so you have different compliance issues that crop up in all these different locations.

Mark Loveless: And so what happens is, you can actually say, "Well, okay. This one is strict up to this point here, and this one over here says, 'Well, on a scale of one to ten, you’ve got to be at least a six or a seven or something like that.'" Anyway, you can just say, "Let's just go to the highest standard that we need to cover each thing, and then just push that out everywhere.” This helps with that, and you have to end up doing that to just make sure you don't have different standards for different people based upon where they're logging in from or where they reside, that kind of thing. This just really cuts down on a lot of that.

Mark Loveless: Another one, this is Okta for ASA. This was really interesting, because what we had was, the infrastructure seemed to have some projects coming up that were going to involve a lot of creation of accounts, and they knew that some of the people that were going to have to be creating these accounts, they were going to have to be pulled off their regular job and have to do this. So, the infrastructure team says, "Hey, we’ve been using this Okta thing that gets everything else. Do you have anything that we could do with SSH?"

Mark Loveless: And it's just like, "Well, guess what? We've got it." We know what we can do and so we started looking at Okta for ASA. We had a planned pilot, I think it was for maybe for a month. And in less than half the time of the pilot, by then everyone was just like, "Oh, this'll work. We're fine." And so we just went ahead and implemented it. Just a real short pilot than what we were thinking we were going to have to do. So it was just a wonderful success to , you know, wham-o, just create accounts like that.

Mark Loveless: And that helped with everything else that we've been talking about, compliance and whatnot, just keeping track of accounts. And this has been fully adopted by the infrastructure team, and so they're in the process of making sure all their people and stuff are on it. And then it's going to get deployed to the rest of engineering at some point. We're looking forward to that because we're big SSH users here.

Mark Loveless: There are a few tips and tricks I wanted to go over with this, too. One of the things that we learned when doing this, is that we need to, we always try to focus on the benefits that you get with this. We tried to sell the user experience. We make the whole thing as transparent as possible. And like I said, we exposed the efficiency thing. The efficiency things usually kind of expose themselves. But, those are thick, you know, become bonuses like, "Oh and by the way, after you do this, it shortens this process here and there, and your productivity increases.” People usually really go for that.

Mark Loveless: And later on, we expose the security benefits, just like we're going to meet compliance and all kinds of stuff. So that's really good. What we try to really do there, outside of selling the user experience, we just want to make it as transparent as we can. We were very upfront and open with how everything worked, what we were trying to do, what we were trying to accomplish. But we really did focus in, though, on the user experiences. And like I said, exposure after the fact is  some of the other benefits, and probably they're our main reasons for doing it.

Mark Loveless: The open beta and the overcommunication things, I can't stress that enough. That was really important and really good for us in getting buy-in from people. So just actually going through and engaging these people and then bringing them in really helped considerably as they went through it. At that point, we were able to actually find business champions, people within the business that just said, "Hey, this is wonderful. This is awesome, you guys. We've got to do this." And they would be communicating it in their departments and saying, "This is the way to go."

Mark Loveless: Like I said, with what happened with Okta for ASA, that whole thing, that was just like a business champion, so to speak. Just came forward and said, "Hey, this is going to work. This is great." We try to celebrate and kind of capitalize on the successes that we have, and just keep the momentum up. At that point, what happens is that the staff expects us to have everything in Okta. If it's not there, they begin to complain and say, "Hey, we want this in Okta. We want that in Okta. It's going to make my life easier."

Mark Loveless: By just pointing out, "Hey, look at what we've done so far," and kind of keeping that up, that's really helped us get this thing fully implemented within the company. Basically, as kind of a conclusion to all this, I just wanted to go over and say that it always helps if you're going to come up with these new ideas that are different and unique for people, it always works better when you have end-user input. That's always a big help to have.

Mark Loveless: Another thing is, we have been trying to do this all along in the security department, which is to just leave things better for the end users than when we actually started. And that actually benefits everybody, okay? Because if it's better for the end users and you've implemented something that you've needed or wanted in there, it's just win-win. And this is something that we really have discovered as well, is that if a solution is very, very hard, if it was extremely complex, we probably should do something else.

Mark Loveless: That's really important, and that has to do with Zero Trust, as well. A Zero-Trust solution, it does need to make things easier, and if it doesn't, then you look for a better solution. For us, this has been great. This has really worked out well for us, and we've been very happy with the choices we've made, and how things are kind of moving forward at this point. And that is pretty much the end of my presentation. I've got my contact information there in this slide, my email address, if you have questions you want to ask.

Mark Loveless: And, certainly, there's a whole series of blog posts  where I go into some of the detail on some of these things, such as, for example, data classification. I do a whole blog post on some of the issues we had with data classification. Just go explore those and hopefully that will help you, as well. But, if you have questions at this point, we'd just love to actually hear what they are and we'll try and get them answered.