SSH is Dead. Long Live SSH: One Million SSH Logins with Okta. Zero SSH Keys.

As the great Mark Twain once wrote in response to reading his own obituary in May of 1897, “reports of my death have been greatly exaggerated.” Fast forward nearly a hundred years to 1995, and a Finnish computer scientist named Tatu Ylönen created a secure transport protocol known simply as Secure Shell (SSH). What do these things have to do with each other? Nothing, aside from perception.

In its most practical terms, SSH enables users to establish a secure, remote connection with a Linux-based machine via a Command Line Interface (CLI). SSH is the de facto standard for secure server access, and has survived the test of time, despite a significant shift in how infrastructure is operated in the cloud.

At Okta, we embrace the shift towards the cloud operating model, where resources are dynamic and ephemeral, by adapting the underlying security properties of SSH to fit. Our customers use Okta Advanced Server Access (Okta ASA) to securely automate identity & access controls for their teams to use SSH safely. And we’ve just reached a significant adoption milestone by registering over 1 million SSH logins per month... and growing!

In this post, I’d like to talk through some of the trends surrounding the cloud operating model, the continued role SSH plays in securing remote access, and how Okta has helped our customers elegantly solve a nagging pain point.

With Great Power Comes Great Responsibility

As the de facto standard for remote access to Linux servers, SSH is naturally a common target for attackers attempting to infiltrate a company’s network. The transport protocol is inherently secure; however, the backing credential mechanism is prone to human error—with potentially catastrophic results. SSH Keys, which are cryptographic key pairs designed to be an attestation of trust, require great care to ensure they don’t fall into the wrong hands. There is no way to guarantee a link between an SSH Key and an identity, so in many ways, possession is 10-tenths of the law.

As such, companies have traditionally been forced to do one of the following:

• Implement a security policy for users to regularly manage and rotate their personal SSH Keys (easiest, least secure).
• Operate a secure vault service that can store SSH Keys, checked out on demand (harder, more secure).
• Purchase and deploy a privileged access management product to act as a gateway for remote access (hardest, most secure).

I imagine you can see the pattern here—as with so many things, you get out what you put into it. But look, security is changing, and fast. First of all, the hardest solution isn’t always the most secure, but more importantly, who has time for all that extra hard work??!! We’re all forced to do more with less, but security teams also have the added challenge of protecting the company without getting in the way of the business. Not an easy thing to be on the hook for by any means.

We understand this continued balancing act, and design our products to be the most secure and easiest to use solution on the market. Before we get to the details of how we enable our customers to achieve a better outcome with Okta ASA, I’d like to address the perception that SSH is dead in the modern cloud era.

The Kubernetes Challenge

With the shift towards the cloud operating model, Kubernetes has emerged as everyone’s favorite configuration management plane/orchestration layer/CLI/runtime/service mesh/dev environment platform. Here’s a phrase I sometimes hear, “we’re all in on Kubernetes, we won’t need SSH access anymore.”

That sure sounds like a dream: a unified abstraction layer that removes all human interaction from managing elastic, multi-cloud infrastructure. Kubernetes is certainly poised to become the proverbial operating system of the cloud, but it does introduce a whole new set of complexities and considerations. Here’s a small sampling of scenarios where human interaction is still required:

• Logging into underlying hosts to run diagnostics scripts, inspect logs, debug network issues, etc.
• Writing and debugging YAML code to describe and configure target environments, services, runtimes, etc.
• Monitoring performance of networks, infrastructure, services, etc.

Standing strong and tall amidst all this innovation and change is SSH, because you just can’t replace that sturdy, direct connection between a user and a host operating system. Even the most automated of environments needs human access in case of emergency, but there’s more to it than that—automation needs security controls too. SSH still fits the bill for both human and service-level access.

The mental picture we all likely have in our heads here is a systems administrator sitting at their desk, connecting to a single Linux server from their terminal by typing, “ssh <hostname>”. While the direct user-to-server login is a common use case, the use of SSH is also prevalent in:

• Developers writing and running scripts that connect to many servers at once to run a series of diagnostic tests
• Configuration Management software (Terraform, Chef, Puppet, Ansible, etc.) that connect to target hosts to make local changes
• CI/CD automation tools that connect to production servers to configure runtimes and push software builds

Who Holds the Keys?

Let’s revisit the problem with SSH Keys. The traditional PKI (Public Key Infrastructure) that backs SSH was built for a different time, where a key exchange meant enough to grant trust. The core problem lies with the false assumption that ownership of a private key equates to an identity profile. While we may say “Alice’s private key”, there’s no associated authentication process that could verify it was Alice, who generated the key pair to begin with, or that Alice is the only person who currently possesses the private key.

As these keys get issued and distributed across fleets of dynamic infrastructure and cloud services, the challenge only compounds. Each public key accrues more privileges over time; the longer it’s been alive, the more likely it’s been shared with a resource, making it extremely difficult to track and subsequently revoke. With this model, time is a bug that can’t be fixed.

In a Zero Trust world, a meaningful trust attestation is correctly adhering to a policy that states a person (or service) from a known device can access a specific resource at a given time. It’s not necessarily about the credential itself, it’s about the context surrounding the request. Even so, we still need a way to represent the identity and permissions associated with the attestation that was made.

Enter SSH Client Certificates

Thankfully, the core OpenSSH technologies have evolved over time, and in 2011 introduced the capability to authenticate using a Client Certificate as opposed to an SSH Key. Unlike standard SSH Keys, which carry no metadata, Client Certificates are cryptographically verifiable metadata objects that can inject things like usernames, roles, and more. But arguably most important, they have an expiration date.

With Client Certificates, SSH access can follow an Identity-centric authentication and authorization workflow, where a credential that is tightly scoped to match a contextual access decision is minted on-demand. A single request, a single credential—nice and clean. With a tight expiration time, there’s no cleanup required, removing the burden associated with tracking, managing, and rotating credentials as previously discussed. Time now becomes a feature.

Now there’s still no such thing as a free lunch; it takes a good amount of work and continued operational responsibilities to build and maintain a Public Key Infrastructure (PKI) system that can generate these Client Certificates upon each valid request. This is where Okta comes into the picture—just as we know you shouldn’t roll your own auth, you shouldn’t roll your own PKI.

How We Manage SSH Access with Okta ASA

Since the introduction of Okta ASA, more of our customers have come to the realization that server access is an identity problem first and foremost. Disparate access management systems with disparate controls lead to unwanted complexity and unmanageable surface area. The more you can unify access with identity, the better, and when you have the market leading platform to begin with, it becomes natural to extend access to more identity-centric resources like Linux servers.

With Okta ASA, SSH logins now follow a familiar Single Sign-On experience, and authenticated and authorized requests are minted with what we call “ephemeral credentials” behind the scenes. To the user, it’s just SSH. To the security team, it’s a whole world of pain, lifted.

Okta Advanced Server Access end user experience

Users install a lightweight Client Application on their workstations that interface with their local SSH tooling. To run, the Client Application does require administrator privileges on the machine. Servers enrolled with Okta run a lightweight agent that performs local account and policy management, and captures login events. To enable Client Certificate authentication, OpenSSH is configured locally to trust certificates signed by Okta as a valid authentication mechanism—this is an important detail we’ll cover when we discuss the backing PKI system.

When a user attempts to login to a server, they are authenticated via Okta, and subject to the customer’s configured sign-on policies. This could include multifactor authentication and/or device trust. Once authenticated, they are authorized based on the role-based access controls (RBAC) of the target server. If all checks out, Okta ASA mints a short-lived Client Certificate, scoped to the request. This Client Certificate is delivered to the Client Application, which uses it in memory to establish a direct, secure connection with the target server.

What makes this so revolutionary is that we are able to abstract so much PKI complexity behind the elegance of our SaaS, exposing a familiar and seamless Single Sign-On experience. So what goes on behind the scenes that is so complex?

Dynamic Client Certificate Architecture

It takes a lot to operate a highly available, resilient, and secure SaaS platform. When you add PKI to the mix, you have quite the beast on your hands, which is why we would never recommend building an equivalent system yourself. The backing Client Certificate architecture of Okta ASA originated from ScaleFT, which was acquired by Okta in July of 2018.

What makes our design unique is that we don’t operate a Certificate Authority (CA) in the traditional sense (think web PKI where the chain of trust is critically important because the certificates are long-lived). We treat our CA architecture as dynamic as our customer’s cloud infrastructure environments, and spin up services that are capable of minting ephemeral Client Certificates on-demand for specific server groups on each independent SSH login.

Okta ASA CA Architecture

The primary goal of this CA architecture is to minimize the scope of each Client Certificate as much as technically possible. As signed credentials, the cryptographic properties of each are, of course, still important. We leverage AWS KMS as the root signing service, taking advantage of their underlying security properties and highly available design. Within Okta ASA, we operate encryption services that perform a number of tasks based on various API events.

The events that are most relevant here are Project Create and SSH Connection:

• When a new Okta ASA Project is created, we create a SHA-256 encrypted signing key. When servers are enrolled with the respective ASA Project, the local OpenSSH is configured to trust Client Certificates signed by that key. As Projects represent the authorization scope of role-based access (RBAC) to servers, having a dedicated signing key also limits the credentials to match.
• When an SSH login is authenticated and authorized to a user on a device at a point-in-time, we mint a new Client Certificate with injected user and device metadata, and a short expiration time intended to limit its validity. It’s that Client Certificate that the user’s Client Application uses to establish a secure SSH connection with the target server per the end user experience workflow. Because of the short expiration time, the Client Certificate does not need to be cleaned up after it’s used.

So much time and money has been spent by companies and vendors to mitigate the risk of credential sprawl. We tackle this problem at its core. The best way to mitigate the risk of credentials is to make them useless!

One Million SSH Logins. Zero SSH Keys.

Now back to that grand milestone. The Server Agent that is installed on each target server captures every login event attributable to a user and a device, which provides a much stronger audit event than what you might get from using shared accounts or shared credentials.

As more Okta customers have adopted Advanced Server Access to secure their server fleets across AWS, GCP, Azure, or on-premises, we’ve seen a continued uptick in server enrollment and SSH login events. And we’ve reached the milestone of 1 million SSH logins in a month, without the use of SSH keys, significantly mitigating the risk of credential sprawl.

Total monthly # of successful SSH logins via Okta ASA

SSH is Here to Stay

While his obituary may have been published in 1897, Mark Twain actually passed away in April of 1910, after completing two more novels and multiple short stories. Similarly, we can expect SSH to continue to stand strong and adapt to changing use cases. Just as in literature, with security, you just can’t beat the tried and true.

To learn more about how Okta does SSH right (and makes SSH fun again), visit the Okta Advanced Server Access overview, or request a demo.