By Dhruv Pratap
Introduction
Organizations which were round for some time often run two identification programs facet by facet. One belongs to the cloud supplier: IAM roles, occasion profiles, execution roles. The opposite is your personal, and it’s the one your inside companies really test once they resolve whether or not to reply a request.
On infrastructure you construct your self, you’ll be able to bootstrap your personal identification nevertheless you want. On managed compute you can’t. The supplier arms your course of a cloud identification and nothing else.
This submit describes how we shut that hole for Apache Spark workloads operating on Amazon EMR. A workload that begins with solely an AWS identification has to finish up holding a first-class inside identification. Doing the change is simple. Making it reliable is the half that took the design work. Little or no of what follows is restricted to Spark or to EMR.
Two identification programs
Inside service-to-service authentication at Netflix runs on a personal PKI known as Metatron. Each workload will get a short-lived X.509 certificates, and companies authenticate one another with mutual TLS. A workload can’t merely ask for a certificates. It’s issued solely after the identification service has happy itself that the requester actually is the workload it claims to be. That verification step known as attestation. In every atmosphere we assist (VMs, containers, features) attestation rests on some environment-specific proof that the platform can test on its personal.
The second idea is the Knowledge Mission, which is our unit of possession for information. A Knowledge Mission owns tables, has a set of licensed customers, and has an identification of its personal. Jobs run because the Knowledge Mission fairly than as the one that launched them. That’s what retains table-level entry management and audit constant throughout scheduled and interactive use.
So the issue is that this. A Spark job on managed compute begins with an AWS execution position and no inside identification. Every part it wants from inside the corporate (studying an encrypted column, evaluating desk ACLs, holding an interactive session open longer than a short-lived token lasts) requires the inner one.
One identification, one position
The design rests on a call made outdoors the Spark stack completely. Every Knowledge Mission identification maps 1:1 to a devoted IAM position, and the service that owns Knowledge Mission metadata information that mapping.
The mapping is what makes translation attainable. It lets a press release within the supplier’s vocabulary (“this course of is operating as position R”) turn into a press release in ours (“this course of is workload W”). With out it there may be nothing to translate into, and no quantity of cryptography helps. The exhausting a part of bridging two identification programs isn’t the protocol. It’s committing to a mapping after which retaining it authoritative.
One sensible objection arrives instantly. A single AWS account can’t maintain tens of 1000’s of IAM roles, and we anticipate information tasks within the order of ten 1000’s. We dealt with this by sharding information mission roles deterministically throughout a small pool of devoted accounts, which scales properly previous the projected variety of tasks and has the helpful facet impact of retaining workload roles on the far facet of an account boundary from the management planes that launch them.
Parts
5 parts take half.
The management aircraft is the one service allowed to launch Spark jobs. It resolves the Knowledge Mission’s IAM position, builds and indicators a workload metadata payload, and submits the job with that position because the execution position. It holds a signing key issued to it for this goal alone.
The Knowledge Mission service is the authority for the identity-to-role mapping.
The Identification service receives attestation requests, verifies them, and points certificates.
A Spark plugin supplies the hook. Spark 3.0+ added a plugin interface with driver-side and executor-side parts which can be initialized whereas these processes are bootstrapping.
AWS STS acts as a notary, which isn’t its normal job.
The first step: the management aircraft makes a signed declare
When a job is submitted, the management aircraft validates the caller, appears up the Knowledge Mission’s IAM position, and builds a small payload describing the workload it’s about to launch. The payload names the applying identification, the mapped position, and the atmosphere, stack and element that the ensuing credentials ought to be scoped to. The management aircraft indicators the payload and passes it and the signature via as abnormal job configuration.
Two properties of this step matter. First, just one service can produce a sound signature, so the set of issues that may assert “this workload is respectable” is small and auditable. Second, the payload is a declare fairly than an authority. By itself it proves nothing, as a result of job configuration travels via infrastructure we don’t absolutely management and something that may learn it may possibly replay it.
Step two: the workload proves it holds the cloud identification
When the managed service launches the driving force course of, it does so beneath an OS consumer that has the execution position’s credentials out there. The motive force-side plugin initializes and makes use of these credentials to signal, however not ship, a request to the supplier’s identification endpoint (sts:GetCallerIdentity). The result’s a short-lived pre-signed URL.
That URL is a transferable proof of possession. Anybody can fetch it. Solely the holder of the position’s credentials may have produced it. And the response comes from AWS fairly than from the workload, stating which position signed the request. The workload can’t lie in regards to the reply as a result of the workload doesn’t provide the reply.
The sample isn’t new. It’s the similar thought behind AWS IAM authentication in HashiCorp Vault and in AWS’s personal perform attestation flows. It generalizes properly: any atmosphere that provides a course of cloud credentials and nothing else can nonetheless produce a verifiable assertion about its personal identification.
The plugin sends one attestation request carrying each artifacts, the pre-signed URL and the signed metadata.
Step three: corroboration
The identification service does 4 issues.
- It fetches the pre-signed URL in opposition to AWS STS and reads again the position that signed it. That is the supplier’s assertion.
- It verifies the metadata signature in opposition to the important thing issued to the management aircraft. That is the platform’s assertion.
- It corroborates the 2. The position AWS experiences have to be the position the management aircraft stated it dispatched, for the identification the metadata names.
- It points certificates for that identification, scoped to the atmosphere, stack and element from the signed metadata.
Step 3 is the purpose of the entire design. Neither assertion is enough by itself, and so they fail in complementary methods. The supplier’s assertion is unforgeable however under-specified: it offers you a task, and a task isn’t a workload. It says nothing about why this course of exists or what it ought to be allowed to be. The platform’s assertion is well-specified however unverifiable by itself, for the explanations above. Put collectively, the signature says the workload was legitimately dispatched and the pre-signed URL says this actually is it. Neither social gathering has to belief the workload’s personal account of itself.
There’s a stress right here that we settled intentionally, and different groups ought to anticipate to fulfill it. The supplier’s reply identifies a task, not an utility. Deriving an utility title from a task title is fragile string matching, and the session identifiers the platform assigns should not beneath our management. We selected to take the applying identification from the signed metadata and use the supplier’s reply just for corroboration. That prices a community spherical journey and a stricter signing requirement, and it buys a a lot clearer belief story.
The fan-out drawback
A Spark utility is one driver and as much as 1000’s of executors, created and destroyed all through the lifetime of a job. Attestation designed round one course of per host doesn’t survive that.
There are two defensible solutions.
Within the first, every executor attests independently. That is uniform, provides no new belief boundary, and grounds each course of’s identification in the identical provider-verified proof. The associated fee is amplification. One massive job can produce 1000’s of STS calls and 1000’s of attestation requests in a burst, which appears quite a bit like an assault and creates a tough dependency on provider-side charge limits.
Within the second, executors inherit credentials from the driving force. The motive force attests as soon as and distributes credentials to executors over Spark’s inside RPC, which we configure with authentication and AES-GCM encryption in order that solely processes in the identical utility can participate. Load on the identification service stays fixed. The associated fee is a second belief boundary and a driver that’s now a credential distribution level.
Contemplating our scale at Netflix right here we went with the second strategy. The final lesson is that identification amplification is a capability query, and it’s higher to reply it on goal than to have it answered for you throughout an incident.
Lifecycle
Certificates are short-lived by design, so bootstrapping is barely half the work. On self-managed hosts a system service handles renewal. Inside managed compute there isn’t a equal hook, so the driving force JVM runs a timer that re-attests on a hard and fast interval, properly forward of expiry. Credential materials is written to a short lived listing readable solely by the method proprietor and eliminated when the method exits.
Executors don’t refresh. They’re often short-lived, and an executor that by some means outlives its credentials can exit and get replaced, which is cheaper than making each executor a renewal consumer.
The rule value carrying into any related system is that attestation must be a repeatable operation fairly than a bootstrap step. Something that may solely occur as soon as at course of begin will finally be the explanation a long-running job dies 9 hours in.
Why this generalizes
Should you run your personal identification system and are transferring workloads onto managed compute, the precise companies right here matter lower than the form of the answer.
- Anchor on one exchangeable primitive. A single authoritative 1:1 mapping between your identification and a supplier identification is what makes translation attainable. The remaining is plumbing.
- Require two impartial claims and corroborate them. One from the supplier, unforgeable and under-specified. One out of your management aircraft, well-specified and replayable. Belief the intersection and by no means both alone.
- Maintain the signer scarce. If precisely one service could make the platform’s declare, your belief story suits in a sentence.
- Hook the layer you continue to personal. On managed compute you hardly ever management the host or its init system, however you nearly at all times management the runtime: a plugin, an agent, an entrypoint. Attestation belongs there.
- Determine the amplification coverage on goal. Distributed engines multiply each per-process operation by their parallelism.
- Make attestation repeatable and credentials short-lived. Renewal is a requirement, not a follow-up.
What makes the end result reliable is that no single participant points an identification by itself. Not the workload, not the management aircraft, not the supplier. And the workload, which is the social gathering within the weakest place to be trusted, isn’t requested to vouch for itself.
Acknowledgements
Due to Amer Hesson for the Knowledge Mission abstraction, Nick Siow for the Knowledge Mission IAM position sharding, and Doug Clark for Metatron attestation.
Buying and selling a Cloud Identification for Your Personal: Workload Attestation on Managed Compute was initially printed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.
