This post is by CoreOS principal security engineer, Matthew Garrett, known for his work in the open source security community. We wish him well in his next endeavor.

CoreOS was founded around a simple idea - the security of the internet can be improved by making it easier and faster to update software. Automated image based updates get patched versions of the OS into people’s hands seamlessly, and the combination of containers and features like Clair allows admins to prioritise updates to critical components without worrying about them breaking less security-sensitive ones while using container-based isolation to reduce the impact of vulnerabilities that do exist.

During my time here at CoreOS we’ve furthered that security by incorporating existing features such as SELinux and Seccomp that are typical around security-oriented Linux distributions. But we’ve also brought in features like dm-verity, making it impossible for attackers to modify the underlying filesystem containing the OS - features that have otherwise not been present in server products. We’ve become the first operating system on the planet to provide known-good TPM measurements for the operating system, and the first container runtime to support TPM-based measurement of individual containers to provide a tamper-proof audit trail of every container run on a system. We’ve even written support for automated detection of a wide range of privilege escalation attacks in containers.

I’m incredibly proud to have been part of a team that’s kept up with much larger organisations while simultaneously developing innovative new security features. As we’ve seen in recent months, internet security isn’t just about compromised credit cards any more - it has the potential to shape the future of entire countries. The CoreOS model of making things as secure as possible by default, of getting security fixes into deployment as quickly as possible, and of making it easier to determine whether or not a system has been compromised will be critical to avoiding the same kind of security failings that we’ve seen so frequently in the past.

But the security of servers and services is only part of the puzzle, and the security of client systems is just as critical. My passion has always been the security and safety of end-users, and so I’ve taken the incredibly difficult decision to leave CoreOS to work on client system security.

CoreOS is in a unique position to do meaningful security work and get it into production rapidly; and I'm proud to have been part of that team that has brought meaningful security improvements to users. Even though I am leaving CoreOS I’ll still be participating in open source development and introducing new exciting security features to the ecosystem. The beauty of collaborative development is that CoreOS will continue to benefit from my work, just as I (and the rest of the internet) will continue to benefit from theirs.

Container Linux by CoreOS ships dm-verity, a technology that builds on trusted boot and secure boot to make it impossible for attackers to modify the underlying filesystem containing the OS. This security mechanism is enabled by default, helping ensure that the whole system is in a trustworthy state.

A core part of Container Linux is the automated image-based update strategy. Each Container Linux install has three partitions that are used by the OS:

• ROOT, which contains local configuration and data and is mounted at /. If this partition is wiped it will be automatically restored to the initial configuration, making it easy to perform a “factory reset” of Container Linux systems.

• USR-A, which contains the operating system itself and is mounted at /usr. This contains the initial version of the operating system after install.

• USR-B, which is initially blank. On system update, the new version of the OS is written to USR-B. On post-update reboot, it will be used as /usr instead of USR-A, and the next update will be written to USR-A instead.

Since all local configuration and data is stored in /, and since updates are written out to a separate partition, this means that there’s no need for /usr to be modified at runtime. In fact, the filesystem is explicitly configured such that the kernel will only permit it to be mounted read-only. However, this does nothing to protect against out-of-band modification such as writing directly to the partition. This is where dm-verity comes in.

What is dm-verity

dm-verity is a Linux kernel feature originally developed by Google as part of the Chrome OS project. It interfaces with the kernel’s Device Mapper layer, which allows code to be interposed between the underlying block device (in this case the /usr partition) and the filesystem layer’s view of the partition. When a filesystem is created with dm-verity, each block of the filesystem is hashed and a tree of those hashes stored. Whenever a block is read from the underlying block device, dm-verity ensures that it hashes to the stored value before passing it to the filesystem layer. If the hash doesn’t match, an I/O error is returned instead. As a consequence, any filesystem modifications will generate errors.

For this to be secure, dm-verity also needs to be able to verify that the stored hashes have not themselves been tampered with. This is achieved by storing a root hash and passing that to the kernel when setting up dm-verity at runtime. If any hashes have been modified, they will fail to match the root hash and again an error will be generated.

But what secures the root hash?

We chose to place the root hash inside the kernel image itself, which allows it to be protected by either Trusted Boot or Container Linux’s forthcoming UEFI Secure Boot support. The bootloader extracts the hash after verifying or measuring the kernel and passes it on the kernel command line, allowing it to be used during dm-verity setup. For this to be possible, the root hash has to be stored in the uncompressed region of the kernel. We do this by taking advantage of an interesting historical quirk.

In the early days of the kernel, a kernel image could be written directly to a floppy disk and booted from the firmware. This was achieved by embedding a small bootloader in the kernel’s setup code. As time passed, this became less useful (kernels got too big to fit on floppies, and floppy drives pretty much vanished) and the code was removed. However, in order to alert anybody who was used to this functionality, a stub was added that simply prints:

Use a bootloader

Remove disk and press any key to reboot…

This string is not used in any other part of the boot process. It is also (including newline characters) exactly 64 bytes long. This is, conveniently, the length of an ascii-encoded dm-verity root hash. The hash is generated during OS build and embedded in the kernel before the kernel is signed and its measurements calculated.

Trustworthiness with dm-verity enabled in Container Linux

This ensures that the root hash cannot be tampered with, and allows users to have a high level of certainty in the trustworthiness of their /usr filesystems. We’ve been shipping with dm-verity enabled by default since the 1214.0 release, providing additional security in an entirely transparent manner. This is enabled by default and should be entirely transparent to users, but file an issue if it appears to cause any problems.

If you are interested in learning more about Container Linux in general, join a live webinar about Staying Ahead of Vulnerabilities with Automatic Updates in Container Linux on January 26 at 9:15 a.m. PT, or watch it any time after.

2017 is the year Kubernetes becomes the backbone of distributed systems. In 2016, the Kubernetes community greatly expanded as more people understood the potential of container orchestration.

As the community of users has grown, so has the number of people and organizations who contribute to the project, overall increasing the capabilities of Kubernetes.

As it continues to gain popularity and widespread use, Kubernetes is blending into the background as the de facto infrastructure powering distributed systems. With CoreOS Tectonic, the first self-driving Kubernetes solution, Kubernetes customers no longer have to focus on their infrastructure and can instead focus on innovating.

This year, the industry will unify behind Kubernetes, Kubernetes technology will stabilize, and Kubernetes will become ubiquitous. By this time next year, we believe the industry will be more focused on the technology that can be built on Kubernetes than Kubernetes itself.

A unified industry push towards Kubernetes

As an industry, we will continue to push Kubernetes forward. Even companies competing in this market will work together to improve the technology towards our shared vision of what the Kubernetes ecosystem looks like. It’s like we’re all inventing an iPhone together: a powerful platform that enables us to build new kinds of app companies. We can’t wait to see what inventive companies will emerge as Kubernetes becomes the standard for infrastructure deployments.

Upleveling the conversation: Kubernetes will get boring as the technology stabilizes as plumbing

Instead of adding features to Kubernetes, development will focus on ease of use, stability, and scalability. This is crucial for enterprise adoption, as this shift in focus will provide the stability needed for large-scale production deployments. This is a necessity for Kubernetes to become the plumbing of distributed systems.

Kubernetes becomes ubiquitous

Kubernetes, as an open source project, has two major advantages: industry momentum, and excellent technology. Kubernetes has enough momentum that it is a force on its own, maintained by a non-partisan community and governing body (under the Cloud Native Computing Foundation). We’ve seen this before with OpenStack, which is now a major player in the infrastructure market.

Second, the technology is solid and tested by internet giants. Born out of Google and based on Google’s Borg technology, Kubernetes is the way hyperscale giants run their distributed systems, and they do it with great success. To go back to the iPhone metaphor: there were smartphones before the iPhone, but the iPhone nailed it in a very special way that changed the way the world interacts with technology on a fundamental level. Similarly, Kubernetes will revolutionize the way we run infrastructure and enable a host of new technologies and applications.

Post-Kubernetes companies will emerge

We’ll see the first set of “post-Kubernetes” companies emerge: ones that are building tech that assumes Kubernetes at their core. These are products that are built just on Kubernetes, like the Operators we’re developing at CoreOS. Deis Workflow is an example of a Kubernetes-first app, as are the “serverless” systems emerging around Kubernetes. These are all apps that become wildly easier to build because of Kubernetes. The Uber of distributed computing will emerge thanks to Kubernetes.

In 2017, we look forward to working with you in the Kubernetes community to make Kubernetes the ubiquitous technology for self-driving infrastructure, as well as working with your business to help you benefit from all that Kubernetes has to offer.

A new year and a new milestone release of etcd. Hot on the heels of 17 bugfix releases to etcd 3.0, two alphas, and two release candidates, the etcd team is proud to announce etcd 3.1. This edition of etcd features performance, reliability, and API enhancements over the 3.0 series. It also introduces the first iteration of the etcd v3 gRPC proxy, a smart proxy for offloading client requests away from the core cluster.

Fast linearized reads

etcd provides both serialized and linearized consistency models for reading keys. A serialized read is fast, no consensus is necessary, but unsuitable for many applications because it may return stale data. A linearized read returns the most recent keys by going through etcd’s underlying raft protocol, and therefore carries greater overhead. While etcd 3.0 processes linearized reads through direct raft proposals, chewing precious disk bandwidth and incurring the corresponding latency penalty, linearized reads in 3.1 issue idempotent, fsync-free linearized raft index requests. These index requests are also batched; concurrent linearized reads coalesce into a single index request. The upshot is linearized reads have lower latency and better throughput.

The graph below, based on quick measurements from a laptop with the etcd benchmark tool, illustrates the improvement as concurrency increases. The linearized read implementation in 3.1 clearly outperforms 3.0 linearized reads given little concurrency and rapidly reaches its maximum throughput. These quick measurements are here to give a sense of the overall trend of these improvements, and we have deliberately redacted specific numbers since only relative trends matter here. We have an in-depth benchmarking series, with rigorous benchmarks, coming soon.

Availability and Reliability

In the past, upgrading an etcd cluster meant temporarily losing the leader and therefore a brief loss of availability. If an etcd member needed to be taken offline, when upgrading for example, and if it was the cluster leader, then the cluster followers would timeout on that leader and initiate an election. Waiting for a new leader would cause a short cluster outage. To improve availability in this case, the leader will automatically transfer its leadership to another member before going offline.

The quorum-based consensus for etcd has a drawback in that permanent quorum loss permanently downs the cluster. To avoid instances of operator error which inflict quorum loss on the cluster, etcd now by default checks member health before reconfiguring cluster membership. These checks use peer liveness information to ensure membership changes are safe. A member removal request is rejected if the removal causes quorum loss when considering how many members are active. A member add request is rejected if quorum would be lost if the new member never joins the cluster, in case the member is configured with a bad address.

New APIs

Based on user feedback, the etcd v3 API now includes features to better manage leases, efficiently process keys by revision, and reduce total round-trips. Leases now support non-destructive TTL fetches, useful for checking the time left on a lease, and listing attached keys, useful for finding all resources attached to a session. Key range requests can specify minimum and maximum modification and creation revisions, useful when monitoring wait lists for distributed locks and elections. Watches can optionally return the old key value on delete events, saving the cost of a round trip.

The etcd v3 authentication API, which was alpha in the 3.0 series, is now stable; any future change to the authentication API will not break older etcd v3 clients. The role-based authentication model is similar to the one found in etcd v2 API. The major differences include authentication tokens, a much faster mechanism than etcd v2’s per-request bcrypt calls, and permissions governed by key range intervals, instead of only key prefixes like in etcd v2.

Introducing gRPC proxy

An etcd cluster replicates its data to all its members. The overhead from this replication counterintuitively causes an etcd cluster to slow down, instead of scaling, after adding more members. This performance loss fixes the number of members to the desired fault tolerance; scaling must be achieved through other means. The new etcd gRPC proxy aims to reduce the amount of load on the core etcd cluster through caching and request coalescing.

The gRPC proxy includes a cache of recently accessed keys. The cache serves serialized key fetches, which don’t need to go through consensus, that would otherwise be handled by a cluster member. The advantage is the proxy absorbs serialized key request spam from misbehaving or misconfigured clients. The graph below shows the effect of this cache on repeatedly fetching the same key until CPU saturation; it effectively eliminates CPU load on the etcd server.

The proxy also coalesces watches from many clients into a single watch stream. This coalescing conserves total open connections to the cluster and reduces overall network traffic from the cluster by deferring event fan-out to the proxy.

Learn more

The latest and greatest etcd developments can be found in the etcd github repository. The project also hosts signed binaries for 3.1.0 and historical releases on the etcd release page. The github repository also has the most up-to-date etcd documentation.

As always, the etcd team is committed to building the best distributed consistent key-value store; feel free to report any bugs, ask questions, or make suggestions on the etcd issue tracker.