WHAT IS LF EDGE

LF Edge is an umbrella organization that aims to establish an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system. By bringing together industry leaders, LF Edge will create a common framework for hardware and software standards and best practices critical to sustaining current and future generations of IoT and edge devices.

We are fostering collaboration and innovation across the multiple industries including industrial manufacturing, cities and government, energy, transportation, retail, home and building automation, automotive, logistics and health care — all of which stand to be transformed by edge computing.

What is. LF Edge

Project EVE Promotes Cloud-Native Approach to Edge Computing

The LF Edge umbrella organization for open source edge computing that was announced by The Linux Foundation last week includes two new projects: Samsung Home Edge and Project EVE. We don’t know much about Samsung’s project for home automation, but we found out more about Project EVE, which is based on Zededa’s edge virtualization technology. Last week, we spoke with Zededa co-founder Roman Shaposhnik about Project EVE, which provides a cloud-native based virtualization engine for developing and deploying containers for industrial edge computers (see below).

LF Edge aims to establish “an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system.” It is built around The Linux Foundation’s telecom-oriented Akraino Edge Stack, as well as its EdgeX Foundry, an industrial IoT middleware project..

Like the mostly proprietary cloud-to-edge platforms emerging from Google (Google Cloud IoT Edge), Amazon (AWS IoT), Microsoft (Azure Sphere), and most recently Baidu (Open Edge), among others, the LF Edge envisions a world where software running on IoT gateway and edge devices evolves top down from the cloud rather than from the ground up with traditional embedded platforms.

The Linux Foundation, which also supports numerous “ground up” embedded projects such as the Yocto Project and Iotivity, but with LF Edge it has taken a substantial step toward the cloud-centric paradigm. The touted benefits of a cloud-native approach for embedded include easier software development, especially when multiple apps are needed, and improved security via virtualized, regularly updated container apps. Cloud-native edge computing should also enable more effective deployment of cloud-based analytics on the edge while reducing expensive, high-latency cloud communications.

None of the four major cloud operators listed above are currently members of LF Edge, which poses a challenge for the organization. However, there’s already a deep roster of companies onboard, including Arm, AT&T, Dell EMC, Ericsson, HPE, Huawei, IBM, Intel, Nokia Solutions, Qualcomm, Radisys, Red Hat, Samsung, Seagate, and WindRiver (see the LF Edge announcement for the full list.)

With developers coming at the edge computing problem from both the top-down and bottom-up perspectives, often with limited knowledge of the opposite realm, the first step is agreeing on terminology. Back in June, the Linux Foundation launched an Open Glossary of Edge Computing project to address this issue. Now part of LF Edge, the Open Glossary effort “seeks to provide a concise collection of terms related to the field of edge computing.”

There’s no mention of Linux in the announcements for the LF Edge projects, all of which propose open source, OS-agnostic, approaches to edge computing. Yet, there’s no question that Linux will be the driving force here.

Project EVE aims to be the Android of edge computing

Project EVE is developing an “open, agnostic and standardized architecture unifying the approach to developing and orchestrating cloud-native applications across the enterprise edge,” says the Linux Foundation. Built around an open source EVE (Edge Virtualization Engine) version of the proprietary Edge Virtualization X (EVx) engine from Santa Clara startup Zededa, Project EVE aims to reinvent embedded using Docker containers and other open source cloud-native software such as Kubernetes. Cloud-native edge computing’s “simple, standardized orchestration” will enable developers to “extend cloud applications to edge devices safely without the need for specialized engineering tied to specific hardware platforms,” says the project.

Earlier this year, Zededa joined the EdgeX Foundry project, and its technology similarly targets the industrial realm. However, Project EVE primarily concerns the higher application level rather than middleware. The project’s cloud-native approach to edge software also connects it to another LF project: the Cloud Native Computing Foundation.

In addition to its lightweight virtualization engine, Project EVE also provides a zero-trust security framework. In conversation with Linux.com, Zededa co-founder Roman Shaposhnik proposed to consign the word “embedded” to the lower levels of simple, MCU-based IoT devices that can’t run Linux. “To learn embedded you have to go back in time, which is no longer cutting it,” said Shaposhnik. “We have millions of cloud-native software developers who can drive edge computing. If you are familiar with cloud-native, you should have no problem in developing edge-native applications.”

If Shaposhnik is critical of traditional, ground-up embedded development, with all its complexity and lack of security, he is also dismissive of the proprietary cloud-to-edge solutions. “It’s clear that building silo’d end-to-end integration cloud applications is not really flying,” he says, noting the dangers of vendor lock-in and lack of interoperability and privacy.

To achieve the goals of edge computing, what’s needed is a standardized, open source approach to edge virtualization that can work with any cloud, says Shaposhnik. Project EVE can accomplish this, he says, by being the edge computing equivalent of Android.

“The edge market today is where mobile was in the early 2000s,” said Shaposhnik, referring to an era when early mobile OSes such as Palm, BlackBerry, and Windows Mobile created proprietary silos. The iPhone changed the paradigm with apps and other advanced features, but it was the far more open Android that really kicked the mobile world into overdrive.

“Project EVE is doing with edge what Android has done with mobile,” said Shaposhnik. The project’s standardized edge virtualization technology is the equivalent of Android package management and Dalvik VM for Java combined, he added. “As a mobile developer you don’t think about what driver is being used. In the same way our technology protects the developer from hardware complexity.”

Project EVE is based on Zededa’s EVx edge virtualization engine, which currently runs on edge hardware from partners including Advantech, Lanner, SuperMicro, and Scalys. Zededa’s customers are mostly large industrial or energy companies that need timely analytics, which increasingly requires multiple applications.

“We have customers who want to optimize their wind turbines and need predictive maintenance and vibration analytics,” said Shaposhnik. “There are a half dozen machine learning and AI companies that could help, but the only way they can deliver their product is by giving them a new box, which adds to cost and complexity.”

A typical edge computer may need only a handful of different apps rather than the hundreds found on a typical smartphone. Yet, without an application management solution such as virtualized containers, there’s no easy way to host them. Other open source cloud-to-edge solutions that use embedded container technology to provide apps include the Balena IoT fleet management solution from Balena (formerly Resin.io) and Canonical’s container-like Ubuntu Core distribution.

Right now, the focus is on getting the open source version of EVx out the door. Project EVE plans to release a 1.0 version of the EVE in the second quarter along with an SDK for developing EVE edge containers. An app store platform will follow later in the year.

Whether or not Edge computing serves as the backbone of mission-critical business worldwide depends on the success of the underlying network.

Linux Foundation's Project EVE: a Cloud-Native Edge Computing Platform

Recognizing the Edge’s potential and urgency to support Edge network, The Linux Foundation earlier this year created LF Edge, an umbrella organization dedicated to creating an open, agnostic and interoperable framework for edge computing. Similar to what the Cloud Native Computing Foundation (CNCF) has done for cloud development, LF Edge aims to enhance cooperation among key players so that the industry as a whole can advance more quickly.

By 2021, Gartner forecasts that there will be approximately 25 billion IoT devices in use around the world. Each of those devices, in turn, has the capacity to produce immense volumes of valuable data. Much of this data could be used to improve business-critical operations — but only if we’re able to analyze it in a timely and efficient manner. As mentioned above, it’s this combination of factors that has led to the rise of edge computing as one of the most rapidly -developing technology spaces today.

This idea of interoperability at the edge is particularly important because the hardware that makes up edge devices is so diverse — much more so than servers in a data center. Yet for edge computing to succeed, we need to be able to run applications right on local gateway devices to analyze and respond to IoT and Industry 4.0 data in near-real time. How do you design applications that are compatible with a huge variety of hardware and capable of running without a reliable cloud connection? This is the challenge that LF Edge is helping to solve.

Part of the solution is Project EVE, an Edge Virtualization Engine donated to LF Edge by ZEDEDA last month. I think of EVE as doing for the edge what Android did for mobile phones and what VMware did for data centers: decoupling software from hardware to make application development and deployment easier.

his curious (and somewhat unexpected) interplay between mobile and server computing requirements is exactly what makes edge so exciting. As an open source project, EVE now has a unique opportunity to blend the best parts of building blocks from projects as diverse as Android, ChromeOS, CoreOS, Qubes OS, Xen, Linuxkit, Linuxboot, Docker, Kubernetes and unikernels (AKA library operating systems — out of which AtmanOS is our favorite). And if you are still not convinced that all of these projects have much in common, simply consider this:

Today’s edge hardware is nothing like underpowered, specialized embedded hardware of yesterday. All of these boxes typically come with a few gigabits of RAM, dozens (if not hundreds) of GBs of flash and modern, high-speed CPUs with the latest features (like virtualization extensions) available by default. In short, they are very capable of supporting exactly the same cloud-native software abstractions developers now take for granted in any public cloud: containers, immutable infrastructure, 12-factor apps and continuous delivery software pipelines. From this perspective, edge hardware starts to look very much like servers in a data center (be it a public cloud or a private colo). At the same time;

These boxes are deployed out in the wild. Which means when it comes to security and network requirements, they exist in a world that looks nothing like a traditional data center. In fact, it looks a lot like the world mobile computing platforms have evolved in. Just like iPhones, these boxes get stolen, disassembled and hacked all the time in the hopes that secrets inside of them can be revealed and used as attack vectors. On the networking side, the similarity is even more striking: the way our smartphones have to constantly cope with ill-defined, flaky and heterogeneous networks (hopping between WiFi and LTE, for example) sets up a really good model for how to approach edge computing networking.

There’s no denying that EVE stands on the shoulders of all these open source giants that came before it and yet it has plenty of its own open source development to be done. In the remainder of this article, I’ll cover some of the technical details of Project EVE.

Project EVE overview

Fundamentally, EVE is a replacement for traditional (or even some of the real-time) operating systems (Linux, Windows, VxWorks, etc.) that are commonplace today in IoT and edge deployments. EVE takes control right after UEFI/BIOS and we have future plans around Linuxboot to have EVE actually replace your UEFI/BIOS altogether.

There are three key components of EVE: a type-1 hypervisor, running directly on bare metal; an Edge Container runtime that allows you to run applications in either a virtual machine or container; and a hardened root-of-trust implementation for security. A full list of hardware that EVE was tested on is available on the project’s Wiki page, but we expect EVE to run on most modern edge computing hardware (including products from major companies like Advantech and Supermicro, as well as architectures from ARM and Intel).

Project EVE Introduction

Once the EVE instance is up and running, the first thing it does is contact a pre-defined controller and receive instructions from the controller on how to configure itself and what workloads to start executing. The controller builds these instruction manifests for every EVE-enabled device that it knows about, based on the overall orchestration requests it receives from the DevOps rolling out a given deployment.

The API that EVE uses to talk to the controller is part of the LF Edge standardization efforts and we fully expect that it can evolve into the industry de-facto standard for how edge virtualization infrastructure is being controlled and monitored. You can see the current version of the API and documentation in EVE’s GitHub repository.

The kinds of workloads that a DevOps will be deploying to all EVE-enabled devices are packaged as Edge Containers. Edge Containers are meant to be an extension of traditional OCI Containers and the effort around their standardization will be ongoing in LF Edge in the coming months. The idea behind Edge Container extensions is to allow for seamless integration between virtual machine, unikernel and container workloads through a single packaging and distribution format.

Continuing with our Android analogy, one may say that while EVE is trying to do for the edge what Android has done for mobile, Edge Containers are meant to be the APKs of the edge.

All of EVE’s functionality is provided by a series of individual Go microservices that are running in full isolation from each other, similar to the pioneering ideas of radical isolation introduced by Qubes OS. Our ultimate goal is to make each one of those microservices be a standalone unikernel running directly on top of a type-1 hypervisor without requiring any operating system at all. We are planning to leverage excellent work done by the AtmanOS community in order to achieve that.

All of EVE’s microservices and infrastructure elements (think boot loader, Linux kernel, etc.) are tied together into a Linuxkit-like distribution that allows us to provide bootable EVE images ready to be deployed on Intel– and ARM-based edge hardware.

Our root-of-trust architecture leverages TPM and TEE hardware elements and provides a solid foundation for implementing flexible secret management, data encryption and measured boot capabilities without burdening application developers with any of that complexity.

Finally, on the connectivity side, EVE offers flexible networking capabilities to its Edge Containers through transparent integration of LISP protocol and crypto-routing. That way, EVE can provide SD-WAN and mesh networking functionality right out of the box, without requiring additional integration efforts.

Putting it all together, the internals of EVE’s architecture look something like this:

While this architecture may seem complex and daunting at times, we’re rapidly investing in documenting it and making it more flexible to work with. The EVE community shares the spirit of the Apache Way and believes in “Community over Code.” We welcome any and all types of contributions that benefit the community at large, not just code contributions:

Providing user feedback;
Sharing your use cases;
Evangelizing or collaborating with related products and technologies;
Maintaining our wiki;
Improving documentation;
Contributing test scenarios and test code;
Adding or improving hardware support;
Fixing bugs and adding new features.

The most important part of Project EVE is that it’s an open standard for the community, designed to make it easier for others to create and deploy applications for the edge. Now that the code is officially open sourced through LF Edge, it’s also available for anyone to contribute to and explore.

Shaposhnik: I think through the introduction, it's pretty clear who I am. If you're interested in talking to me about some of the other things that I do in the open source, feel free to do that. I happen to be very involved in Apache Software Foundation and Linux Foundation.

Today, we will be talking about edge computing. Let's start by defining the term, what is edge computing? I think we started a long time ago with IoT, Internet of Things. Then Cisco introduced this term called fog computing, which was telco-ish, IoT view. I think edge computing to me is very simple. It is basically cloud native IoT. It is when the small devices, I call them computers outside of data centers, they start to be treated by developers in a very cloud native way. People say, "We've been doing it for years. What's different?" The difference is it's all of the APIs and all of the things that we take for granted in the cloud and even in the private data center today. That actually took time to develop. We didn't start with Kubernetes, and Docker, and orchestration tools, and mesh networks. We started with individual machines. We started with individual rackable servers. That's basically what IoT is still, individual machines. The whole hope is that we can make it much better and much more exciting by applying some of the cloud native paradigms like liquid software, pipeline delivery, CI/CD, DevOps, that type of thing, but with the software running outside of your data center.

When I talk about edge, let's actually be very specific, because there are different types of edge. I will cover the edge I will not be talking about. Very specifically, let's talk about the edge that's very interesting to me, and I think it should be interesting to all of you. These are the type of devices that some people called deep edge, some people call enterprise edge. These are basically computers that are attached to some physical object. That physical object could be a moving vehicle. It could be a big turbine generating electricity. It could be a construction site. The point being is that something is happening in the real world and you either need to capture data about that something, or you need to drive the process of that something. Manufacturing is a really good example. You have your pipeline. You're manufacturing your product. You need to control that process. You have a computer that is typically called industrial PC attached to it. Same deal with a construction site, or even your local McDonald's. In McDonald's, you want to orchestrate the experience of your customers. You have a little computer that's attached to the cash register. You have a little computer that's attached to the display, and all of that needs to be orchestrated.

What I'm not talking about, I'm not actually talking about two things. I'm not talking about Raspberry Pis. There's definitely a lot of excitement about Raspberry Pis. It's interesting because if you think about the original motivation for the Raspberry Pi, it was to give underprivileged kids access to computing. It was basically to replace your personal laptop or desktop with essentially a very inexpensive device. The fact that Raspberry Pis now find their way into pretty much every single personal IoT project, there's almost a byproduct of how they designed the thing. I am yet to see Raspberry Pis being used for business, most of the time they just stop at the level of you personally doing something, or maybe you doing something with your friends, your hackerspace. Today, we'll not be talking about any of that. The reason we're not talking about that is because just like with container orchestration and Docker, you don't really need those tools unless you actually do some level of production. You don't really need those tools if you're just tinkering. You don't need Kubernetes to basically run your application if you're just writing an application for yourself. You only need Kubernetes if that is something that actually generates some business. We will not be talking about Raspberry Pis. We'll not be talking about telco edge, edge of the network, all of that.

Even this slice of the edge computing alone, given various estimations, represents a huge total addressable market. The biggest reason for that is the size of the data. These computers are connected to something that is in the real world. The data originates in the real world. The previous presentation today about self-driving vehicle from Uber is a perfect example of that. There's so much data that the vehicle is gathering, even if it was legally allowed, it is completely impossible to transfer all of that data to the big cloud in the sky for any processing. You have to orchestrate that behavior on the edge. As practitioners, we actually have to figure out how to do that. I was a little bit underwhelmed that Uber is focusing more on the machine learning. I understand why, but I'm an infrastructure guy. Today, I will be talking to you about infrastructure, how to make those types of applications easily deployable.

The good news is the total addressable market. The bad news is that it's a little bit of a situation like building the airplane while it's in flight. I think it would be fair to say that edge computing today is where cloud computing was in 2006. 2006, Amazon was starting to introduce EC2. Everybody was saying, it's crazy, it will never work. People at Netflix started doing microservices. Everybody says it's crazy, it will never work. The rest is history. Edge computing is a little bit of that. My goal today is to give you enough understanding of the space, to give you enough understanding of the challenges in this space but also the opportunities in this space. Also, explain maybe a little bit of the vocabulary of this space so you can orient yourself. I cannot give you the tools. I cannot really give you something that you will be immediately productive at your workplace, the same way that I can talk about Kubernetes, or Kafka, or any other tool that's fairly mature. Edge computing is just happening in front of our eyes. To me, that's what makes it exciting.

In a way, when I say cloud native, to me, edge computing represents basically one final cloud that we're building, because we've built a lot of the public clouds. There's Google. There is Microsoft. There is obviously Amazon. All of these are essentially in the business of getting all of the applications that don't have to have any physicality attached to them. What we're trying to do is we're trying to basically build a distributed cloud from the API perspective that will be executing on the equipment that doesn't belong to the same people who run public clouds. Edge computing is where ownership belongs to somebody else, not the infrastructure provider. From any other perspective, it's just the cloud. People always ask me, "If edge is just another cloud, can we actually reuse all of the software that we developed for the cloud and run it on these small computers"?

Project EVE Architecture Overview

It used to be a challenge even to do that, because those computers used to be really small. The good news now is that the whole space of IoT bifurcated. The only constraint that you have from now on is power budget. It might still be the case that you have to count every single milliamp. If you're in that type of a business, you're doing essential Snowflake's and bespoke things all the time. There's really no commonality that I can give you because everything has to be so super tightly integrated, because you're really in a very constrained power budget. Everything else where power is not a problem, it used to be that silicon cost used to be a problem, but that's not the case anymore. Thanks to the economy of scale, you can basically get Raspberry Pi class devices for essentially a couple dozen bucks. It actually costs more to encase them in a way that would make them weatherproof than to actually produce the silicon.

The computers are actually pretty powerful. These are the type of computers we used to have in our data centers five years ago. Five years ago, public cloud existed. Five years ago, Kubernetes already existed. Docker definitely existed. The temptation is to take that software and run it at the edge. There have been numerous attempts to rub some Kubernetes on it because, obviously, that's what we do. We try to reuse as much as possible. Pretty much every attempt of reusing the implementation that I know of failed. I can talk in greater details of why that is. APIs are still very useful. If you're taking the implementation that Kubernetes gives you today, that will not work for two reasons. First of all, it will not work because of the network issues. All of those devices happen to be offline more than they are online. Kubernetes is not happy about that type of situation. Second of all, and this is where you need to start appreciating the differences of why edge is different, interestingly enough, in the data center, the game that Kubernetes and all of these orchestration technologies play is essentially a game of workload consolidation. You're trying to run as many containers on as few servers as possible. The scalability requirements that we're building the Kubernetes-like platforms with are essentially not as many servers and tons of containers and applications. On the edge, it's exactly the reverse. On the edge, you basically have maybe half a dozen applications on each box, because boxes are ok, but they're still 4, 8 gigs of memory. It's not like your rackable server, but you have a lot of them.

Here's one data point that was given to us by one of our biggest customers. There's an industrial company called Siemens. That industrial company is in the business of managing and supporting industrial PCs that are attached to all things. Today, they have a challenge of managing 10 million of those industrial PCs. By various estimations, total number of servers inside of all of the Amazon data centers is single digit millions. That gives you a feel for what scale we should actually be building this for.

Finally, the economics of the edge is not the same as with the data center. All of these challenges essentially, make you think, we can reuse some of the principles that made cloud so successful and so developer friendly nowadays. We actually have to come up with slightly different implementations. My thesis is that the edge computing will be this really interesting, weird mix of traditional data center requirements, and actually mobile requirements. Because edge computing is like the original edge computing is this. Actually, the original edge computing, I would argue, is Microsoft Xbox. With this we really got our first taste for what an edge computing-like platform could look like. All of the things that made it so, the platforms, Android or iOS, the mobile device management approaches, cloud, Google Play Store or Google services, all of that will actually find its way into the edge. We have to think about, how will it look like? We also need to think about traditional data center architectures, like operating systems, hypervisors, all of that. I will try to outline and map out how Linux Foundation is trying to approach this space.

Open Source Edge Computing Platforms - Overview

Edge is actually pretty diverse, not just in terms of the ownership, but also in terms of the hardware and applications. Today, let's take industrial PCs. Pretty much all of them are running Windows. They're all x86 based hardware running Windows. When I say Windows, I actually mean Windows XP. Yes, it exists. A lot of SCADA applications are still based on Windows XP. If you show up as a developer and start razzle-dazzling these customers with your cloud native microservices-based architectures, the first question that they're going to ask you is, "It's all great. This is the new stuff. What about my old stuff? I want to keep running my old stuff. Can you give me a platform that would be able to support my old stuff, while I am slowly rebuilding it in this new next-generation architecture?" That becomes one of the fundamental requirements.

Scale, we already talked about the geographic aspect of it and deployments and the maintenance. The security is also interesting. Edge computing, unlike data center is much closer to this. Because edge computing is physical, which means you cannot really rely on physical security to protect it. It's not like there is a guy holding a machine gun in front of a data center, you cannot put that guy in front of every single edge computing device. You basically have to build your platform, very similarly to how iOS and Android are protecting all of your personal data. That's not something that data center people are even thinking about, because in a data center, you have your physical security and you have your network security. We are done with that. On a perimeter, you pay a lot of attention to it, but within the data center, not so much.

Also, interestingly enough, what I like about edge is that edge is probably the hardest one to really succumb to a vendor lock-in. Because the diversity is such that not a single vendor like a big cloud provider can actually handle it all. Edge is driven a lot by system integrator companies, SIs. SIs are typically pretty vertical. There may be an SI that is specializing in industrial, in retail, this and that. That diversity is actually good news for us as developers because we will not see the same concentration of power like we're seeing in the public cloud today, so I think it's good for us.

A lot of what I will be covering, in this talk, I wanted to pitch this other talk that just was made publicly available, taken out. This is the first time ever that Microsoft Xbox team talked about how they develop the platform for Xbox. That was done about a month ago, maybe two months ago, first time ever. A lot of the same principles apply, which makes me happy because we thought about them independently. The tricks that they played are really fascinating. The challenges they faced are very similar to the edge. If you want to hear from somebody who can claim that they successfully developed an edge platform, listen to those guys. I'm talking about the platform that's being developed. Mine can still fail, theirs is pretty successful.

Enterprise Edge Computing with Project EVE - Jason Shepherd, Zededa

Let's switch gears a little bit and talk about how Linux Foundation got involved in all of this. I shouldn't be the one to tell you that Cloud Native Compute Foundation has been super successful. In a way, I would say that Kubernetes was the first Google project that was successful precisely because of CNCF. I love Google, but they have a tendency of just throwing their open-source project over the wall and basically say, "If you like it, use it, if you don't, not our problem." Kubernetes was the first one where they actively tried to build a community. The fact that they went and donated it to Linux Foundation, and that was the anchor tenant for the Cloud Native Compute Foundation, I think made all the difference. Obviously, Linux Foundation itself was pretty happy about this outcome. They would like to do more of it.

The thought process went exactly like what I was talking about. When I say inside of data centers, I mean public cloud or your private data center. It doesn't matter. It's just a computer inside of a data center. For all of that, there's basically a forum of technologists that can decide, what is the common set of best practices that we all need to apply to the space to be more productive, more effective? That's CNCF, Cloud Native Compute Foundation. For all of the computers outside of data centers, it feels like we at least need to provide that type of forum even if we don't really have an anchor tenant like Kubernetes still. We need to give people a chance to talk among themselves, because otherwise there is really no way for them to synchronize on how the technology gets developed. That's LF EDGE.

Linux Foundation Edge Initiative was announced, not that long ago, actually, this year. It was announced in January, February this year. My company, ZEDEDA, we ended up being one of the founding members. We donated our project. There are a lot of companies in the space that are now part of the LF EDGE, so if you're interested, you can go to this lfedge.org website. The membership is pretty vast at this point. These are the premium members. There are also tons of general members. A lot of the good discussions are already happening within LF EDGE.

To give you a complete picture, what does LF EDGE cover? LF EDGE basically covers all of the computers outside of data centers. It starts with what we consider to be partial edge. A partial edge would be a quasi data center. It's not quite a data center, but it looks almost like a data center if you squint. A good example of that would be a telco central office, a telco CO. It's not really built to the same specification that a telco data center or a hyperscale data center would be built for, but a lot of technologies still apply. That's definitely in scope for LF EDGE. Then we basically go to telco access points. These are already physical devices. We're talking base stations. We're talking 5G deployments. These are all of the things in the CD infrastructure, or any infrastructure that would have to run some compute on them. That's definitely in scope for LF EDGE. Both of these are pretty dominated by telcos today, for good reason, because they're probably the best example of that type of an edge computing.

Then there are two other examples of edge. One that I will spend a lot of time talking about, we call it, for now, enterprise edge. This is basically all of those industrial PCs, IoT gateways. An example of the enterprise edge would be also a self-driving vehicle. Uber or Tesla building it would be also an example. Finally, there's obviously consumer edge. This is all of your washers, and dryers, and your refrigerators, all of that is in scope for LF EDGE. Every single one of these areas basically has a project that was donated by one of the founding companies. HomeEdge is from Samsung, which is not surprising because they're making all of these devices that you buy. Enterprise edge is us, ZEDEDA, and a few big enterprise companies like Dell, those types of guys. There's project Akraino that's dominated by telcos.

Interestingly enough, I have a friend of mine from Dell, Jason Shepherd, who keeps joking that this edge thing, it's very similar to how this country was settled. Because it feels we're now running away from the big hyperscale cloud providers, just like in the good old days people were running away for big businesses on the East Coast. The only place for us to actually build this exciting technology now is on the edge because everything else is dominated, and you have to join Google or Facebook to have a play in there. Go West, young man, go Edge.

These are the projects. I will be specifically talking about one of them, Edge Virtualization Engine. Check out the rest on the Linux Foundation website. I think you will find it very useful. Edge Virtualization Engine is what was donated by my company, ZEDEDA. We're actually working very closely with Fledge. Fledge is a middleware that runs on top of the project EVE. EVE stands for Edge Virtualization Engine.

Specifically, what requirements does EVE try to address? We basically approach looking at these boxes essentially from the ground up. We feel that we have to take control pretty much from the BIOS level up. I will talk about why that is important, because a lot of the technology that you would find at the BIOS and board management level in the data center simply doesn't exist on the edge. For those of you who know BMCs and iLOs, those things are not present on the edge for obvious reasons, because the control plane is not really to be had on the edge. Who are you going to talk to even if you have a BMC? Which creates an interesting challenge for how you can cut down on BIOS, and things like that. We feel that we need to start supporting hardware from the ground up. The hardware at the same time has to be zero touch. The experience of actually deploying the edge computing device should be as much similar to you buying a mobile device as possible. You get a device with an Android pre-installed. You turn it on, and you can run any applications that are compatible with an Android platform, so zero touch deployment.

We also feel that we need to run legacy applications. The legacy applications would include Windows XP. For Windows XP, you actually have to make sure that the application can access a floppy drive. That's a requirement. You also need to run real-time operating systems for control processes. You need to basically do hard partitioning of the hardware to guarantee the real-time SLAs on these applications. You need to build it at IoT scale, but what it really means is it needs to be at the same scale that all of the services that support your mobile devices operate at. What it means is that when you talk about edge computing, just building a service, a control plane in a single data center is not good enough, because your customers will be all over the place, sometimes even in Antarctica, or in the middle of the ocean. That also happens. You have to figure that one out. The platform has to be built with zero trust, absolutely zero trust, because we all know the stories of hacks that happened at uranium enrichment plant at Iranian facilities. The attack vector was very simple. It was a physical attack vector. Those things will keep happening unless we secure the platforms, and make them trustworthy as much as possible.

Finally, and that's where all of you come in, those platforms have to be made cloud native, in a sense that what APIs we give to developers to actually provide applications on top of them. Because if you look at the state of the industry today, and I already scared you at least a little bit with my Windows XP story, but Windows XP is actually a good story. The rest of the industry is still stuck in the embedded mindset. It's not a good embedded mindset. It's not like using Yocto or something. It's using some god-awful, embedded operating system that the company purchased 12, 15, 20 years ago, where people cannot even use modern GCC to compile the binary. That's the development experience in the edge and IoT today. I think it is only if we allow the same developers who built the cloud to actually develop for these platforms, it's only then that edge computing will actually take off. Because we are artificially restricting the number of innovative people that can come to the platform by not allowing the same tools that allowed us to make cloud as successful as it is today.

I talked a lot about various things that we plan to tackle. As developers, when I talk about cloud native, people tend to really just focus and assume app deployments. They're like, "Give me app deployments, and I'm done." The trouble is, app deployments, the way we think about them in a data center is just the tip of the iceberg on the edge. My favorite example that I give to everyone is, even if you assume virtualization, on the edge you basically have to solve the following problem. Suppose you decided on Docker containers, and now there is one Docker container that needs to drive a certain process, and another Docker container that needs to get a certain set of data. The process and the data happened to be connected to the single GPIO. This is a single physical device that basically has a pin out. Now you're in business of making sure that one container gets these two pins, and the other container gets those two pins. It's not something that would even come up as a problem in a data center. Because in a data center, all of your IO is basically restricted to networking, maybe a little bit of GPU. That's about it. Edge, is all about IO. All of that data that we're trying to get access to and unlock, that is the data that we can only access through a reasonable IO.

There are a lot of interesting plumbing challenges that need to be solved first before we can even start deploying our Docker containers. Docker containers are great. I think the thesis that we have at LF EDGE, at least within the project EVE, is basically very similar to what you would see in a data center, but with a certain set of specific details attached to it. We feel that edge needs to be treated exactly like you treat your Kubernetes cluster edge. The physical nodes, like your pods will be out there. There will be a controller sitting typically in the cloud, or it can sit on-prem, either one. All of these devices will basically talk to the controller just like your pods talk to the Kubernetes controller. Then somebody deploying the applications would talk to the control through typically a Kubernetes-like API. It is very much guaranteed to be a Kubernetes-like API. I think the API itself is great. That's very familiar to all of you. The question is, how do we build the layer that actually makes it all possible? That's where the project EVE comes in.

If I were to go through EVE's architecture, high level view, very quickly. It all starts with the hardware. Actually, it starts with the physical devices that you attach to the hardware. Then there needs to be some operating system that would allow you to do all of the above. That operating system needs to be open source. It needs to be Android of the edge type of an offering. That operating system will talk to the control plane. The control plane will sit in the cloud. On top of that offering of an operating system, you would be running your applications just like you do today in a data center, so a very typical, very familiar architecture.

Typically, your applications will talk to the big clouds in the sky from time to time, because that's where the data ends up anyway. You need to help them do that. Because a lot of times, people will talk to me and say, "I'm deploying my edge application today using Docker." I'm like, "That's great." They're like, "Now we need to make sure that the traffic flows into this particular Amazon VPC. How can we do that?" It just so happens that now you have to read a lot of documentation, because there's strongSwan involved, there's IPsec. It's not really configured by default. It's like, how can we actually connect the big cloud in the sky with this last cloud that we're building called edge computing? That has to come out of the box. These are essentially the requirements. That's the high-level architecture. I will deep dive into one specific component, which is EVE today.

State of the Edge: Exploring the Intersection of IoT, AI, 5G and Edge Computing

What we're trying to accomplish is, at the open-source layer, we need to standardize on two components. One is the runtime itself. The other one is the notion of an application. An application we're now trying to standardize we're calling that standard edge containers. The runtime is project EVE. At the top you basically have catalogs, and you have control planes. That's where companies can innovate and monetize. I would expect a lot of big cloud providers to basically join LF EDGE and essentially start building their controller offerings. Just like Amazon today gives you a lot of managed services, that will be one of the services that they would give you.

Deep diving into project EVE. EVE today is based on the type-1 hypervisor, currently Xen. We actually just integrated patches for ACRN. ACRN is Intel's type-1 hypervisor. It's a pretty simple layered cake, very traditional virtualization architecture. I will explain why virtualization is involved. It's hardware, a hypervisor, then there's a bunch of microservices that are running on that hypervisor. Finally, you get to run your containers.

That is to say that we're building the very same architecture that Android had to build for the mobile. The biggest difference being that Android built it in 2003. They essentially answered the same questions that we're answering just in a different way, because those were different times. The hardware was different. The questions are still the same. The questions are, how can you do application and operating system sandboxing because you don't want your applications to affect the operating system and vice versa? How do you do application bundling? How do you do application deployment? What hardware do you support? We are answering it more closely to a traditional virtualization play. Android basically did it through the sandboxing on top of JVM, because it made sense at the time. At the end of the day, I think Android also had this idea in mind that mobile platforms will only be successful if we invite all of the developers to actually develop for them. At the time developing for mobile was painful. It was that type of an embedded development experience. It's god-awful compilers, tool chains from the '80s. One of the key pieces of innovation of Android was like, let's actually pick a language that everybody understands and can program in called Java. We're essentially doing the same, but we're saying, language nowadays doesn't matter because we have this technology called Docker container. Language can be anything. It's the same idea of opening it up to the biggest amount of people who can actually bring their workloads to the platform.

EVE happens to be a post-, post-modern operating system. When I say it like that, I've built a couple of operating systems. I used to work at Sun Microsystems for a long time. I've built a couple of those. I used to hack on plotnine. I spent a bit of time doing that. All throughout my career, an operating system wanted to be a point of aggregation for anything that you do, hence packaging, shared libraries. An operating system wanted to be that point, that skeleton on which you hang everything. What happened a few years ago with basically the help of virtualization and technologies like unikernels, and things like that, is that we no longer view an operating system as that central aggregation point. An operating system these days is basically just enough operating system to run my Docker engine. I don't actually update my operating system, hence CoreOS. I don't really care about my operating system that much. I care about it running a certain type of workload. That's about it. That's what I mean by post-, post-modern operating system. It is an operating system in support of a certain type of workload. In case of EVE, that workload happens to be edge container.

Testing Challenges and Approaches in Edge Computing

Inside of EVE, there is a lot of moving parts. I will be talking about a few of those today. If you're interested, we actually have a really good documentation, which I'm proud of, because most of the open source projects lack that aspect of it. Go to our GitHub if you want to read some of the other stuff, so it's LF EDGE EVE, and click on the docs folder. There's the whole design and implementation of EVE that would be available to you. Let's quickly cover a few interesting bits and pieces. Here, I'm doing this hopefully to explain to you that what we're building is legit, but also maybe generate some interest so you can help us build it. If anything like that sounds interesting to you just talk to me after the presentation, we can figure out what pull request and GitHub issues I can assign to you.

EVE was inspired by a few operating systems that I had privilege to be associated with, one is Qubes OS. How many of you do know about Qubes OS? That's surprisingly few. You absolutely should check out Qubes OS. Qubes OS is the only operating system that Edward Snowden trusts. That's what he's running on his laptop, because that is the only one that he trusts. When he was escaping, his whole journey was Qubes OS that was running on his laptop. It's not perfect, but it's probably the best in terms of security thinking that I have seen in a long while.

Then there is Chrome OS. It's basically this idea that you can take an operating system and make it available on devices that you don't really manage. SmartOS was like Chrome OS or CoreOS, but derived from Solaris. EVE today is based on the type-1 hypervisor. People always ask me, why type-1? Why KVM is not allowed. The answer is simple. It's that requirement for the real-time workloads. Yes, patches for the real-time Linux kernel exist. They are really tricky. If you're talking about a pretty heterogeneous set of hardware, it's actually really tricky to maintain this single view of guaranteeing that your scheduler in Linux kernel would really be real-time. We use type-1 hypervisors and an ACRN, our choice today. We're running containers. We're running VMs. We're running unikernels. Basically, everything gets partitioned into its own domain by hypervisor but those domains can be super lightweight. With projects like Firecracker, that becomes faster and faster and pretty much indistinguishable from just starting a container.

DomU, basically where all of the microservices run, that is based on LinuxKit. LinuxKit is one of the most exciting projects in building specialized Linux-based distributions that I found in the last five years. It came out of Docker. It basically came out of Docker trying to build Docker Desktop. LinuxKit is how Docker manages that VM that happens to give you all of the Docker Desktop Services. It's also based on Alpine Linux. We get a lot of Alpine Linux dependencies.

We're driving towards unikernel architecture. Every single instance of a service will be running in its own domain. All of our stuff is implemented in Go. One of the really interesting projects that we're looking at is called AtmanOS, which basically allows you to do this, see that line, GOOS equals Xen, and you just do, go build. AtmanOS figured out that you can create very little infrastructure to allow binary run without an operating system, because it so happens that Go is actually pretty good about sandboxing you. Go needs a few services from an operating system, like memory management, scheduling, and that's about it. All of those services are provided directly by the hypervisor. You can actually do, go build, with GOOS Xen and have a binary that's a unikernel.

Edge Computing Standardisation and Initiatives

Finally, we're actually trying to standardize edge containers, which is pretty exciting. We are trying to truly extend the OCI specification. There have been a few areas in the OCI that we're looking at. Image specification itself doesn't require much of a change. The biggest focus that we have is on registry support. We don't actually need runtime specification because OCI had this problem that they needed to integrate with other tools. Remember, classical operating system, when all the classical operating systems were black box execution engine. We don't need to integrate with anything but ourselves, hence runtime specification is not really needed. Good news is that there are actually a lot of the parallel efforts of extending the OCI into supporting different types of containers. Two that I would mention are Kata Containers, which are more traditional OCI, but also Singularity Containers, which came more from HPC and giving you access to hardware. Weaveworks is doing some of that same thing. Check them out. Obviously, Firecracker is pretty cool as a container execution environment that also gives you isolation of the hypervisor.

Top three goals that we have for edge containers are, basically allow you not only file system level composition, which is what a traditional container gives you. You can basically compose layers. We happen to be glorified tarball. You do everything at the level of the file system. You add this file, you remove that file. We're also allowing you block-level composition. You can basically compose block-level devices, which allows you then to manage disks, VMs, unikernels, this and that. We allow you hardware mapping. You can basically associate how the hardware maps to a given container, not at the runtime level, but at the container level itself.

We still feel that the registry is the best thing that ever happened to Docker. The fact that you can produce a container is not interesting enough. The fact that you can share that container with everybody else, that is interesting. We feel that the registry basically has to take onto an ownership of managing as many artifacts as possible, which seems to be the trajectory of OCI anyway. Things like Helm charts and all the other things that you need for orchestration, I would love for them to exist in the registry. Because that becomes my single choke point for any deployment that then happens throughout my enterprise.

Eve's networking is intent based. You will find that very familiar to any type of networking architecture that exists in VMware or any virtualization product, with a couple of exceptions. One is cloud network, which is, literally, intent is, connect me to that cloud. I don't care how. I'm willing to give you my credentials but I need my traffic to flow into the Google, into Amazon, into Microsoft Cloud. Just make it happen. The way we do make it happen is each container or each VM, because everything is virtualized, basically gets a virtualized NIC, network interface card. What happens on the other side of that NIC? Think of it as basically one glorified sidecar, but instead of using the sidecar that has to communicate through the operating system. We communicate through the hypervisor. Basically, the VM is none the wiser of what happens to the traffic. All of that is configured by the system, which allows us really interesting tricks, like networking that Windows XP into the Amazon cloud. Otherwise, it would be impossible. You can install IPsec to Windows XP but it's super tricky. Windows XP that just communicates over the virtualized NIC and the traffic happens to flow through IPsec and to Amazon cloud, that Windows XP instance, is none the wiser.

Another cool thing that we do networking-wise is called mesh network. It is basically based on the standard called LISP, which has an RFC 6830. It allows you to have a flat IPv6 overlay namespace where anything can see anything else. That IPv6 is true overlay. It doesn't change if you move the device. What allows it to do is basically bypass all of the NetBoxes, and all of the things that may be in between this edge device and that edge device, so that they can directly communicate with each other. Think about it as one gigantic Skype or peer-to-peer system that allows everything to basically have a service mesh that is based on IPv6 instead of some interesting service discovery. That's networking.

On trust, we're basically building everything through the root-of-trust that's rooted at the hardware element. On Intel most of the time it happens to be TPM. TPMs exist in pretty much every single system I've seen. Yet nobody but Microsoft seems to be using them. Why? Because the developer support still sucks. On Linux, it actually takes a lot of time to enable TPM and configure TPM. We're virtualizing the TPM. We use it internally, but then the applications, the edge containers get the virtualized view of the TPM. We also deal with a lot of the crap that exists today in the modern x86 based system. Because a lot of people don't realize it but there is a lot of processors and software that runs on your x86 system that you don't know about. Your operating system, even your hypervisor is not the only piece of software. We're trying to either disable it or make it manageable. Our management starts from the BIOS level up. Thanks to Qubes for pioneering this. Everything runs in its own domain. We're even disaggregating device drivers. If you have a device driver for Bluetooth and it gets compromised, since it's running in its own domain, that will not compromise the rest of the system. Stuff like that.

EVE's software update model is super easy for applications. It's your traditional cloud native deployment. You push to the cloud and the application happens to run. If you don't like it, you push the next version. You can do canary deployments. You can do all of the stuff that you expect to see from Kubernetes. EVE itself needs to be updated. That's where ideas from Chrome OS and CoreOS kick in. It's pretty similar to what happens on your cell phone. It's dual partitioned with multiple levels of fallback, lots of burn-in testing that we do. We're trying to avoid the need for physical contact with edge nodes as much as possible, which means that a lot of things that would have you press a key would have to be simulated by us. That's a whole tricky area of how to do that. That's something that we also do in EVE. We are really big fans of the open-source BIOS reimplementation from coreboot, and especially u-root on top of coreboot. That allows us to basically have a complete open-source stack on everything from the BIOS level up.

The most interesting work that we're doing with TPM, and I have to plug it because I get excited about every single time, we're trying to basically do a hardware-protected vTPM, something that hasn't been done before, even in the data center. There's a group of us who is doing it, if you're interested you can contact any one of us. TrenchBoot is the name of the project. There's Dave Smith and LF EDGE in general.

Eve itself is actually super easy to develop. That's the demo that I wanted to give, because it's not a QCon without a demo. EVE is based on LinuxKit. There is a little makefile infrastructure that allows you to do all of the traditional operating system developer things. Basically, typing make run would allow you to manage the operating system, run the operating system. The only reason I'm mentioning this is because people get afraid a lot of times if I talk about operating system development and design, because there's a little bit of a stigma. It's like, "I need a real device. I need some J-Tech connector. I need a serial port to debug it." No, with EVE, you can actually debug it all in the comfort of your terminal window on macOS.

The entire build system is Docker based. Basically, all of the artifacts in EVE get packaged as Docker containers. It's actually super easy to develop within a single artifact. Because we're developing edge containers in parallel, we are planning to start using that for the unikernel development as well, which might, interestingly enough, bifurcate and be its own project. Because, I think when it comes to unikernels, developers still don't really have the tools. There's a few available like UniK and a few others. There's not, really, that same level of usefulness of the tools that Docker Desktop just gives me. We're looking into that as well.

Edge computing today is where public cloud was in '06. Sorry, I cannot give you ready-made tools, but I can invite you to actually build the tools with me and us at Linux Foundation. Edge computing is one final cloud that's left. I think it's the cloud that will never ever be taken away from us. By us, I mean people who actually run the actual physical hardware. Because you could tell, I'm an infrastructure guy. It sucked when people stopped buying servers and operating systems, and now everything just moved to the cloud. My refuge is edge. Edge computing is a huge total addressable market. As a founder of a startup company, I can assure you that there is tremendous amount of VC activity in the space. It's a good place to be if you're trying to build a company. Kubernetes as an implementation is dead, but long live Kubernetes as an API. That stays with us. Edge computing is a lot of fun. Just help us build either EVE, a super exciting project, or there are a few projects to pick in the LF EDGE in general.

Participant 1: We see the clouds, AWS, Azure, and all that. Is there L2 connectivity? Are you using, for example, the AWS Direct Connect APIs, and for Azure, ExpressRoute? That's what you're doing?

Shaposhnik: Yes, exactly.

Participant 1: I belong to Aconex and we are delving into a similar thing, we already allow people to connect to the cloud. We'll look deeper into this.

Shaposhnik: Absolutely. That's exactly right. That's why I'm saying it's a different approach to running an operating system because I see a lot of companies trying to still integrate with Linux, which is great. There is a lot of business in that. What we're saying is Linux itself doesn't matter anymore. It's the Docker container that matters. We're extending it into the edge container. Docker container is an edge container. It almost doesn't matter what an operating system is. We're replacing all layers of it with this very built for purpose engine. While it's still absolutely a valid approach to still say, "I need Yocto," or some traditional Linux distribution that integrates with that. I think my only call to action would be, let's build tools that would be applicable in both scenarios. That way we can help each other grow.

Participant 2: I know in your presentation you mentioned that edge is going to be more diverse. What's your opinion on cloud providers extending to the edge through projects like Azure Sphere and Azure IoT Edge?

Shaposhnik: They will be doing it, no question about it. I think they will come from the cloud side. Remember that long range of what's edge and what's not edge. They will basically start addressing the issues at the CO, the central office. They will start addressing the issues at the maybe Mac access points. I don't see them completely flipping and basically running on the deep edge. The reason for that is, business-wise, they're not set up to do that. The only company that I see that potentially can do that is Microsoft. Because if you want to run on the deep edge, you need to develop and foster your ecosystem, the same way that Microsoft developed and fostered the ecosystem that made every single PC run Windows. Amazon and ecosystem don't go together in the same sentence. Google is just confused. If anybody tackles it, that would be Microsoft, but they are distracted by so much of a low-hanging fruit in front of them just moving their traditional customers into the cloud, that I just don't see them as applying effort in that space. It may happen in five years, but for now, running this company, at least I don't see any of that happening.

Participant 3: What about drivers for sensors on these edge devices? It seems EVE abstracts the OS away from you, but in industrial, for instance, you need to detect things, so you need peripherals.

Shaposhnik: Correct. What about drivers? Because it's a hypervisor based architecture, we can just assign the hardware directly to you. If you want to have that Windows XP based VM drive your hardware, we can do that. That's not interesting, because we need software abstractions that will make it easier for developers to basically not think about it. That is the work that is a very nascent chunk of work. How do you provide software abstractions for a lot of things that we took for granted, like there's a file in /dev someplace, and I do something with it through Yocto. Now we're flipping it back and saying, "If I'm running a Docker container, what would be the most natural abstraction to a particular hardware resource?" A lot of times, surprisingly, to me, that abstraction happens to be a network socket. We can manage the driver on the other side of the hypervisor. Again, we will still run the driver in its own domain. To all of the containers that want to use it, we will basically present a nice software abstraction such as network socket.