Introduction

With Amazon ECS Anywhere, you can run and manage containers on any customer-managed infrastructure using the same cloud-based, fully managed, and highly scalable container orchestration service you use in AWS today. Amazon ECS Anywhere provides support for registering an external instance, such as an on-premises server or virtual machine (VM), to your Amazon ECS cluster. External instances are optimized for running applications that generate outbound traffic or process data, and can also be used to host applications that service inbound requests such as frontend web applications, microservices, or Application Programming Interfaces (APIs).

When using Amazon ECS Anywhere to manage applications that service inbound requests, it’s possible to configure an Amazon Elastic Containers Service (Amazon ECS) service to run and maintain a specified number of instances of a task definition simultaneously across multiple Amazon ECS external instances to provide increased capacity, availability, and reliability of your applications. In this scenario, deploying a load balancer to an external instance helps simplify the distribution of inbound traffic evenly between each of the instances that host an application.

This post we demonstrate how to deploy and load balance a web application across multiple Amazon ECS Anywhere external instances, using the Traefik Proxy (Traefik) network load balancer.

Traefik is an open-source cloud-native load balancer and reverse proxy application developed by the software company Traefik Labs. Traefik supports automatic service discovery and configuration for a variety of orchestrators, including Docker, Amazon ECS, Kubernetes, and Marathon. Traefik also provides support for encrypted traffic termination, automatic security certificate issuance and renewal, and additional load balancing capabilities such as circuit breakers, and rate limiting.

Solution overview

In reference to the Solution overview diagram, the solution demonstrated in this post has the following key characteristics:

• The load balanced web application is deployed by Amazon ECS Anywhere to multiple managed external instances.
• The web application is deployed as an Amazon ECS service Whoami, and the associated task definition simultaneously runs four instances of the containerized whoami web application distributed across two separate external instances.
• The Traefik load balancer is deployed as an Amazon ECS service LoadBalancer, and is configured to load balance incoming HTTP requests evenly across all instances of the whoami web application using rule based traffic routing.
• The solution implements the default Traefik load balancing algorithm, which is round robin with an equal weighting applied to each web application instance.

Traefik configuration

In reference to the Solution routing architecture diagram, the Traefik request routing and load balancing architecture is comprised of four major components: EntryPoint, Router, Service, and Provider.

EntryPoints are the network entry points into the Traefik load balancer, which define the port and protocol on which to listen for incoming network packets. In the reference solution, the LoadBalancer Traefik instance is configured with a single HTTP EntryPoint listening on TCP port 80.

Services are responsible for configuring how to reach the actual application endpoints that handle the incoming requests. In the reference solution, we create and configure a Traefik Service named lb-svc-whoami that‘s configured to use the inbuilt Amazon ECS provider. Via the inbuilt Amazon ECS provider, Traefik automatically enumerates active application endpoints via the Amazon ECS control plane and update its routing rules in real-time — dynamically load balancing incoming requests across the pool of active application instances which may expand or contract in response to factors such as fluctuating load, application failure, or updates.

Routers analyze incoming requests, using rules to decide which Service to forward any given request to. Forwarding rules provide the ability to match traffic based on request characteristics such as host, path, headers, and method. Two Routers have been implemented in the example solution, each configured to forward traffic to the Traefik Service lb-svc-whoami:

• whoami-host is configured with a host based rule, which forwards incoming requests to the lb-svc-whoami service when the request domain (i.e., host header value) matches “whoami.domain.com”.
• whoami-path is configured with a path based rule, which forwards incoming requests to the lb-svc-whoami service when the HTTP request path matches “/whoami”.

Solution operation

In reference to the Solution operation diagram, the solution demonstrated in this blog post has the following key operational characteristics:

1. The Amazon ECS services LoadBalancer and Whoami are provisioned to the external instance by the Admin, who submits configuration and deployment requests to the Amazon ECS service in the AWS Region.
2. In communication with the Amazon ECS service, the Amazon ECS agent on the external instances launch the LoadBalancer and Whoami workloads via the local Docker API.
3. In communication with the Amazon ECS service, the LoadBalancer Traefik instance enumerates containers backing the Whoami service, and updates its routing table accordingly([2b], [2c]).
4. User-1 requests to “http://whoami.domain.com” match the Traefik host based route whoami-host and are load balanced across each of the whoami web application instances [A].
5. User-2 request to “http://domain.com/whoami” match the Traefik path based route whoami-path and are load balanced across each of the whoami web application instances [A].

Walkthrough

In this section, I’ll guide you in the process of implementing the load balanced whoami web application to Amazon ECS Anywhere. We configure the solution in accordance with the deployment scenario detailed in the Solution overview section. The solution walkthrough proceeds in the following order of operation:

1. Deploy the LoadBalancer service to external instance #1.
2. Deploy the Whoami service to external instances #2, and #3.
3. Demonstrate host based load balancing to the Whoami service.
4. Demonstrate path based load balancing to the Whoami service.
5. Cleaning up.

Prerequisites

To complete this walkthrough, you will need the following prerequisites:

• An AWS account with necessary permissions to create the resources.
• The AWS Command Line Interface (AWS CLI) installed and configured.
• An Amazon ECS Cluster with three registered Amazon ECS Anywhere external instances.
• For each external instance you register with an Amazon ECS cluster, it requires the AWS Systems Manager (SSM) Agent, the Amazon ECS container agent, and Docker installed. To register the external instance to an Amazon ECS cluster, it must first be registered as an AWS Systems Manager managed instance. You can generate the comprehensive installation script in a few clicks on the Amazon ECS console. Follow the instructions as described here in the Amazon ECS product documentation.
• An AWS Identity and Access Management (AWS IAM) role provisioned with appropriate policy to permit the Traefik proxy to read the required Amazon ECS attributes. You can follow this link for instructions on creating an AWS IAM role and associated policy. The Amazon ECS Task IAM role requires the following policy configuration in order to read required Amazon ECS information.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TraefikECSReadAccess",
            "Effect": "Allow",
            "Action": [
                "ecs:ListClusters",
                "ecs:DescribeClusters",
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:DescribeContainerInstances",
                "ecs:DescribeTaskDefinition",
                "ec2:DescribeInstances",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

1. Assign roles to external instances using Amazon ECS custom attributes

You can add custom metadata to your container instances, known as attributes. Each attribute has a name and an optional string value. To assist in targeting the deployment of our Amazon ECS services to specific external instances, we assign our external instances a logical role using custom attributes. The custom attributes are used to configure task placement constraints.

One of the external instances is assigned the role of loadbalancer. By following this work instruction, add the following custom attribute to one of your external instances:

• Name = role, Value = loadbalancer

The remaining two external instances are assigned the role of webserver. Add the following custom attribute to each of the remaining external instances:

• Name = role, Value = webserver

2. Deploy the LoadBalancer service to the external instance

Next we deploy the Traefik load balancer. We first create an Amazon ECS task definition, which describes the container configuration. Then we deploy the task definition to an external instance as an Amazon ECS service.

The following is an example JSON task definition, which contains the required configuration to implement the Traefik load balancer. Some points to note:

• The Amazon ECS launch type compatibility is set to EXTERNAL to ensure correct operation on an external instance.
• The task definition includes the placement constraint matching the loadbalancer custom attribute value.
• The Traefik web user interface has been enabled, and will be accessible via Transport Control Protocol (TCP) port 8080 on the external instance host IP address.

Note: Replace the string <TASK_ROLE_ARN> with the Amazon Resource Name (ARN) of the AWS IAM role configured with the TraefikECSReadAccess policy as configured in the prerequisites section.

{
    "family": "LoadBalancer",
    "cpu": "256",
    "memory": "128",
    "containerDefinitions": [
      {
        "name": "traefik",
        "image": "traefik:latest",
        "entryPoint": [],
        "portMappings": [
          {
            "hostPort": 80,
            "protocol": "tcp",
            "containerPort": 80
          },
          {
            "hostPort": 8080,
            "protocol": "tcp",
            "containerPort": 8080
          }
        ],
        "command": [
          "--api.dashboard=true",
          "--api.insecure=true",
          "--accesslog=true",
          "--providers.ecs.ecsAnywhere=true",
          "--providers.ecs.region=ap-southeast-2",
          "--providers.ecs.autoDiscoverClusters=true",
          "--providers.ecs.exposedByDefault=true"
        ]
      }
    ],
    "placementConstraints": [
      {
        "type": "memberOf",
        "expression": "attribute:role == loadbalancer"
      }
    ],
  "taskRoleArn": <TASK_ROLE_ARN>,
  "requiresCompatibilities": [
    "EXTERNAL"
  ]
}

Copy the example task definition JSON content to a file named “task-definition-LoadBalancer.json”, and register the task definition with your cluster using the following AWS CLI command:

aws ecs register-task-definition --cli-input-json file://task-definition-LoadBalancer.json

Next, create the LoadBalancer service using the LoadBalancer task definition by running the following AWS CLI command.

Note: Replace the string <CLUSTER_NAME> with the target Amazon ECS cluster name. This should be the cluster with which the external instance is registered.

aws ecs create-service \
    --cluster <CLUSTER_NAME> \
    --service-name LoadBalancer \
    --task-definition LoadBalancer:1 \
    --desired-count 1 \
    --launch-type EXTERNAL

You can use the following AWS CLI command to validate that the service has deployed correctly.

aws ecs describe-services —cluster <CLUSTER_NAME> —services LoadBalancer

The Traefik load balancer is now running on the external instance. You can access the Traefik web user interface by browsing to the following URL.

Note: Replace the string <HOST_IP> with the target Amazon ECS external instance host IP address, or DNS hostname.

http://<HOST_IP>:8080/dashboard/

3. Deploy the Whoami service to the webserver external instances

Next we deploy the example web application. First we create an Amazon ECS task definition, which describes the web application container configuration. Then we’ll deploy the task definition to our external instances as an Amazon ECS service.

The following is an example JSON task definition, containing the required configuration to implement the whoami web application. Some points to note:

• The Amazon ECS launch type compatibility is set to EXTERNAL to ensure correct operation on an external instance.
• The task definition includes the placement constraint matching the webserver custom attribute value.
• Docker label traefik.http.routers is used to configure host and path based routing rules.
• As the whoami container exposes the single TCP port 80, Docker label traefik.http.services.whoami is used to configure this port for private communication with the Traefik load balancer.

{
  "family": "Whoami",
  "cpu": "256",
  "memory": "128",
  "containerDefinitions": [
    {
      "name": "whoami",
      "image": "traefik/whoami:latest",
      "entryPoint": [],
      "portMappings": [
        {
          "hostPort": 0,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "command": [],
      "dockerLabels": {
        "traefik.http.services.whoami.loadbalancer.server.port": "80",
        "traefik.http.routers.whoami-host.rule": "Host(`whoami.domain.com`)",
        "traefik.http.routers.whoami-path.rule": "Path(`/whoami`)"
      }
    }
  ],
  "placementConstraints": [
    {
      "type": "memberOf",
      "expression": "attribute:role == webserver"
    }
  ],
  "volumes": [],
  "requiresCompatibilities": [
    "EXTERNAL"
  ]
}

Copy the example task definition JSON content to a file named “task-definition-Whoami.json”, and register the task definition with your cluster using the following AWS CLI command:

aws ecs register-task-definition --cli-input-json file://task-definition-Whoami.json

Then create the Whoami service using the Whoami task definition by running the following AWS CLI command.

Note: The directive --desired count 4 instructs the Amazon ECS service scheduler to schedule and maintain 4 running instances of the whoami task spread evenly across the two webserver external instances.

Note: Replace the string <CLUSTER_NAME> with the desired Amazon ECS cluster name. This should be the cluster with which the external instance is registered.

aws ecs create-service \
    --cluster <CLUSTER_NAME> \
    --service-name Whoami\
    --task-definition Whoami:1 \
    --desired-count 4 \
    --launch-type EXTERNAL

You can use the following AWS CLI command to validate that the service has deployed correctly.

Note: Replace the string <CLUSTER_NAME> with the target Amazon ECS cluster name.

aws ecs describe-services —cluster <CLUSTER_NAME> —services Whoami

The whoami web application is now running on the external instances. You can use the Traefik web user interface to inspect the solution configuration. To do this, first browse to the following address in order to inspect the whoami service configuration.

Note: Replace the string <HOST_IP> with the target Amazon ECS external instance host IP address, or DNS hostname.

http://<HOST_IP>:8080/dashboard/#/http/services/whoami@ecs

Per the following illustration, the Traefik web user interface provides an overview of the major components comprising the whoami service.

• In the [1] Service Details section we see the service type as loadbalancer, using the Amazon ECS provider.
• In the [2] Servers section are the x4 whoami web application instances distributed equally across the two webserver external instances, each listening on TCP port 80.
• In the [3] Routers section the two Traefik routers whoami-host and whoami-path are listed with their respective host and path based routing rules, and association with the Service whoami.

With the solution now in place, let’s now validate the traffic routing behavior by sending HTTP requests to the whoami web application using both the path and host based routing rules.

4. Demonstrate host based load balancing to the Whoami service

Using a terminal emulator, let’s send x4 HTTP requests to the host “whoami.domain.com”.

Note: For name resolution to function in the test scenario, update the local hosts file on the test client machine or a DNS A record for host “whoami.domain.com” as associated with the LAN IP address of the loadbalancer external instance. If required this can also be exposed to the public internet via your preferred firewall/gateway solution and publicly resolvable DNS zone.

Note: HTTP response data in the below examples has been truncated for brevity.

$ curl whoami.domain.com
Hostname: c87e8f82af9f
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 192.168.1.115:42752
GET / HTTP/1.1
Host: whoami.domain.com

$ curl whoami.domain.com
Hostname: 618e18ed985a
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 192.168.1.115:34496
GET / HTTP/1.1
Host: whoami.domain.com

$ curl whoami.domain.com
Hostname: 0fb436d99af7
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 192.168.1.115:55168
GET / HTTP/1.1
Host: whoami.domain.com

$ curl whoami.domain.com
Hostname: ab91f6ec8bcc
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 192.168.1.115:44678
GET / HTTP/1.1
Host: whoami.domain.com

The Hostname field in the curl response represents the container ID of the load balanced container responding to the HTTP request. We can see in the example output that the Traefik load balancer has forwarded each of the four HTTP requests evenly across each of the four whoami web application containers c87e8f82af9f, 618e18ed985a, 0fb436d99af7, and ab91f6ec8bcc by using the default round robin algorithm.

5. Demonstrate path based load balancing to the Whoami service

Next, using a terminal emulator send four HTTP requests to the resource <HOST_IP>/whoami.

Note: Replace the string <HOST_IP> with the target Amazon ECS external instance host IP address.

$ curl <HOST_IP>/whoami
Hostname: c87e8f82af9f
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 192.168.1.115:43674
GET /whoami HTTP/1.1
Host: 192.168.1.115

$ curl <HOST_IP>/whoami
Hostname: 618e18ed985a
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 192.168.1.115:35418
GET /whoami HTTP/1.1
Host: 192.168.1.115

$ curl <HOST_IP>/whoami
Hostname: 0fb436d99af7
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 192.168.1.115:56090
GET /whoami HTTP/1.1
Host: 192.168.1.115

$ curl <HOST_IP>/whoami
Hostname: ab91f6ec8bcc
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 192.168.1.115:45552
GET /whoami HTTP/1.1
Host: 192.168.1.115

And again in accordance with our solution configuration, the Traefik load balancer has forwarded each of the four path-based HTTP requests across each of the individual whoami web application containers c87e8f82af9f, 618e18ed985a, 0fb436d99af7, and ab91f6ec8bcc by using the round robin algorithm.

Cleaning up

In order to avoid incurring future charges, follow these procedures to delete the resources provisioned during the work instruction.

First, let’s delete the Amazon ECS services. To do so, run the following AWS CLI commands.

Note: Replace the string <CLUSTER_NAME> with the desired Amazon ECS cluster name. This should be the cluster with which the external instance is registered.

aws ecs delete-service --cluster <CLUSTER_NAME> --service Whoami --force
aws ecs delete-service --cluster <CLUSTER_NAME> --service LoadBalancer --force

Next, we deregister the external instances from the Amazon ECS cluster. To do so, run the following AWS CLI command for each external instance.

Note: Replace the string <CLUSTER_NAME> with the desired Amazon ECS cluster name. This should be the cluster with which the external instance is registered. Replace the string <INSTANCE_NAME> with the name of the external instance to be deregistered.

aws ecs deregister-container-instance \
    --cluster <CLUSTER_NAME> \
    --container-instance <INSTANCE_NAME>
    --force

To delete the Amazon ECS cluster, run the following AWS CLI command.

Note: Replace the string <CLUSTER_NAME> with the desired Amazon ECS cluster name.

aws ecs delete-cluster --cluster <CLUSTER_NAME>

Finally, follow this work instruction to delete the AWS IAM role and associated policy configured to permit the Traefik proxy to read required Amazon ECS attributes.

Conclusion

In this post, we showed you how to deploy and load balance a web application across multiple Amazon ECS Anywhere external instances, using the Traefik Proxy (Traefik) network load balancer. Implementing network load balancing for Amazon ECS Anywhere workloads such as frontend web applications, microservices, or APIs, is straightforward and effective when using the open source Traefik network load balancer. By incorporating the built-in Traefik ECS provider, you are able to dynamically discover and distribute requests across pools of active application instances as they expand or contract in response to factors such as fluctuating load, application failure, or updates. Additional support for encrypted traffic termination, automatic security certificate issuance and renewal, and additional load balancing capabilities such as circuit breakers, and rate limiting offer a comprehensive toolkit for securing and scaling your distributed workloads with Amazon ECS Anywhere.

To learn more, see Amazon ECS Anywhere in the Amazon ECS Developer Guide, and we encourage you to give it a try with the Amazon ECS Anywhere workshop as a next step.

Photo by Igor Omilaev on Unsplash.

Introduction

Kubernetes, with it's implementation of the cluster construct has simplified the ability to schedule workloads across a collection of VMs or nodes. Declarative configuration, immutability, auto-scaling, and self healing have vastly simplified the paradigm of workload management within the cluster - which has enabled teams to move at increasing velocities.

As the rate of Kubernetes adoption continues to increase, there has been a corresponding increase in the number of use cases that require workloads to break through the perimeter of the single cluster construct. Requirements concerning workload location/proximity, isolation, and reliability have been the primary catalyst for the emergence of deployment scenarios where a single logical workload will span multiple Kubernetes clusters:

• Location based concerns include network latency requirements (e.g. bringing the application as close to users as possible), data gravity requirements (e.g. bringing elements of the application as close to fixed data sources as possible), and jurisdiction based requirements (e.g. data residency limitations imposed via governing bodies);
• Isolation based concerns include performance (e.g. reduction in "noisy-neighbor" influence in mixed workload clusters), environmental (e.g. by staged or sandboxed workload constructs such as "dev", "test", and "prod" environments), security (e.g. separating untrusted code or sensitive data), organisational (e.g. teams fall under different business units or management domains), and cost based (e.g. teams are subject to separate budgetary constraints);
• Reliability based concerns include blast radius and infrastructure diversity (e.g. preventing an application based or underlying infrastructure issue in one cluster or provider zone from impacting the entire solution), and scale based (e.g. the workload may outgrow a single cluster)

Multi-cluster application architectures tend to be designed to either be replicated in nature - with this pattern each participating cluster runs a full copy of each given application; or alternatively they implement more of a group-by-service pattern where the services of a single application or system are split or divided amongst multiple clusters.

When it comes to the configuration of Kubernetes (and the surrounding infrastructure) to support a given multi-cluster application architecture - the space has evolved over time to include a number of approaches. Implementations tend draw upon a combination of components at various levels of the stack, and generally speaking they also vary in terms of the "weight" or complexity of the implementation, number and scope of features offered, as well as the associated management overhead. In simple terms these approaches can be loosely grouped into two main categories:

• Network-centric approaches focus on network interconnection tooling to implement connectivity between clusters in order to facilitate cross-cluster application communication. The various network-centric approaches include those that are tightly coupled with the CNI (e.g. Cillium Mesh), as well as more CNI agnostic implementations such as Submariner and Skupper. Service mesh implementations also fall into the network-centric category, and these include Istio’s multi-cluster support, Linkerd service mirroring, Kuma from Kong, AWS App Mesh, and Consul’s mesh gateway. There are also various multi-cluster ingress approaches, as well as virtual-kubelet based approaches including Admiralty, Tensile-kube, and Liqo.
• Kubernetes-centric approaches focus on supporting and extending the core Kubernetes primitives in order to support multi-cluster use cases. These approaches fall under the stewardship of the Kubernetes Multicluster Special Interest Group whose charter is focused on designing, implementing, and maintaining API’s, tools, and documentation related to multi-cluster administration and application management. Subprojects include:
  • kubefed (Kubernetes Cluster Federation) which implements a mechanism to coordinate the configuration of multiple Kubernetes clusters from a single set of APIs in a hosting cluster. kubefed is considered to be foundational for more complex multi-cluster use cases such as deploying multi-geo applications, and disaster recovery.
  • work-api (Multi-Cluster Works API) aims to group a set of Kubernetes API resources to be applied to one or multiple clusters together as a concept of “work” or “workload” for the purpose of multi-cluster workload lifecycle mangement.
  • mcs-api (Multi-cluster Services APIs) implements an API specification to extend the single-cluster bounded Kubernetes service concept to function across multiple clusters.

About the Multi-cluster Services API

Kubernetes' familiar Service object lets you discover and access services within the boundary of a single Kubernetes cluster. The mcs-api implements a Kubernetes-native extension to the Service API, extending the scope of the service resource concept beyond the cluster boundary - providing a mechanism to stitch your multiple clusters together using standard (and familiar) DNS based service discovery.

KEP-1645: Multi-Cluster Services API provides the formal description of the Multi Cluster Service API. KEP-1645 doesn't define a complete implementation - it serves to define how an implementation should behave.
At the time of writing the mcs-api version is: multicluster.k8s.io/v1alpha1

The primary deployment scenarios covered by the mcs-api include:

• Different services each deployed to separate clusters: I have 2 clusters, each running different services managed by different teams, where services from one team depend on services from the other team. I want to ensure that a service from one team can discover a service from the other team (via DNS resolving to VIP), regardless of the cluster that they reside in. In addition, I want to make sure that if the dependent service is migrated to another cluster, the dependee is not impacted.
• Single service deployed to multiple clusters: I have deployed my stateless service to multiple clusters for redundancy or scale. Now I want to propagate topologically-aware service endpoints (local, regional, global) to all clusters, so that other services in my clusters can access instances of this service in priority order based on availability and locality.

The mcs-api is able to support these use cases through the described properties of a ClusterSet, which is a group of clusters with a high degree of mutual trust and shared ownership that share services amongst themselves - along with two additional API objects: the ServiceExport and the ServiceImport.

Services are not visible to other clusters in the ClusterSet by default, they must be explicitly marked for export by the user. Creating a ServiceExport object for a given service specifies that the service should be exposed across all clusters in the ClusterSet. The mcs-api implementation (typically a controller) will automatically generate a corresponding ServiceImport object (which serves as the in-cluster representation of a multi-cluster service) in each importing cluster - for consumer workloads to be able to locate and consume the exported service.

DNS-based service discovery for ServiceImport objects is facilitated by the Kubernetes DNS-Based Multicluster Service Discovery Specification which extends the standard Kubernetes DNS paradigms by implementing records named by service and namespace for ServiceImport objects, but as differentiated from regular in-cluster DNS service names by using the special zone .clusterset.local. I.e. When a ServiceExport is created, this will cause a FQDN for the multi-cluster service to become available from within the ClusterSet. The domain name will be of the format <service>.<ns>.svc.clusterset.local.

AWS Cloud Map MCS Controller for Kubernetes

The AWS Cloud Map MCS Controller for Kubernetes (MCS-Controller) is an open source project that implements the multi-cluster services API specification.

The MCS-Controller is a controller that syncs services across clusters and makes them available for multi-cluster service discovery and connectivity. The implementation model is decentralsised, and utilises AWS Cloud Map as a registry for management and distribution of multi-cluster service data across clusters.

At the time of writing, the MCS-Controller release version is v0.3.0 which introduces new features including the ClusterProperty CRD, and support for headless services. Milestones are currently in place to bring the project up to v1.0 (GA), which will include full compliance with the mcs-api specification, support for multiple AWS accounts, and Cloud Map client-side traffic shaping.

AWS Cloud Map

AWS Cloud Map is a cloud resource discovery service that allows applications to discover web-based services via the AWS SDK, API calls, or DNS queries. Cloud Map is a fully managed service which eliminates the need to set up, update, and manage your own service discovery tools and software.

Tutorial

Overview

Let's consider a deployment scenario where we provision a Service into a single EKS cluster, then make the service available from within a second EKS cluster using the AWS Cloud Map MCS Controller.

This tutorial will take you through the end-end implementation of the solution as outlined herein, including a functional implementation of the AWS Cloud Map MCS Controller across x2 EKS clusters situated in separate VPCs.

The cloud-map-mcs-controller git repository provides a detailed work instruction and associated artefacts required for the end-end implementation of the AWS Cloud Map MCS Controller.

Solution Baseline

The Solution Baseline environment implements each of the prerequisites required in order to provision multi-cluster services.

In reference to the Solution Baseline diagram:

• We have x2 EKS clusters (Cluster 1 & Cluster 2), each deployed into separate VPCs within a single AWS region.
  • Cluster 1 VPC CIDR: 10.10.0.0/16, Kubernetes service IPv4 CIDR: 172.20.0.0/16
  • Cluster 2 VPC CIDR: 10.12.0.0/16, Kubernetes service IPv4 CIDR: 172.20.0.0/16
• VPC peering is configured to permit network connectivity between workloads within each cluster.
• The CoreDNS multicluster plugin is deployed to each cluster.
• The AWS Cloud Map MCS Controller for Kubernetes is deployed to each cluster.
• Clusters 1 & 2 are each configured as members of the same mcs-api ClusterSet.
  • Cluster 1 mcs-api ClusterSet: clusterset1, Cluster Id: cls1.
  • Cluster 2 mcs-api ClusterSet: clusterset1, Cluster Id: cls2.
• Clusters 1 & 2 are both provisioned with the namespace demo.
• Cluster 1 has a ClusterIP Service nginx-hello deployed to the demo namespace which frontends a x3 replica Nginx deployment nginx-demo.
  • Service | nginx-hello: 172.20.150.33:80
  • Endpoints | nginx-hello: 10.10.66.181:80,10.10.78.125:80,10.10.86.76:80

Service Provisioning

With the required dependencies in place, the admin user is able to create a ServiceExport object in Cluster 1 for the nginx-hello Service, such that the MCS-Controller implementation will automatically provision a corresponding ServiceImport in Cluster 2 for consumer workloads to be able to locate and consume the exported service.

In reference to the Service Provisioning diagram:

1. The administrator submits the request to the Cluster 1 Kube API server for a ServiceExport object to be created for ClusterIP Service nginx-hello in the demo Namespace.
2. The MCS-Controller in Cluster 1, watching for ServiceExport object creation provisions a corresponding nginx-hello service in the Cloud Map demo namespace. The Cloud Map service is provisioned with sufficient detail for the Service object and corresponding Endpoint Slice to be provisioned within additional clusters in the ClusterSet.
3. The MCS-Controller in Cluster 2 responds to the creation of the nginx-hello Cloud Map Service by provisioning the ServiceImport object and corresponding EndpointSlice objects via the Kube API Server.
4. The CoreDNS multicluster plugin, watching for ServiceImport and EndpointSlice creation provisions corresponding DNS records within the .clusterset.local zone.

Service Consumption

Here we deploy a client application, and consume the example multi-cluster service using native Kubernetes Service Discovery.

In reference to the Service Consumption diagram:

1. The client-hello pod in Cluster 2 needs to consume the nginx-hello service, for which all Endpoints are deployed in Cluster 1. The client-hello pod requests the resource http://nginx-hello.demo.svc.clusterset.local:80. DNS based service discovery [1b] responds with the IP address of the local nginx-hello ServiceExport Service ClusterSetIP.
2. Requests to the local ClusterSetIP at nginx-hello.demo.svc.clusterset.local are proxied to the Endpoints located on Cluster 1.

Note: In accordance with the mcs-api specification, a multi-cluster service will be imported by all clusters in which the service's namespace exists, meaning that each exporting cluster will also import the corresponding multi-cluster service. As such, the nginx-hello service will also be accessible via ServiceExport Service ClusterSetIP on Cluster 1. Identical to Cluster 2, the ServiceExport Service is resolvable by name at nginx-hello.demo.svc.clusterset.local.

Implementation

Solution Baseline

To prepare your environment to match the Solution Baseline deployment scenario, the following prerequisites should be addressed.

Clone the cloud-map-mcs-controller git repository

Sample configuration files will be used through the course of the tutorial, which have been made available in the cloud-map-mcs-controller repository.

Clone the repository to the host from which you will be bootstrapping the clusters:

git clone https://gitlab.com/byteQualia/cloud-map-mcs-controller.git

Note: All commands as provided should be run from the root directory of the cloned git repository.

Note: Certain values located within the provided configuration files have been configured for substitution with OS environment variables. Work instructions below will identify which environment variables should be set before issuing any commands which will depend on variable substitution.

Create EKS Clusters

x2 EKS clusters should be provisioned, each deployed into separate VPCs within a single AWS region.

• VPCs and clusters should be provisioned with non-overlapping CIDRs.
• For compatibility with the remainder of the tutorial, it is recommended that eksctl be used to provision the clusters and associated security configuration. By default, the eksctl create cluster command will create a dedicated VPC.

Sample eksctl config file /config/eksctl-cluster.yaml has been provided:

• Environment variables AWS_REGION, CLUSTER_NAME, NODEGROUP_NAME, and VPC_CIDR should be configured. Example values have been provided in the below command reference - substitute values to suit your preference.
• Example VPC CIDRs match the values provided in the Baseline Configuration description.

Run the following commands to create clusters using eksctl.

Cluster 1:

export AWS_REGION=ap-southeast-2
export CLUSTER_NAME=cls1
export NODEGROUP_NAME=cls1-nodegroup1
export VPC_CIDR=10.10.0.0/16
envsubst < config/eksctl-cluster.yaml | eksctl create cluster -f -

Cluster 2:

export AWS_REGION=ap-southeast-2
export CLUSTER_NAME=cls2
export NODEGROUP_NAME=cls2-nodegroup1
export VPC_CIDR=10.12.0.0/16
envsubst < config/eksctl-cluster.yaml | eksctl create cluster -f -

Create VPC Peering Connection

VPC peering is required to permit network connectivity between workloads provisioned within each cluster.

To create the VPC Peering connection, follow the instruction Create a VPC peering connection with another VPC in your account for guidance.

VPC route tables in each VPC require updating, follow the instruction Update your route tables for a VPC peering connection for guidance. For simplicity, it's recommended to configure route destinations as the IPv4 CIDR block of the peer VPC.

Security Groups require updating to permit cross-cluster network communication. EKS cluster security groups in each cluster should be updated to permit inbound traffic originating from external clusters. For simplicity, it's recommended the Cluster 1 & Cluster 2 EKS Cluster Security groups be updated to allow inbound traffic from the IPv4 CIDR block of the peer VPC.

The VPC Reachability Analyzer can be used to test and diagnose end-end connectivity between worker nodes within each cluster.

Enable EKS OIDC Provider

In order to map required Cloud Map AWS IAM permissions to the MCS-Controller Kubernetes service account, we need to enable the OpenID Connect (OIDC) identity provider in our EKS clusters using eksctl.

• Environment variables REGION and CLUSTERNAME should be configured.

Run the following commands to enable OIDC providers using eksctl.

Cluster 1:

export AWS_REGION=ap-southeast-2
export CLUSTER_NAME=cls1
eksctl utils associate-iam-oidc-provider \
    --region $AWS_REGION \
    --cluster $CLUSTER_NAME \
    --approve

Cluster 2:

export AWS_REGION=ap-southeast-2
export CLUSTER_NAME=cls2
eksctl utils associate-iam-oidc-provider \
    --region $AWS_REGION \
    --cluster $CLUSTER_NAME \
    --approve

Implement CoreDNS multicluster plugin

The CoreDNS multicluster plugin implements the Kubernetes DNS-Based Multicluster Service Discovery Specification which enables CoreDNS to lifecycle manage DNS records for ServiceImport objects. To enable the CoreDNS multicluster plugin within both EKS clusters, perform the following procedure.

Update CoreDNS RBAC

Run the following command against both clusters to update the system:coredns clusterrole to include access to additional multi-cluster API resources:

kubectl apply -f config/coredns-clusterrole.yaml

Update the CoreDNS configmap

Run the following command against both clusters to update the default CoreDNS configmap to include the multicluster plugin directive, and clusterset.local zone:

kubectl apply -f config/coredns-configmap.yaml

Update the CoreDNS deployment

Run the following command against both clusters to update the default CoreDNS deployment to use the container image ghcr.io/aws/aws-cloud-map-mcs-controller-for-k8s/coredns-multicluster/coredns:v1.8.4 - which includes the multicluster plugin:

kubectl apply -f config/coredns-deployment.yaml

Install the aws-cloud-map-mcs-controller-for-k8s

Configure MCS-Controller RBAC

Before the Cloud Map MCS-Controller is installed, we will first pre-provision the controller Service Account, granting IAM access rights AWSCloudMapFullAccess to ensure that the MCS Controller can lifecycle manage Cloud Map resources.

• Environment variable CLUSTER_NAME should be configured.

Run the following commands to create the MCS-Controller namespace and service accounts in each cluster.

Note: Be sure to change the kubectl context to the correct cluster before issuing commands.

Cluster 1:

export CLUSTER_NAME=cls1
kubectl create namespace cloud-map-mcs-system
eksctl create iamserviceaccount \
--cluster $CLUSTER_NAME \
--namespace cloud-map-mcs-system \
--name cloud-map-mcs-controller-manager \
--attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapFullAccess \
--override-existing-serviceaccounts \
--approve

Cluster 2:

export CLUSTER_NAME=cls2
kubectl create namespace cloud-map-mcs-system
eksctl create iamserviceaccount \
--cluster $CLUSTER_NAME \
--namespace cloud-map-mcs-system \
--name cloud-map-mcs-controller-manager \
--attach-policy-arn arn:aws:iam::aws:policy/AWSCloudMapFullAccess \
--override-existing-serviceaccounts \
--approve

Install the MCS-Controller

Now to install the MCS-Controller.

• Environment variable AWS_REGION should be configured.

Run the following command against both clusters to install the MCS-Controller latest release:

export AWS_REGION=ap-southeast-2
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release"

Assign mcs-api ClusterSet membership and Cluster identifier

To ensure that  ServiceExport and  ServiceImport objects propagate correctly between clusters, each cluster should be configured as a member of a single mcs-api ClusterSet (clusterset1 in our example deployment scenario), and should be assigned a unique mcs-api Cluster Id within the ClusterSet (cls1 & cls2 in our example deployment scenario).

• Environment variable CLUSTER_ID should be configured.
• Environment variable CLUSTERSET_ID should be configured.

Run the following commands to configure Cluster  Id and ClusterSet membership.

Cluster 1:

export CLUSTER_ID=cls1
export CLUSTERSET_ID=clusterset1
envsubst < config/mcsapi-clusterproperty.yaml | kubectl apply -f -

Cluster 2:

export CLUSTER_ID=cls2
export CLUSTERSET_ID=clusterset1
envsubst < config/mcsapi-clusterproperty.yaml | kubectl apply -f -

Create nginx-hello Service

Now that the clusters, CoreDNS and the MCS-Controller have been configured, we can create the demo namespace in both clusters and implement the nginx-hello Service and associated Deployment into Cluster 1.

Run the following commands to prepare the demo environment on both clusters.

Note: be sure to change the kubectl context to the correct cluster before issuing commands.

Cluster 1:

kubectl create namespace demo
kubectl apply -f config/nginx-deployment.yaml
kubectl apply -f config/nginx-service.yaml

Cluster 2:

kubectl create namespace demo

Service Provisioning

With the Solution Baseline in place, let's continue by implementing the Service Provisioning scenario. We'll create a ServiceExport object in Cluster 1 for the nginx-hello Service. This will trigger the Cluster 1 MCS-Controller to complete service provisioning and propagation into Cloud Map, and subsequent import and provisioning by the MCS-Controller in Cluster 2.

Create nginx-hello ServiceExport

Run the following command against Cluster 1 to to create the ServiceExport object for the nginx-hello Service:

kubectl apply -f config/nginx-serviceexport.yaml

Verify nginx-hello ServiceExport

Let's verify the ServiceExport creation has succeeded, and that corresponding objects have been created in Cluster 1, Cloud Map, and Cluster 2.

Cluster 1

Inspecting the MCS-Controller logs in Cluster 1, we see that the controller has detected the ServiceExport object, and created the corresponding demo Namespace and nginx-hello Service in Cloud Map:

$ kubectl logs cloud-map-mcs-controller-manager-5b9f959fc9-hmz88 -c manager --namespace cloud-map-mcs-system
{"level":"info","ts":1665108812.7046816,"logger":"cloudmap","msg":"namespace created","nsId":"ns-nlnawwa2wa3ajoh3"}
{"level":"info","ts":1665108812.7626762,"logger":"cloudmap","msg":"service created","namespace":"demo","name":"nginx-hello","id":"srv-xqirlhajwua5vkvo"}
{"level":"info","ts":1665108812.7627065,"logger":"cloudmap","msg":"fetching a service","namespace":"demo","name":"nginx-hello"}
{"level":"info","ts":1665108812.8299918,"logger":"cloudmap","msg":"registering endpoints","namespaceName":"demo","serviceName":"nginx-hello","endpoints":[{"Id":"tcp-10_10_86_76-80","IP":"10.10.86.76","EndpointPort":{"Name":"","Port":80,"TargetPort":"","Protocol":"TCP"},"ServicePort":{"Name":"","Port":80,"TargetPort":"80","Protocol":"TCP"},"ClusterId":"cls1","ClusterSetId":"clusterset1","ServiceType":"ClusterSetIP","ServiceExportCreationTimestamp":1665108776000,"Ready":true,"Hostname":"","Nodename":"ip-10-10-77-143.ap-southeast-2.compute.internal","Attributes":{"K8S_CONTROLLER":"aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)"}},{"Id":"tcp-10_10_66_181-80","IP":"10.10.66.181","EndpointPort":{"Name":"","Port":80,"TargetPort":"","Protocol":"TCP"},"ServicePort":{"Name":"","Port":80,"TargetPort":"80","Protocol":"TCP"},"ClusterId":"cls1","ClusterSetId":"clusterset1","ServiceType":"ClusterSetIP","ServiceExportCreationTimestamp":1665108776000,"Ready":true,"Hostname":"","Nodename":"ip-10-10-77-143.ap-southeast-2.compute.internal","Attributes":{"K8S_CONTROLLER":"aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)"}},{"Id":"tcp-10_10_78_125-80","IP":"10.10.78.125","EndpointPort":{"Name":"","Port":80,"TargetPort":"","Protocol":"TCP"},"ServicePort":{"Name":"","Port":80,"TargetPort":"80","Protocol":"TCP"},"ClusterId":"cls1","ClusterSetId":"clusterset1","ServiceType":"ClusterSetIP","ServiceExportCreationTimestamp":1665108776000,"Ready":true,"Hostname":"","Nodename":"ip-10-10-77-143.ap-southeast-2.compute.internal","Attributes":{"K8S_CONTROLLER":"aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)"}}]}

Using the AWS CLI we can verify Namespace and Service resources provisioned to Cloud Map by the Cluster 1 MCS-Controller:

$ aws servicediscovery list-namespaces
{
    "Namespaces": [
        {
            "Id": "ns-nlnawwa2wa3ajoh3",
            "Arn": "arn:aws:servicediscovery:ap-southeast-2:911483634971:namespace/ns-nlnawwa2wa3ajoh3",
            "Name": "demo",
            "Type": "HTTP",
            "Properties": {
                "DnsProperties": {
                    "SOA": {}
                },
                "HttpProperties": {
                    "HttpName": "demo"
                }
            },
            "CreateDate": "2022-10-07T02:13:32.310000+00:00"
        }
    ]
}
$ aws servicediscovery list-services
{
    "Services": [
        {
            "Id": "srv-xqirlhajwua5vkvo",
            "Arn": "arn:aws:servicediscovery:ap-southeast-2:911483634971:service/srv-xqirlhajwua5vkvo",
            "Name": "nginx-hello",
            "Type": "HTTP",
            "DnsConfig": {},
            "CreateDate": "2022-10-07T02:13:32.744000+00:00"
        }
    ]
}
$ aws servicediscovery discover-instances --namespace-name demo --service-name nginx-hello
{
    "Instances": [
        {
            "InstanceId": "tcp-10_10_78_125-80",
            "NamespaceName": "demo",
            "ServiceName": "nginx-hello",
            "HealthStatus": "UNKNOWN",
            "Attributes": {
                "AWS_INSTANCE_IPV4": "10.10.78.125",
                "AWS_INSTANCE_PORT": "80",
                "CLUSTERSET_ID": "clusterset1",
                "CLUSTER_ID": "cls1",
                "ENDPOINT_PORT_NAME": "",
                "ENDPOINT_PROTOCOL": "TCP",
                "HOSTNAME": "",
                "K8S_CONTROLLER": "aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)",
                "NODENAME": "ip-10-10-77-143.ap-southeast-2.compute.internal",
                "READY": "true",
                "SERVICE_EXPORT_CREATION_TIMESTAMP": "1665108776000",
                "SERVICE_PORT": "80",
                "SERVICE_PORT_NAME": "",
                "SERVICE_PROTOCOL": "TCP",
                "SERVICE_TARGET_PORT": "80",
                "SERVICE_TYPE": "ClusterSetIP"
            }
        },
        {
            "InstanceId": "tcp-10_10_66_181-80",
            "NamespaceName": "demo",
            "ServiceName": "nginx-hello",
            "HealthStatus": "UNKNOWN",
            "Attributes": {
                "AWS_INSTANCE_IPV4": "10.10.66.181",
                "AWS_INSTANCE_PORT": "80",
                "CLUSTERSET_ID": "clusterset1",
                "CLUSTER_ID": "cls1",
                "ENDPOINT_PORT_NAME": "",
                "ENDPOINT_PROTOCOL": "TCP",
                "HOSTNAME": "",
                "K8S_CONTROLLER": "aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)",
                "NODENAME": "ip-10-10-77-143.ap-southeast-2.compute.internal",
                "READY": "true",
                "SERVICE_EXPORT_CREATION_TIMESTAMP": "1665108776000",
                "SERVICE_PORT": "80",
                "SERVICE_PORT_NAME": "",
                "SERVICE_PROTOCOL": "TCP",
                "SERVICE_TARGET_PORT": "80",
                "SERVICE_TYPE": "ClusterSetIP"
            }
        },
        {
            "InstanceId": "tcp-10_10_86_76-80",
            "NamespaceName": "demo",
            "ServiceName": "nginx-hello",
            "HealthStatus": "UNKNOWN",
            "Attributes": {
                "AWS_INSTANCE_IPV4": "10.10.86.76",
                "AWS_INSTANCE_PORT": "80",
                "CLUSTERSET_ID": "clusterset1",
                "CLUSTER_ID": "cls1",
                "ENDPOINT_PORT_NAME": "",
                "ENDPOINT_PROTOCOL": "TCP",
                "HOSTNAME": "",
                "K8S_CONTROLLER": "aws-cloud-map-mcs-controller-for-k8s d07e680 (d07e680)",
                "NODENAME": "ip-10-10-77-143.ap-southeast-2.compute.internal",
                "READY": "true",
                "SERVICE_EXPORT_CREATION_TIMESTAMP": "1665108776000",
                "SERVICE_PORT": "80",
                "SERVICE_PORT_NAME": "",
                "SERVICE_PROTOCOL": "TCP",
                "SERVICE_TARGET_PORT": "80",
                "SERVICE_TYPE": "ClusterSetIP"
            }
        }
    ]
}

Cluster 2

Inspecting the MCS-Controller logs in Cluster 2, we see that the controller has detected the nginx-hello Cloud Map Service, and created the corresponding Kubernetes ServiceImport:

$ kubectl logs cloud-map-mcs-controller-manager-5b9f959fc9-v72s4 -c manager --namespace cloud-map-mcs-system
{"level":"info","ts":1665108822.2781157,"logger":"controllers.Cloudmap","msg":"created ServiceImport","namespace":"demo","name":"nginx-hello"}
{"level":"info","ts":1665108824.2420218,"logger":"controllers.Cloudmap","msg":"created derived Service","namespace":"demo","name":"imported-9cfu7k5mkr"}
{"level":"info","ts":1665108824.2501283,"logger":"controllers.Cloudmap","msg":"ServiceImport IPs need update","ServiceImport IPs":[],"cluster IPs":["172.20.80.119"]}
{"level":"info","ts":1665108824.2618752,"logger":"controllers.Cloudmap","msg":"updated ServiceImport","namespace":"demo","name":"nginx-hello","IP":["172.20.80.119"],"ports":[{"protocol":"TCP","port":80}]}

Inspecting the Cluster 2 Kubernetes ServiceImport object:

$ kubectl get serviceimports.multicluster.x-k8s.io nginx-hello -n demo -o yaml
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport
metadata:
  annotations:
    multicluster.k8s.aws/derived-service: '[{"cluster":"cls1","derived-service":"imported-9cfu7k5mkr"}]'
  creationTimestamp: "2022-10-07T02:13:42Z"
  generation: 2
  name: nginx-hello
  namespace: demo
  resourceVersion: "12787"
  uid: a53901af-57a8-49c7-aeb1-f67c4a44c2d2
spec:
  ips:
  - 172.20.80.119
  ports:
  - port: 80
    protocol: TCP
  type: ClusterSetIP
status:
  clusters:
  - cluster: cls1

And the corresponding Cluster 2 Kubernetes Endpoint Slice:

$ kubectl get endpointslices.discovery.k8s.io -n demo
NAME                        ADDRESSTYPE   PORTS   ENDPOINTS                               AGE
imported-9cfu7k5mkr-dc7q9   IPv4          80      10.10.78.125,10.10.86.76,10.10.66.181   14m

Important points to note:

• the ServiceImport Service is assigned an IP address from the local Kubernetes service IPv4 CIDR: 172.22.0.0/16 (172.20.80.119) so as to permit service discovery and access to the remote service endpoints from within the local cluster.
• the endpoint IP addresses match those of the nginx-demo Endpoints in Cluster 1 (i.e. from the Cluster 1 VPC CIDR: 10.10.0.0/16).

Service Consumption

With the Solution Baseline and Service Provisioning in place, workloads in Cluster 2 are now able to consume the nginx-hello Service Endpoints located in Cluster 1 via the locally provisioned ServiceImport object. To complete the Service Consumption deployment scenario we'll deploy the client-hello Pod into Cluster 2, and observe how it's able to perform cross-cluster service consumption of the nginx-hello Service Endpoints in Cluster 1.

Create client-hello Pod

Run the following command against Cluster 2 create the client-hello Pod:

kubectl apply -f config/client-hello.yaml

Verify multi-cluster service consumption

Let's exec into the client-hello Pod and perform an nslookup to cluster-local CoreDNS for the ServiceImport Service nginx-hello.demo.svc.clusterset.local:

$ kubectl exec -it client-hello -n demo /bin/sh
/ # nslookup nginx-hello.demo.svc.clusterset.local
Server:         172.20.0.10
Address:        172.20.0.10:53

Name:   nginx-hello.demo.svc.clusterset.local
Address: 172.20.80.119

Note that the Pod resolves the address of the ServiceImport object on Cluster 2.

Finally, we generate HTTP requests from the client-hello Pod to the local nginx-hello ServiceImport Service:

/ # apk --no-cache add curl
/ # curl nginx-hello.demo.svc.clusterset.local
Server address: 10.10.86.76:80
Server name: nginx-demo-59c6cb8d7b-m4ktw
Date: 07/Oct/2022:02:31:45 +0000
URI: /
Request ID: 17d43e6e8801a98d05059dfaf88d0abe
/ # 
/ # curl nginx-hello.demo.svc.clusterset.local
Server address: 10.10.78.125:80
Server name: nginx-demo-59c6cb8d7b-8w6rp
Date: 07/Oct/2022:02:32:26 +0000
URI: /
Request ID: 0ddc09ffe7fd45c52903ce34c955f555
/ # 
/ # curl nginx-hello.demo.svc.clusterset.local
Server address: 10.10.66.181:80
Server name: nginx-demo-59c6cb8d7b-mtm8l
Date: 07/Oct/2022:02:32:53 +0000
URI: /
Request ID: 2fde1c34008a5ec18b8ae23797489c3a

Note that the responding Server Names and Server addresses are those of the nginx-demo Pods on Cluster 1 - confirming that the requests to the local ClusterSetIP at nginx-hello.demo.svc.clusterset.local originating on Cluster 2 are proxied cross-cluster to the Endpoints located on Cluster 1!

Conclusion

The proliferation of container adoption is presenting new challenges in supporting workloads that have broken through the perimeter of the single cluster construct.

For teams that are looking to implement a Kubenetes-centric approach to managing multi-cluster workloads, the mcs-api describes an effective approach to extending the scope of the service resource concept beyond the cluster boundary - providing a mechanism to weave multiple clusters together using standard (and familiar) DNS based service discovery.

The AWS Cloud Map MCS Controller for Kubernetes is an open source project that integrates with AWS Cloud Map to offer a decentralised implementation of the multi-cluster services API specification that's particularly suited for teams looking for a lightweight and effective Kubenetes-centric mechanism to deploy multi-cluster workloads to the AWS cloud.

Photo by Susan Wilkinson on Unsplash

The eINS has been designed to provide an additional layer of resilience for ECS external instances in deployment scenarios where connectivity to the on-region ECS control-plane may be unreliable or intermittent.

Deploying the eINS to ECS external instances will ensure that during periods where there is a loss of connectivity to the on-region ECS control-plane, any ECS managed containers which exit due to error will be automatically restarted.

The ecs-external-instance-network-sentry git repository contains the application source code and end-user guide for configuring and deploying the ECS External Instance Network Sentry (eINS).

Introduction

ECS Anywhere is an extension of Amazon ECS that will allow customers to deploy native Amazon ECS tasks in any environment. This includes the existing model on AWS managed infrastructure, as well as customer-managed infrastructure.

When extending ECS to customer managed infrastructure, external instances are registered to the ECS cluster. External instances are compute resources (hosts) external to an AWS region where ECS can schedule tasks to run. External instances are typically an on-premises server or virtual machine (VM).

During normal operation, where there is network connectivity between an ECS external instance and the on-region ECS control-plane - ECS monitors for errors or failures that occur to managed containers running on external instances and will restart any containers which have stopped due to an error.

For the duration of time that an ECS external instance loses network connectivity to the ECS on-region control-plane, any managed containers which have stopped due to an error will not be restarted by ECS until the point in time that network connectivity to the ECS on-region control-plane has been restored.

The eINS has been designed to detect any loss of connectivity to the on-region ECS control-plane, and to proactively ensure that for the duration of the outage that ECS managed containers which stop due to an error condition are automatically restarted.

Overview

The eINS is a Python application which can either be run manually, or be configured to run as a service on ECS Anywhere external instances. See the Installation section below for instruction for both deployment scenarios.

Connected Operation

The eINS periodically attempts to establish a TLS connection with the ECS on-region control-plane to determine region availability status, and the on-region ECS control-plane responds without error.

In reference to the diagram:

• eINS TLS connection with the ECS on-region control-plane [1] completes successfully:
  • eINS takes no further action.
• In communication with the on-region control-plane [2] the ECS agent on the external instance orchestrates local managed container lifecycle, including restarting containers which exit due to error condition [3].

Disconnected Operation

The eINS periodically attempts to establish a TLS connection with the ECS on-region control-plane to determine region availability status, and the on-region ECS control-plane either does not respond, or returns an error.

In reference to the diagram:

• eINS TLS connection with the ECS on-region control-plane [1] experiences timeout or return error condition:
  • The ECS agent is paused [3] via the local Docker API [2]*.
  • eINS updates Docker restart policy to on-failure for each ECS managed container [4]. This ensures that containers exiting with an error code will be automatically restarted by the Docker daemon.
• When the ECS control-plane becomes reachable:
  • ECS managed containers that have been automatically restarted by the Docker daemon during network outage are stopped and removed.**
  • ECS managed containers that have not been automatically restarted during network outage have their Docker restart policy set back to no.
  • The local ECS agent is un-paused.

    At this point the operational environment has been restored back to the Connected Operation scenario. eINS will continue to monitor for network outage or ECS control-plane error.

Notes

*ECS agent is paused, as if left in a running state the agent will detect and kill ECS managed containers that have been restarted by the Docker daemon during the period of network outage.

**These containers are stopped and removed by eINS to avoid duplication:

• Containers that have been restarted by the Docker daemon during a network outage become orphaned by ECS once back online.
• The related ECS tasks are re-launched by ECS on the external instance once the ECS agent has established communication with the control-plane.

Configuration Parameters

The eINS provides the ability to submit configuration parameters as command line arguments. Running the application with the --help parameter generates a summary of available parameters:

$ python3 ecs-external-instance-network-sentry.py --help
usage: ecs-external-instance-network-sentry [-h] -r REGION [-i INTERVAL] [-n RETRIES] [-l LOGFILE] [-k LOGLEVEL]

Purpose:
--------------
For use on ECS Anywhere external hosts:
Configures ECS orchestrated containers to automatically restart
on failure when on-region ecs control-plane is detected to be unreachable.

Configuration Parameters:
--------------
  -h, --help            Show this help message and exit.
  -r REGION, --region REGION
                        AWS region where ecs cluster is located.
  -i INTERVAL, --interval INTERVAL
                        Interval in seconds sentry will sleep between connectivity checks.
  -n RETRIES, --retries RETRIES
                        Number of times Docker will restart a crashing container.
  -l LOGFILE, --logfile LOGFILE
                        Logfile name & location.
  -k LOGLEVEL, --loglevel LOGLEVEL
                        Log data event severity.

Configuration parameters are described in further detail following:

--region
Provide the name of the AWS region where the ECS cluster that manages the external instance is hosted. eINS will attempt to establish a TLS connection to the ECS public endpoint at the nominated region to evaluate ECS control-plane availability.

• optional=no
• default=""

$ python3 ecs-external-instance-network-sentry.py --region ap-southeast-2

--interval
Specify the number of seconds between connectivity tests.

• optional=yes
• default=20

$ python3 ecs-external-instance-network-sentry.py --region ap-southeast-2 --interval 15

--retries
Specify the number of times failing containers will be restarted during periods where the ECS control-plane is unavailable. The default setting is 0 which configures the Docker daemon to restart containers an unlimited number of times.

• optional=yes
• default=0

$ python3 ecs-external-instance-network-sentry.py --region ap-southeast-2 --interval 15 --retries 5

--logfile
Specify logfile name and file-system path. The default value is /tmp/ecs-anywhere-network-sentry.log.

• optional=yes
• default=/tmp/ecs-external-instance-network-sentry.log

$ python3 ecs-external-instance-network-sentry.py --region ap-southeast-2 --interval 15 --retries 5 --logfile /mypath/myfile.log

--loglevel
Specify log data event severity.

• optional=yes
• default=INFO

$ python3 ecs-external-instance-network-sentry.py --region ap-southeast-2 --interval 15 --retries 5 --logfile /mypath/myfile.log --loglevel DEBUG

Installation

It's recommended that the external instance first be registered with ECS before installing the eINS. Installation instructions for eINS are provided below in the correct order of precedence.

Commands provided assume that the external instance host operating system is Ubuntu 20.

Prerequisites

The following prerequisites should be implemented prior to deploying the eINS.

ECS Anywhere

For each external instance you register with an Amazon ECS cluster, it requires the SSM Agent, the Amazon ECS container agent, and Docker installed. To register the external instance to an Amazon ECS cluster, it must first be registered as an AWS Systems Manager managed instance. You can generate the comprehensive installation script in a few clicks on the Amazon ECS console. Follow the instructions as described here.

Python

The eINS has been developed and tested running on Python version 3.8.10.

Python Docker SDK

The eINS interacts with the Docker API, which requires installation of the Python Docker SDK on each external instance where the eINS will run. To install the Python Docker SDK, run the commands as follows:

# update package index files..
$ apt get update
# install python docker sdk..
$ python3 pip install docker

Clone the eINS git repository

On the ECS external instance, clone the ecs-external-instance-network-sentry repository:

# clone eins git repo..
$ git clone https://github.com/aws-samples/ecs-external-instance-network-sentry.git

Commands from this point forward will assume that you are in the root directory of the local git repository clone.

Manual Operation

At this point the external instance host operating system is ready to run the eINS. For testing or evaluation the application can be launched manually.

The application is located within the /python directory of the git repository. See the Configuration Parameters section for required and optional parameters to be submitted at runtime. Remember to provide the correct AWS region code:

# manual launch..
$ python3 /python/ecs-external-instance-network-sentry.py --region ap-southeast-2

Background Service

Configuring the application as an OS background service is an effective mechanism to ensure that the eINS remains running in the background at all times.

Service configuration requires the implementation of a unit configuration file which encodes information about the process that will be controlled and supervised by systemd.

Configuration Procedure

The following describes configuring the eINS as an OS background service.

Copy application and configuration files

Run the following commands to copy application and configuration files to the appropriate locations on the external instance file system:

# copy eins application file..
$ cp /python/ecs-external-instance-network-sentry.py /usr/bin
# copy eins service unit config file..
$ cp /config/ecs-external-instance-network-sentry.service /lib/systemd/system

Update service unit configuration file

Next, update the service unit configuration file /lib/systemd/system/ecs-external-instance-network-sentry.service.

$ cat /lib/systemd/system/ecs-external-instance-network-sentry.service

[Unit]
Description=Amazon ECS External Instance Network Service Documentation=https://github.com/aws-samples/ecs-external-instance-network-sentry Requires=docker.service
After=ecs.service

[Service]
Type=simple
Restart=on-failure RestartSec=10s
ExecStart=python3 /usr/bin/ecs-external-instance-network-sentry.py --region <INSERT-REGION-NAME-HERE>
[Install] WantedBy=multi-user.target

Make necessary modifications to the service unit config file ExecStart directive on line-11 as follows:

• Update the --region configuration parameter with the AWS region name where your on-region ECS cluster is provisioned.
• Optionally, include any additional Configuration Parameters to suit the particular requirements of your deployment scenario.

Configure and start service

# reload systemd..
$ systemctl daemon-reload # enable eins service..
$ sudo systemctl enable ecs-external-instance-network-sentry.service
# start eins service..
$ systemctl start ecs-external-network-sentry

Check service status

To validate that the service has started successfully, run the following command. If the service has started correctly, the output should be similar to the following:

$ systemctl status ecs-external-instance-network-sentry

● ecs-external-instance-network-sentry.service - Amazon ECS External Instance Network Service
     Loaded: loaded (/lib/systemd/system/ecs-external-instance-network-sentry.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-07-30 07:57:08 UTC; 22min ago
       Docs: https://github.com/aws-samples/ecs-external-instance-network-sentry
   Main PID: 28366 (python3)
      Tasks: 1 (limit: 9412)
     Memory: 19.7M
     CGroup: /system.slice/ecs-external-instance-network-sentry.service
             └─28366 /usr/bin/python3 /usr/bin/ecs-external-instance-network-sentry.py --region ap-southeast-2 --interval 10 --retries 3 --logfile /tmp/ecs->

Jul 30 07:57:08 ubu20 systemd[1]: Started Amazon ECS External Instance Network Service.

Logging

The eINS has been configured to provide basic logging regarding it's operation.

The default logfile location is /tmp/ecs-external-instance-network-sentry.log, which can be modified by submitting the --logfile configuration parameter.

Log Level

By default, the loglevel is set to logging.INFO and can be updated at runtime using the --loglevel configuration parameter.

Log Output

The following eINS logfile excerpt illustrates;

• A detected loss of connectivity to on-region control-plane, and associated Docker policy configuration actions for ECS managed containers.
• Container cleanup and Docker policy configuration once ECS control-plane becomes reachable.

2021-07-10 09:00:01,200 INFO PID_713928 [startup] ecs-external-instance-network-sentry - starting..
2021-07-10 09:00:01,200 INFO PID_713928 [startup] arg - aws region: ap-southeast-2
2021-07-10 09:00:01,200 INFO PID_713928 [startup] arg - interval: 10
2021-07-10 09:00:01,201 INFO PID_713928 [startup] arg - retries: 0
2021-07-10 09:00:01,201 INFO PID_713928 [startup] arg - logfile: /tmp/ecs-external-instance-network-sentry.log
2021-07-10 09:00:01,201 INFO PID_713928 [startup] arg - loglevel: logging.INFO
...
...
2021-07-10 09:39:33,756 INFO PID_713928 [begin] connectivity test..
2021-07-10 09:39:33,757 INFO PID_713928 [connect] connecting to ecs at ap-southeast-2..
2021-07-10 09:39:33,757 INFO PID_713928 [connect] create network socket..
2021-07-10 09:39:43,764 ERROR PID_713928 [connect] error creating network socket: [Errno -3] Temporary failure in name resolution
2021-07-10 09:39:43,764 INFO PID_713928 [connect] connecting to host..
2021-07-10 09:39:43,765 INFO PID_713928 [ecs-offline] ecs unreachable, configuring container restart policy..
2021-07-10 09:39:43,880 INFO PID_713928 [ecs-offline] container name: ecs-alpine-crash-test-9adba798f5f189968701
2021-07-10 09:39:43,881 INFO PID_713928 [ecs-offline] ecs cluster: ecs-anywhere-cluster-1
2021-07-10 09:39:43,882 INFO PID_713928 [ecs-offline] set container restart policy: {'Name': 'on-failure', 'MaximumRetryCount': 0}
2021-07-10 09:39:43,958 INFO PID_713928 [ecs-offline] container name: ecs-nginx-1-nginx-eaa6e7a9b0cd88988201
2021-07-10 09:39:43,959 INFO PID_713928 [ecs-offline] ecs cluster: ecs-anywhere-cluster-1
2021-07-10 09:39:43,959 INFO PID_713928 [ecs-offline] set container restart policy: {'Name': 'on-failure', 'MaximumRetryCount': 0}
2021-07-10 09:39:44,022 INFO PID_713928 [ecs-offline] ecs agent paused..
2021-07-10 09:39:44,022 INFO PID_713928 [end] sleeping for 10 seconds..
...
...
2021-07-10 09:41:14,298 INFO PID_713928 [begin] connectivity test..
2021-07-10 09:41:14,299 INFO PID_713928 [connect] connecting to ecs at ap-southeast-2..
2021-07-10 09:41:14,299 INFO PID_713928 [connect] create network socket..
2021-07-10 09:41:23,133 INFO PID_713928 [connect] connecting to host..
2021-07-10 09:41:23,258 INFO PID_713928 [connect] send/receive data..
2021-07-10 09:41:30,563 INFO PID_713928 [connect] ecs at ap-southeast-2 is available..
2021-07-10 09:41:30,564 INFO PID_713928 [ecs-online] ecs is reachable..
2021-07-10 09:41:30,621 INFO PID_713928 [ecs-online] container name: ecs-alpine-crash-test-9adba798f5f189968701
2021-07-10 09:41:30,621 INFO PID_713928 [ecs-online] ecs cluster: ecs-anywhere-cluster-1
2021-07-10 09:41:30,622 INFO PID_713928 [ecs-online] container has been restarted by docker, stopping & removing..
2021-07-10 09:41:41,330 INFO PID_713928 [ecs-online] container name: ecs-nginx-1-nginx-eaa6e7a9b0cd88988201
2021-07-10 09:41:41,330 INFO PID_713928 [ecs-online] ecs cluster: ecs-anywhere-cluster-1
2021-07-10 09:41:41,331 INFO PID_713928 [ecs-online] set container restart policy: {'Name': 'no', 'MaximumRetryCount': 0}
2021-07-10 09:41:41,470 INFO PID_713928 [ecs-online] ecs agent unpaused..
2021-07-10 09:41:41,471 INFO PID_713928 [end] sleeping for 10 seconds..

Log Rotation

Logfile will rotate at 5Mb and a history of the five most recent logfiles will be maintained.

Considerations

The eINS currently has the following limitations:

• During periods where the ECS control-plane is unavailable, and there is either an external instance OS reboot or Docker daemon restart: eINS will not start previously running ECS managed containers.
• As described in the Disconnected Operation section, containers that have been restarted during a period where the ECS control-plane is unavailable will be stopped once the ECS control-plane becomes available.

Photo by Solen Feyissa on Unsplash

Introduction

Containers have introduced a paradigm shift in how we work with applications, and due to the additional efficiencies in deployment, packaging, and development, the rate of adoption has been skyrocketing.

Many people new to containerisation tend to adopt the mental model that containers are simply a better and faster way of of running virtual machines (VMs). In many respects this analogy holds up (albeit from a very simplistic point of view), however from a security perspective, the two technologies provide a very different posture.

Standard Linux containers allow applications to make system calls directly to the host operating system (OS) kernel, in a similar way that non-containerised applications do - whereas in a VM environment, processes in a virtual machine simply do not have visibility of the host OS kernel.

If you're not running untrusted code in your containers, or hosting a multi-tenant platform; and you've implemented good security practices for the services running within each container, you probably don't need to worry.

But for those of us that are faced with the challenge of needing to run untrusted code in our containers, or perhaps are hosting a multi-tenant platform - providing the highest levels of isolation between workloads in a Kubernetes environment can be challenging.

An effective approach to improve workload isolation is to run each Pod within its own dedicated VM. This provides each Pod with a dedicated hypervisor, OS kernel, memory, and virtualized devices which are completely separate from the host OS. In this deployment scenario, when there's a vulnerability in the containerised workload - the hypervisor within the Pod provides a security boundary which protects the host operating system, as well as other workloads running on the host.

If you're running on the AWS cloud, Amazon have made this approach very simple. Scheduling Pods using the managed Kubernetes service EKS with Fargate actually ensures that each Kubernetes Pod is automatically encapsulated inside it's own dedicated VM. This provides the highest level of isolation for each containerised workload.

If you need to provide a similar level of workload isolation as EKS with Fargate when operating outside of the AWS cloud (e.g. on premises, or at the edge in a hybrid deployment scenario), then Kata Containers is a technology worth considering. Kata Containers is an implementation of a lightweight VM that seamlessly integrates with the container ecosystem, and can be used by Kubernetes to schedule Pods inside of VMs.

The following tutorial will take you through a deployment scenario where we bootstrap a Kubernetes cluster using Amazon EKS Distro (EKS-D), and configure Kubernetes to be capable of scheduling Pods inside VMs using Kata Containers.

This is a deployment pattern that can be adopted to provide a very high degree of workload isolation when provisioning clusters outside of the AWS cloud, for example on-premises, edge locations, or on alternate cloud platforms:

• EKS-D provides the same software that has enabled tens of thousands of Kubernetes clusters on Amazon EKS. This includes the latest upstream updates, as well as extended security patching support.
• In-cluster workload isolation is further enhanced by providing the ability to schedule Pods inside a dedicated VM using Kata Containers.

About Kata Containers

Kata Containers utilizes open source hypervisors as an isolation boundary for each container (or collection of containers in a Pod).

With Kata Containers, a second layer of isolation is created on top of those provided by traditional namespace containers. The hardware virtualization interface is the basis of this additional layer. Kata launches a lightweight virtual machine, and uses the VM guest’s Linux kernel to create a container workload, or workloads in the case of multi-container Pods. In Kubernetes and in the Kata implementation, the sandbox is implemented at the Pod level. In Kata, this sandbox is created using a virtual machine.

Kata currently supports multiple hypervisors, including: QEMU/KVM, Cloud Hypervisor/KVM, and Firecracker/KVM.

Kata Containers with Kubernetes

Kubernetes Container Runtime Interface (CRI) implementations allow using any OCI-compatible runtime with Kubernetes, such as the Kata Containers runtime. Kata Containers support both the CRI-O and CRI-containerd CRI implementations.

Kata Containers 1.5 introduced the shimv2 for containerd 1.2.0, reducing the components required to spawn Pods and containers. This is currently the preferred way to run Kata Containers with Kubernetes.

When configuring Kubernetes to integrate with Kata, typically a Kubernetes RuntimeClass is created. The RuntimeClass provides the ability to select the container runtime configuration to be used for a given workload via the Pod spec submitted to the Kubernetes API.

About Amazon EKS Distro

Amazon EKS Distro is a Kubernetes distribution used by Amazon EKS to help create reliable and secure clusters. EKS Distro includes binaries and containers from open source Kubernetes, etcd (cluster configuration database), networking, storage, and plugins, all tested for compatibility. You can deploy EKS Distro wherever your applications need to run.

You can deploy EKS Distro clusters and let AWS take care of testing and tracking Kubernetes updates, dependencies, and patches. The source code, open source tools, and settings are provided for consistent, reproducible builds. EKS Distro provides extended support for Kubernetes, with builds of previous versions updated with the latest security patches. EKS Distro is available as open source on GitHub.

Tutorial

Overview

This tutorial will guide you through the following procedure:

• Installing Kata Containers onto a bare metal host
• Installing and configuring containerd to integrate with Kata Containers
• Bootstrapping an EKS Disto Kubernetes cluster using kubeadm
• Configuring a Kubernetes RuntimeClass to schedule Pods to Kata VMs running the QEMU/KVM hypervisor

The example EKS-D cluster deployment uses kubeadm to bring up the control-plane, which may not be your preferred method to bootstrap a cluster in an environment outside of a managed cloud provider.  A number of AWS partners are also providing installation support for EKS Distro, including: Canonical (MicroK8s), Kubermatic (KubeOne), Kubestack, Nirmata, Rancher, and Weaveworks. For further information, see the Partners section at the EKS Distro website.

Prerequisites

Kubeadm is a tool built to provide kubeadm init and kubeadm join as best-practice "fast paths" for creating Kubernetes clusters.

You will need to use a Linux system that kubeadm supports, as described in the kubeadm documentation to verify that the system has the required amount of memory, CPU, and other resources.

Kata Containers requires nested virtualization or bare metal. Review the hardware requirements to see if your system is capable of running Kata Containers.

Clone the eksd-kata-containers git repository

Sample configuration files will be used through the course of the tutorial, which have been made available within the eksd-kata-containers repository.

Clone the eksd-kata-containers repository to the host on which you will be bootstrapping the cluster:

git clone https://gitlab.com/byteQualia/eksd-kata-containers.git

Bootstrap the Cluster

Next, we bootstrap the cluster using kubeadm.

Prepare the host

Make sure SELinux is disabled by setting SELINUX=disabled in the /etc/sysconfig/selinux file. To turn it off immediately, type:

sudo setenforce 0

Make sure that swap is disabled and that no swap areas are reinstated on reboot. For example, type:

sudo swapoff -a

Permanently disable swap by commenting out or deleting any swap areas in /etc/fstab.

Depending on the exact Linux system you installed, you may need to install additional packages. For example, with an RPM-based (Amazon Linux, CentOS, RHEL or Fedora), ensure that the iproute-tc, socat, and conntrack-tools packages are installed.

To optionally enable a firewall, run the following commands, including opening ports required by Kubernetes:

sudo yum install firewalld -y
sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo firewall-cmd --zone=public --permanent --add-port=6443/tcp --add-port=2379-2380/tcp --add-port=10250-10252/tcp

Install container runtime, Kata Containers, and supporting services

Next we need to install a container runtime (containerd in this example), Kata Containers, and the EKS-D versions of Kubernetes software components.

It's recommended that for production environments both the containerd runtime and Kata Containers are installed using official distribution packages. In this example we will utilise Kata Manager, which will perform a scripted installation of both components:

repo="github.com/kata-containers/tests"
go get -d "$repo"
PATH=$PATH:$GOPATH/src/${repo}/cmd/kata-manager
kata-manager.sh install-packages

Once installed, update the system path to include Kata binaries:

sudo su
PATH=$PATH:/opt/kata/bin/
echo "PATH=$PATH:/opt/kata/bin/" >> .profile
exit

Verify the host is capable of running Kata Containers:

kata-runtime kata-check

Example output generated on a supported system will read as similar to the following:

sudo kata-runtime kata-check
WARN[0000] Not running network checks as super user      arch=amd64 name= pid=9064 source=runtime
System is capable of running Kata Containers
System can currently create Kata Containers

Configure container runtime for Kata

cri is a native plugin of containerd 1.1 and above, and it's built into containerd and enabled by default. In order to configure containerd to schedule Kata containers, you need to update the containerd configuration file located at  /etc/containerd/config.toml with the following configuration which includes three runtime classes:

• plugins.cri.containerd.runtimes.runc: the runc, and it is the default runtime
• plugins.cri.containerd.runtimes.kata: The function in containerd (reference the document here) where the dot-connected string io.containerd.kata.v2 is translated to containerd-shim-kata-v2 (i.e. the binary name of the Kata implementation of Containerd Runtime V2 (Shim API)).
• plugins.cri.containerd.runtimes.katacli: the containerd-shim-runc-v1 calls kata-runtime, which is the legacy process.

Example config.toml:

plugins.cri.containerd]
 no_pivot = false
plugins.cri.containerd.runtimes]
 [plugins.cri.containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v1"
    [plugins.cri.containerd.runtimes.runc.options]
      NoPivotRoot = false
      NoNewKeyring = false
      ShimCgroup = ""
      IoUid = 0
      IoGid = 0
      BinaryName = "runc"
      Root = ""
      CriuPath = ""
      SystemdCgroup = false
 [plugins.cri.containerd.runtimes.kata]
    runtime_type = "io.containerd.kata.v2"
[plugins.cri.containerd.runtimes.kata.options]
  ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
 [plugins.cri.containerd.runtimes.katacli]
    runtime_type = "io.containerd.runc.v1"
    [plugins.cri.containerd.runtimes.katacli.options]
      NoPivotRoot = false
      NoNewKeyring = false
      ShimCgroup = ""
      IoUid = 0
      IoGid = 0
      BinaryName = "/opt/kata/bin/kata-runtime"
      Root = ""
      CriuPath = ""
      SystemdCgroup = false

From the eksd-kata-containers repository, copy the file /config/config.toml to /etc/containerd/config.toml and restart containerd:

sudo systemctl stop containerd
sudo cp eksd-kata-containers/config/config.toml /etc/containerd/
sudo systemctl start containerd

containerd is now able to run containers using the Kata Containers runtime.

Test Kata with containerd

In order to test that containerd can successfully run a Kata container, a shell script named test-kata.sh has been provided in the script directory within the eksd-kata-containers repository.

test-kata.sh uses the ctr CLI util to pull and run a busybox image as a Kata container, and retrieves the kernel version from within the Kata VM. The script returns both the kernel version reported by busybox from within the Kata VM, as well as the host OS kernel version. Per the sample output, the container kernel (hosted in the VM) is different to the host OS kernel:

chmod +x eksd-kata-containers/script/check-kata.sh
./eksd-kata-containers/script/check-kata.sh

Testing Kata Containers..

docker.io/library/busybox:latest:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:ae39a6f5c07297d7ab64dbd4f82c77c874cc6a94cea29fdec309d0992574b4f7:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:1ccc0a0ca577e5fb5a0bdf2150a1a9f842f47c8865e861fa0062c5d343eb8cac: exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f531cdc67389c92deac44e019e7a1b6fba90d1aaa58ae3e8192f0e0eed747152:    exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:388056c9a6838deea3792e8f00705b35b439cf57b3c9c2634fb4e95cfc896de6:   exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.0 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:ae39a6f5c07297d7ab64dbd4f82c77c874cc6a94cea29fdec309d0992574b4f7...
done

Test successful:
  Host kernel version      : 4.14.225-169.362.amzn2.x86_64
  Container kernel version : 5.4.71

The sample containerd configuration file will direct Kata to use the QEMU/KVM hypervisor, per the ConfigFile directive on line 19. Configuration files for Cloud Hypervisor/KVM, and Firecracker/KVM are also installed with Kata Containers:

• Firecracker: /opt/kata/share/defaults/kata-containers/configuration-fc.toml
• Cloud Hypervisor: /opt/kata/share/defaults/kata-containers/configuration-clh.toml

To select an alternate hypervisor, update the ConfigFile directive and restart containerd.

Prepare Kubernetes environment

Pull and retag the pause, coredns, and etcd containers (copy and paste as one line):

sudo ctr image pull public.ecr.aws/eks-distro/kubernetes/pause:v1.18.9-eks-1-18-1;\
sudo ctr image pull public.ecr.aws/eks-distro/coredns/coredns:v1.7.0-eks-1-18-1; \
sudo ctr image pull public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-18-1; \
sudo ctr image tag public.ecr.aws/eks-distro/kubernetes/pause:v1.18.9-eks-1-18-1 public.ecr.aws/eks-distro/kubernetes/pause:3.2; \
sudo ctr image tag public.ecr.aws/eks-distro/coredns/coredns:v1.7.0-eks-1-18-1 public.ecr.aws/eks-distro/kubernetes/coredns:1.6.7; \
sudo ctr image tag public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-18-1 public.ecr.aws/eks-distro/kubernetes/etcd:3.4.3-0

Add the RPM repository to Google cloud RPM packages for Kubernetes by creating the following /etc/yum.repos.d/kubernetes.repo file:

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl

Install the required Kubernetes packages:

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Load the br_netfilter kernel module, and create /etc/modules-load.d/k8s.conf:

echo br_netfilter | sudo tee /etc/modules-load.d/k8s.conf
sudo modprobe br_netfilter
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

Create the /var/lib/kubelet directory, then configure the /var/lib/kubelet/kubeadm-flags.env file:

sudo su
mkdir -p /var/lib/kubelet
cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd —network-plugin=cni —Pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.2"
exit

Get compatible binaries for kubeadm, kubelet, and kubectl:

cd /usr/bin
sudo rm kubelet kubeadm kubectl
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-18/releases/1/artifacts/kubernetes/v1.18.9/bin/linux/amd64/kubelet; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-18/releases/1/artifacts/kubernetes/v1.18.9/bin/linux/amd64/kubeadm; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-18/releases/1/artifacts/kubernetes/v1.18.9/bin/linux/amd64/kubectl
sudo chmod +x kubeadm kubectl kubelet

Enable the kubelet service:

sudo systemctl enable kubelet

Configure kube.yaml

A sample kube.yaml file has been provided in the config directory within the eksd-kata-containers repository.

Update the sample kube.yaml by providing the values for variables surrounded by {{ and }} within the localAPIEndpoint and nodeRegistration sections:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: {{ primary_ip }}
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: {{ primary_hostname }}
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

Start the Kubernetes control-plane

Run the kubeadm init command, identifying the config file as follows:

sudo kubeadm init --config eksd-kata-containers/config/kube.yaml
...
[init] Using Kubernetes version: v1.18.9-eks-1-18-1
[preflight] Running pre-flight checks
...
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

Your Kubernetes cluster should now be up and running. The kubeadm output shows the exact commands to use to add nodes to the cluster. If something goes wrong, correct the problem and run kubeadm reset to prepare you system to run kubeadm init again.

Configure the cluster to schedule Kata Containers

Configure the Kubernetes client locally:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Deploy a Pod network to the cluster. For this example, we deploy a Weaveworks network:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=v1.18.9-eks-1-18-1"
...
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net (http://clusterrole.rbac.authorization.k8s.io/weave-net) created
clusterrolebinding.rbac.authorization.k8s.io/weave-net (http://clusterrolebinding.rbac.authorization.k8s.io/weave-net) created
role.rbac.authorization.k8s.io/weave-net (http://role.rbac.authorization.k8s.io/weave-net) created
rolebinding.rbac.authorization.k8s.io/weave-net (http://rolebinding.rbac.authorization.k8s.io/weave-net) created
daemonset.apps/weave-net created
You can also consider Calico or Cilium networks. Calico is popular because it can be used to propagate routes with BGP, which is often used on-prem.

If you are testing with a single node, untaint your master node:

kubectl taint nodes --all node-role.kubernetes.io/master-

A sample runtimeclass.yaml file has been provided in the config directory within the eksd-kata-containers repository:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

Create the kata RuntimeClass:

kubectl apply -f eksd-kata-containers/config/runtimeclass.yaml

Schedule Kata Containers with Kubernetes

Sample Pod specs have been provided in the config directory within the eksd-kata-containers repository.

nginx-kata.yaml will schedule a pod within a VM using Kata Containers by specifying kata as the runtimeClassName:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-kata
spec:
  runtimeClassName: kata
  containers:
  - name: nginx
    image: nginx

nginx.yaml will schedule a pod using the default containerd runtime (runc):

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx

Schedule the Pods using kubectl:

kubectl apply -f eksd-kata-containers/config/nginx-kata.yaml
kubectl apply -f eksd-kata-containers/config/nginx.yaml

You will now have x2 nginx pods running in the cluster, each using different container runtimes. To validate that the nginx-kata Pod has been scheduled inside a VM, exec into each container and retrieve the kernel version:

kubectl exec -it nginx-kata -- bash -c "uname -r"
5.4.71
kubectl exec -it nginx -- bash -c "uname -r"
4.14.225-169.362.amzn2.x86_64

The nginx-kata Pod returns the kernel version reported by the kernel running inside the Kata VM, whereas the nginx Pod reports the kernel version of the host OS as it's running as a traditional runc container.

Conclusion

The industry shift to containers presents unique challenges in securing user workloads within multi-tenant untrusted environments.

Kata Containers utilizes open source hypervisors as an isolation boundary for each container (or collection of containers in a pod); this approach solves the shared kernel dilemma with existing bare metal container solutions.

Combining Kata with EKS-D provides secure VM workload isolation on the same software that has enabled tens of thousands of Kubernetes clusters on Amazon EKS. This includes the latest upstream updates, as well as extended security patching support.

Photo by Malena Gonzalez Serena on Unsplash

oke-autoscaler is an open source Kubernetes node autoscaler for Oracle Container Engine for Kubernetes (OKE). The oke-autoscaler function provides an automated mechanism to scale OKE clusters by automatically adding or removing nodes from a node pool.

When you enable the oke-autoscaler function, you don't need to manually add or remove nodes, or over-provision your node pools. Instead, you specify a minimum and maximum size for the node pool, and the rest is automatic.

The oke-autoscaler git repository has everything you need to implement node autoscaling in your OKE clusters, and a comprehensive step-by-step work instruction.

Introduction

Node autoscaling is one of the key features provided by the Kubernetes cluster. Node autoscaling provides the cluster with the ability to increase the number of nodes as the demand for service response increases, and decrease the number of nodes as demand decreases.

There are a number of different cluster autoscaler implementations currently available in the Kubernetes community. Of the available options (including the Kubernetes Cluster Autoscaler, Escalator, KEDA, and Cerebral), the Kubernetes Cluster Autoscaler has been adopted as the “de facto” node autoscaling solution. It’s maintained by SIG Autoscaling, and has documentation for most major cloud providers.

The Cluster Autoscaler follows a straightforward principle of reacting to “unschedulable pods” to trigger scale-up events. By leveraging a cluster-native metric, the Cluster Autoscaler offers a simplified implementation and a typically friction-less experience to getting up and running with node autoscaling.

The oke-autoscaler function implements the same approach to node autoscaling for OKE as the Cluster Autoscaler does for other Kubernetes implementations (i.e. using unschedulable pods as a signal). Read-on for a more detailed overview of how oke-autoscaler functions.

About oke-autoscaler

The oke-autoscaler function automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

• there are pods that have failed to run in the cluster due to insufficient resources;
• nodes in the node pool have been underutilized for an extended period of time

By default the oke-autoscaler function implements only the scale-up feature, scale-down is an optional feature.

Oracle Container Engine for Kubernetes (OKE) is a developer friendly, container-native, and enterprise-ready managed Kubernetes service for running highly available clusters with the control, security, and predictable performance of Oracle’s Cloud Infrastructure.

OKE nodes are hosts in a Kubernetes cluster (individual machines: virtual or bare metal, provisioned as Kubernetes worker nodes). OKE nodes are deployed into logical groupings known as node pools. OKE nodes run containerized applications in deployment units known as pods. Pods schedule containers on nodes, which utilize resources such as CPU and RAM.

Overview

oke-autoscaler is implemented as an Oracle Function (i.e. an OCI managed serverless function):

• the Oracle Function itself is written in Python: oke-autoscaler/func.py
• the function uses a custom container image based on oraclelinux:7-slim, and also includes  rh-python36, the OCI CLI, and kubectl: oke-autoscaler/Dockerfile

Evaluation Logic

The autoscaler function interacts with the Kubernetes API and a number of OCI control plane APIs to evaluate the state of the cluster, and the target node pool.

When the function is invoked, it follows an order of operation as follows:

1. evaluates the state of the node pool, to ensure that the node pool is in a stable condition
2. if the node pool is determined to be mutating (e.g. in the process of either adding or deleting a node, or stabilizing after adding a node), the autoscaler function will exit without performing any further operation
3. if the node pool is in a stable condition, scale-up is evaluated first
4. if scale-up is not triggered, the autoscaler function evaluates for scale-down (where scale-down has been enabled by the cluster administrator)

Scheduling

The oke-autoscaler function is designed to be invoked on a recurring schedule. The periodicity by which the function is scheduled to be invoked is configurable. As a starting point, consider scheduling the oke-autoscaler function to be invoked at an interval of every 3 minutes.

Once invoked, the function will run in accordance with the Evaluation Logic described herein.

A dimension to consider is the length of the window of time that the function will use to calculate the average resource utilization in the node pool.

The cluster administrator assigns a value to the function custom configuration parameter node_pool_eval_window. The oke-autoscaler function implements node_pool_eval_window as the number of minutes over which to:

• calculate the average CPU & RAM utilization for the node pool when evaluating for scale-down

node_pool_eval_window is by default is intended to represent the number of minutes between each invocation of the function, i.e. the periodicity of the function. In the case where the oke-autoscaler function is scheduled for invocation every 3 minutes, setting node_pool_eval_window to 3 minutes will configure the oke-autoscaler function to use the 3 minutes lapsed since the previous invocation as the window of time to evaluate and node pool utilization metrics.

node_pool_eval_window does not need to match the function invocation schedule, and can be set to calculate resource utilization over either a longer or shorter window of time.

Scaling

The cluster administrator defines what the node pool maximum and minimum number of nodes should be. The oke-autoscaler function operates within these boundaries.

Scale-Up

The oke-autoscaler function queries the cluster to enumerate the number of pods for the given node pool which are in a Pending state, and are flagged as being in an Unschedulable condition.

The number of pods flagged as being Unschedulable will be greater than zero when there are insufficient resources in the cluster on which to schedule pods. This condition will trigger the oke-autoscaler function to scale-up the node pool by adding an additional node.

The scale-up feature is enabled by default.

Stabilization

During the period immediately after a new node becomes active in a node pool, the cluster may still report some pods as being Unschedulable - despite the fact that the cluster has introduced enough resource to support the desired state.

In order to prevent a premature scale-up event, the oke-autoscaler function implements a node pool stabilization window to fence off the node pool from any modification during the period immediately after the addition of a new node. The node pool stabilization widow is implemented to afford the cluster the required time to schedule the backlog of unschedulable pods to the new node.

Where the oke-autoscaler function is invoked during the stabilization window, and it detects the presence of Unschedulable pods - a scale-up event will not be triggered.

Scale-Down

The cluster administrator defines as a percentage, the average node pool CPU and/or RAM utilization that below which the autoscaler function will scale-down the node pool by deleting a node.

The oke-autoscaler function calculates the node pool average CPU and RAM utilization as a mean score expressed as a percentage, which is derived by obtaining average utilization across all nodes in the node pool.

The calculated node pool averages for CPU and/or RAM utilization are evaluated against the percentage based thresholds provided by the cluster administrator.

If the node pool average CPU and/or RAM utilization is less than the thresholds provided by the cluster administrator, the node pool will scale-down.

Administrator defined scale-down thresholds can be set to evaluate for:

• node pool average CPU utilization
• node pool average RAM utilization
• either node pool average CPU or RAM utilization

When a specified scale-down condition is met, the oke-autoscaler function will cordon and drain the worker node, then calls the Container Engine API to delete the node from the node pool.

Enabling the scale-down feature is optional.

Multiple Node Pools

The oke-autoscaler function supports clusters which contain multiple node pools. For a given cluster hosting multiple node pools, oke-autoscaler functions can be enabled for one or more of the associated node pools.

Function Return Data

The function will return a JSON array containing summary data describing node pool status, any action and associated result.

Function completed successfully - no action performed as no resource pressure:

Result: { 
    "success": { 
        "action": "none", 
        "reason": "no-resource-pressure", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "ready", 
        "node-count": "2.0" 
    } 
} 

Function completed successfully - no action performed as node pool is stabilizing:

Result: { 
    "success": { 
        "action": "none", 
        "reason": "node-pool-status", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "stabilizing", 
        "unschedulable-pods-count": "4.0",
        "node-count": "2.0" 
    } 
} 

Function completed successfully - scale-up:

Result: { 
    "success": { 
        "action": "scale-up", 
        "reason": "unschedulable-pods", 
        "unschedulable-pods-count": "10.0", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "ready", 
        "node-count": "3.0" 
    } 
}

Function completed with warning - scale-up, max node count limit reached:

Result: { 
    "warning": { 
        "action": "none", 
        "reason": "node-max-limit-reached", 
        "unschedulable-pods-count": "10.0", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "ready", 
        "node-count": "3.0" 
    } 
} 

Function completed successfully - scale-down, low CPU utilization:

Result: { 
    "success": { 
        "action": "scale-down", 
        "reason": "cpu", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "ready", 
        "node-count": "3.0" 
    } 
}

Function failed - missing user input data:

Result: { 
    "error": { 
        "reason": "missing-input-data" 
    } 
}

Function Log Data

The oke-autoscaler function has been configured to provide some basic logging regarding it's operation.
The following excerpt illustrates the function log data relating to a single oke-autoscaler function invocation:

Node Lifecycle State:  
ACTIVE 
ACTIVE 
...
...
Node Data: { 
    "availability_domain": "xqTA:US-ASHBURN-AD-1", 
    "fault_domain": "FAULT-DOMAIN-2", 
    "id": "ocid1.instance.oc1.iad.anuwcljrp7nzmjiczjcacpmcg6lw7p2hlpk5oejlocl2qugqn3rxlqlymloq", 
    "lifecycle_details": "", 
    "lifecycle_state": "ACTIVE", 
    "name": "oke-c3wgoddmizd-nrwmmzzgy2t-sfcf3hk5x2a-1", 
    "node_error": null, 
    "node_pool_id": "ocid1.nodepool.oc1.iad.aaaaaaaaae3tsyjtmq3tan3emyydszrqmyzdkodgmuzgcytbgnrwmmzzgy2t", 
    "private_ip": "10.0.0.92", 
    "public_ip": "193.122.162.73", 
    "subnet_id": "ocid1.subnet.oc1.iad.aaaaaaaavnsn6hq7ogwpkragmzrl52dwp6vofkxgj6pvbllxscfcf3hk5x2a" 
} 
...
...
Nodes: { 
    "0": { 
        "name": "10.0.0.75", 
        "id": "ocid1.instance.oc1.iad.anuwcljrp7nzmjicknuodt727iawkx32unhc2kn53zrbrw7fubxexsamkf7q", 
        "created": "2020-05-20T11:50:04.988000+00:00", 
        "cpu_load": 2.3619126090991616, 
        "ram_load": 15.663938512292285 
    }, 
    "1": { 
        "name": "10.0.0.92", 
        "id": "ocid1.instance.oc1.iad.anuwcljrp7nzmjiczjcacpmcg6lw7p2hlpk5oejlocl2qugqn3rxlqlymloq", 
        "created": "2020-05-24T05:33:14.121000+00:00", 
        "cpu_load": 3.01701506531393, 
        "ram_load": 14.896256379084324 
    } 
} 
...
...
Result: { 
    "success": { 
        "action": "none", 
        "reason": "no-resource-pressure", 
        "node-pool-name": "prod-pool1", 
        "node-pool-status": "ready", 
        "node-count": "2" 
    } 
}

Limitations

The oke-autoscaler function has the following limitations:

• This function should not be configured for invocation on a recurring schedule with an interval less than 2.5 minutes
• The function scale-down feature is not designed for use in clusters scheduling stateful workloads that utilise Local PersistentVolumes
• The function initiates a call to the Container Engine API updateNodePool() to scale, which can take several minutes to complete
• The function cannot wait for the updateNodePool() operation to complete, the cluster administrator will need to monitor the success or failure of the operation outside of the function

If resources are deleted or moved when autoscaling your node pool, workloads might experience transient disruption. For example, if the workload consists of a controller with a single replica, that replica's pod might be rescheduled onto a different node if its current node is deleted.

Before enabling the autoscaler function scale-down feature, workloads should be configured to tolerate potential disruption, or to ensure that critical pods are not interrupted.

Conclusion

Node autoscaling is one of the key features provided by the Kubernetes cluster, and with the oke-autoscaler you can easily implement this functionality into an OKE cluster.

Head over to the oke-autoscaler code repository for necessary code and a detailed work instruction to start gaining the benefits of Kubernetes node autoscaling in your OKE clusters.

Cover Photo by Paweł Czerwiński on Unsplash.

The ESP8266 chip is a great candidate for IoT development. It provides built-in 2.4GHz WiFi capabilities, has a multitude of different general purpose I/O (GPIO) pins available, and can take advantage of the extensive libraries available in the larger Arduino community - in fact many different commercial IoT devices use the ESP8266 chip.

Introduction

The Oracle Code Card is a Wi-Fi-enabled IoT device that's built around the ESP8266 microcontroller, and includes an e-paper display.

Oracle actually give these these away at their in-person events - amazing!

The Code Cards serve as a super-cool id-badge at the actual in-person events, and they also provide a platform for a bunch of hands-on learning and development exercises post event.

This post will outline a solution that I've built specifically for the Code Card platform: it's a serverless function that transforms your Code Card into an awesome, personalized id-badge.

Overview

The serverless function is designed to assemble and display a bitmap image which includes a unique card-owner identicon that's generated on-the-fly via a 3rd party API. Generation of the identicon avatar is based on a hash of the card owner's name.

Apart from turning your Code Card into a personalized id-badge, the avatar function is a great reference for building an Oracle function, which when invoked coordinates a number of interactions with a range of OCI services, external services, and the Code Card IoT device itself.

The avatar function is implemented as an Oracle Function (i.e. an OCI managed serverless function):

• the function is invoked via an API published via the OCI API Gateway Service
• the Oracle Function itself is written in Python: codecard-avatar/func.py
• the function uses a custom container image based on oraclelinux:7-slim, and also includes rh-python36 and ImageMagick: codecard-avatar/Dockerfile

Operation

In reference to the workflow illustration, there are two main elements to the workflow:

1. During the "Configure" phase, the Code Card is configured using the Code Card Configurator mobile application (here the Code Card unique ID and owner's name are registered in a database table hosted on Oracle APEX)

2. During the "Run" phase, the avatar function is then invoked by the Code Card via the API Gateway, which initiates a series of interactions with:

• the Code Card designer APEX backend
• the identicon generation web service (http://identicon-1132.appspot.com)
• OCI object storage

Combining the gathered artefacts the function proceeds to assemble the id-badge custom bitmap using ImageMagick, and directs the Code Card to download and display the image via the object storage service.

About Oracle Functions

Oracle Functions is a fully managed, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform, built on enterprise-grade Oracle Cloud Infrastructure and powered by the Fn Project open source engine. With Oracle Functions, you can deploy your code, call it directly or trigger it in response to events, and get billed only for the resources consumed during the execution.

Oracle Functions are "container-native". This means that each function is a completely self-contained Docker image that is stored in your OCIR Docker Registry and pulled, deployed and invoked when you invoke your function.

Container Image

In order to make available each of the tools used during the functions operation (e.g. image manipulation, etc.) the avatar function is implemented using a BYO container image (see snippet following) - having this degree of latitude when working with Oracle functions comes in very handy.

FROM oraclelinux:7-slim
ENV OCI_CLI_SUPPRESS_FILE_PERMISSIONS_WARNING=True
ENV PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rh/rh-python36/root/usr/bin:/opt/rh/rh-python36/root/usr/bin/oci:${PATH}"
ENV LC_ALL=en_US.utf8
ENV LANG=en_US.utf8
ARG CLI_VERSION=2.10.3
RUN mkdir /oci-cli
RUN mkdir /function
ADD requirements.txt /function/
WORKDIR /function
RUN yum -y install oracle-release-el7 && \
    yum -y install oracle-softwarecollection-release-el7 && \
    yum-config-manager --enable software_collections && \
    yum-config-manager --enable ol7_latest ol7_optional_latest ol7_addons && \
    yum-config-manager --disable ol7_ociyum_config && \
    yum -y install scl-utils && \
    yum -y install rh-python36 && \
    yum -y install gcc && \
    yum -y install wget && \
    yum -y install unzip && \
    yum -y install jq && \
    yum -y install ImageMagick && \
    export PATH=$PATH:/opt/rh/rh-python36/root/usr/bin && \
    rm -rf /var/cache/yum && \
    pip3 install --no-cache --no-cache-dir -r requirements.txt && rm -fr ~/.cache/pip /tmp* requirements.txt func.yaml Dockerfile .venv
WORKDIR /oci-cli
RUN wget -qO- -O oci-cli.zip "https://github.com/oracle/oci-cli/releases/download/v${CLI_VERSION}/oci-cli-${CLI_VERSION}.zip" && \
    unzip -q oci-cli.zip -d .. && \
    rm oci-cli.zip && \
    pip3 install oci_cli-*-py2.py3-none-any.whl && \
    yes | oci setup autocomplete && \
    groupadd --gid 1000 fn && \
    adduser --uid 1000 --gid fn fn
ADD . /function/
ENTRYPOINT ["/opt/rh/rh-python36/root/usr/bin/fdk", "/function/func.py", "handler"]

By specifying the runtime as docker in the func.yaml configuration file, the Fn CLI will build the custom container image as a part of the function deployment process. The entrypoint directive instructs the Python FDK to run the custom Code Card identicon script upon function invocation.

schema_version: 20180708
name: codecard-avatar
version: 0.0.1
runtime: docker
entrypoint: /python/bin/fdk /function/func.py handler
memory: 256

Solution work instruction

For a full work instruction, and access to the code - follow this link to the solution git repository.

For access to a range of other tools and demos built around the Code Card platform - this link to the Code Card git repository.

Photo by Jida Li on Unsplash

okectl is an open source CLI utility designed for use with Oracle Container Engine for Kubernetes (OKE). okectl provides a command-line interface for management of OKE and associated resources, including Kubernetes cluster lifecycle.

Introduction

okectl is designed as a stand-alone tool to automate operations such as the Kubernetes cluster creation process, and is typically best used as part of an automation pipeline.

Aside from being a useful automation tool, it's also a useful example of a practical application development scenario that leverages an OCI Software Development Kit (SDK).

In this blog post I'll provide an overview of the OCI Go SDK, as well as an overview of okectl, and how it's built.

About okectl

okectl is built using the Go SDK for Oracle Cloud Infrastructure (OCI).

Supported Operations

• --createOkeCluster
  Creates Kubernetes cluster control plane, node pool, worker nodes, & configuration data (kubeconfig & JSON cluster desctiption).
• --deleteOkeCluster
  Deletes specified cluster.
• --getOkeNodePool
  Retreives cluster, node pool, and node details for a specified node pool.
• --createOkeKubeconfig
  Creates kubeconfig authentication artefact for kubectl.

Interesting Features

As a little context, I created okectl a while back - right at the time when the OKE service had just been released.

I needed a means to end-end provision a Kubernetes cluster to OCI entirely as code. Terraform provided the ability to create all of OKE's OCI related dependencies; including VCN, subnets, load balancers, etc, but at the time lacked support for the OKE service.

I created okectl to solve for this gap. okectl was created to work in tandem with Terraform; that is, to be executed by Terraform as a local-exec operation. With okectl, a single Terraform configuration could be composed to first create the OCI resources necessary to support an OKE cluster, then in-turn run okectl to build the cluster.

With this use-case in mind, I also built some interesting features into okectl to provide Terraform with the ability to perform automated, remote software installation to OKE clusters - again, as a part of a single Terraform configuration (in this case, I leverage Terraform remote-exec and Helm).

Note: The OCI Terraform provider now supports OKE, including operations such as cluster lifecycle management.

--waitNodesActive

By default, OKE will report the status of a worker node pool as ACTIVE when the node pool entity itself is created - however this precedes the instantiation of the worker nodes themselves. It will typically be some minutes after the node pool is ACTIVE that the worker nodes in the pool will have been instantiated, software installation completed, and the worker nodes themselves reach the status of being ACTIVE - thus ready to run containers in support of the cluster.

--getOkeNodePool retrieves cluster, node pool, and node details for a specified node pool, and also provides the ability to wait for nodes in a given node pool to become fully active, via the flag --waitNodesActive.

This is handy when used as part of automation pipeline, where any next action is dependent on having cluster worker nodes fully provisioned and active. e.g. use when Terraforming to pause all operations until worker nodes are active and ready for further configuration or software installation. (See flag --tfExrernalDS below for further detail on providing worker node IP address to Terraform).

--waitNodesActive has three modes of operation:

• --waitNodesActive=false
  okectl will not wait & will return when the nominated node pool is ACTIVE, regardless of the state of nodes within the pool.
• --waitNodesActive=any
  okectl will wait & return when the nominated node pool is ACTIVE, and any of the nodes in the nominated node pool return as ACTIVE.
• --waitNodesActive=all
  okectl will wait & return when the nominated node pool is ACTIVE, and all of the nodes in the nominated node pool return as ACTIVE.

okectl implements --waitNodesActive by enumerating the lifecycle-state of nodes in a given node pool via the NodeLifecycleStateEnum enumerator within the OCI Go SDK.

Node lifecycle state is exposed via the SDK (ContainerEngineClient) GetNodePool function, which in the following example is providing data to okectl's getNodeLifecycleState function:

// get worker node lifecycle status..
func getNodeLifeCycleState(
	ctx context.Context,
	client containerengine.ContainerEngineClient,
	nodePoolId string) containerengine.GetNodePoolResponse {

	req := containerengine.GetNodePoolRequest{}
	req.NodePoolId = common.String(nodePoolId)

	resp, err := client.GetNodePool(ctx, req)
	helpers.FatalIfError(err)

	// marshal & parse json..
	nodePoolResp := resp.NodePool
	nodesJson, _ := json.Marshal(nodePoolResp)
	jsonParsed, _ := gabs.ParseJSON(nodesJson)
	nodeLifeCycleState = (jsonParsed.Path("nodes.lifecycleState").String())

	return resp
}

--tfExternalDs

Where the --getOkeNodePool flag --tfExternalDs=true is used, okectl will run as a Terraform external data source.

The Terraform external data source allows an external program implementing a specific protocol to act as a data source, exposing arbitrary data to Terraform for use elsewhere in the Terraform configuration.

In this circumstance, okectl provides a JSON response containing the public IP address of a worker node in a format compatible with the Terraform external data source specification:

./okectl getOkeNodePool --tfExternalDs=true
{"workerNodeIp":"132.145.156.184"}

In combination with the --waitNodesActive flag, this provides the ability to have Terraform first wait for worker nodes in a new node pool to become active, then obtain the public IP address of a worker node from okectl. With the public IP address of the worker node, Terraform can proceed to call a remote-exec provisioner to then perform operations such as cluster configuration, and perform application workload deployment.

About the OCI Software Development Kits

Oracle Cloud Infrastructure provides a number of SDKs to facilitate development of custom solutions.

The OCI SDKs are designed to streamline the process of building and deploying applications that integrate with Oracle Cloud Infrastructure services.

Each SDK provides the tools you need to develop an application, including code samples and documentation to create, test, and troubleshoot solutions.

If you want to contribute to the development of the SDKs, they are all open source and available on GitHub.

At present Oracle offer SDKs for Java, Python, Ruby, & Go.

OCI REST APIs & SDKs

Generally speaking, the Oracle Cloud Infrastructure APIs are typical REST APIs that adopt the following characteristics:

• The Oracle Cloud Infrastructure APIs use standard HTTP requests and responses.
• All Oracle Cloud Infrastructure API requests must support HTTPS and SSL protocol TLS 1.2.
• All Oracle Cloud Infrastructure API requests must be signed for authentication purposes.

Further detail regarding the OCI REST APIs can be found here, including the API for the Container Engine for Kubernetes service - which can be used to build, deploy, and manage OKE clusters.

Whilst it's possible to to program directly against the OCI REST APIs, the OCI SDKs provide a lot of pre-built functionality, and abstract away much of the complexity required when interacting directly with APIs - for example, creating authorisation signatures, and parsing responses.

In addition to the SDKs, Oracle also provide the OCI CLI and the OCI Terraform provider as additional options for a more streamlined experience when developing with the OCI REST API.

OCI Go SDK

The OCI Go SDK contains the following components:

• Service packages: All packages except common and any other package found inside cmd. These packages represent the Oracle Cloud Infrastructure services supported by the Go SDK. Each package represents a service. These packages include methods to interact with the service, structs that model input and output parameters, and a client struct that acts as receiver for the above methods.
• Common package: Found in the common directory. The common package provides supporting functions and structs used by service packages. Includes HTTP request/response (de)serialization, request signing, JSON parsing, pointer to reference and other helper functions. Most of the functions in this package are meant to be used by the service packages.
• cmd: Internal tools used by the oci-go-sdk.

The Go SDK also provides a broad range of examples for programming with many of the available OCI services, including the core services (compute, network, etc.), identity and access management, database, email, DNS, and more.

Full documentation can be found on the GoDocs site here.

To start working with the Go SDK, you need to import the service packages that serve your requirements, create a client, and then proceed to use the client to make calls.

okectl utilises the common, containerengine, and helpers packages from the OCI Go SDK:

// import libraries..
import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"os"
	"path/filepath"
	"regexp"
	"strings"
	"time"

	"github.com/Jeffail/gabs"
	"gopkg.in/alecthomas/kingpin.v2"
	"github.com/oracle/oci-go-sdk/common"
	"github.com/oracle/oci-go-sdk/containerengine"
	"github.com/oracle/oci-go-sdk/example/helpers"
)

The containerengine package is provided to simplify much of the heavy lifting associated with the orchestration of the OKE service. okectl leverages the containerengine package to create/destroy clusters, cluster node pools, and kubeconfig artefacts.

Before using the SDK to interact with a service, we first call the common.DefaultConfigProvider() function to provide necessary configuration and authentication data. See the Authentication section for information on creating a configuration file.

config := common.DefaultConfigProvider()
client, err := identity.NewContainerEngineClientWithConfigurationProvider(config)
if err != nil { 
     panic(err)
}

After successfully creating a client, requests can now be made to the service. Generally, all functions associated with an operation accept context.Context and a struct that wraps all input parameters. The functions then return a response struct that contains the desired data, and an error struct that describes the error if an error occurs:

// create cluster..
func createCluster(
	ctx context.Context,
	client containerengine.ContainerEngineClient,
	clusterName, vcnId, compartmentId, kubeVersion, subnet1Id, subnet2Id string) containerengine.CreateClusterResponse {

	req := containerengine.CreateClusterRequest{}
	req.Name = common.String(clusterName)
	req.CompartmentId = common.String(compartmentId)
	req.VcnId = common.String(vcnId)
	req.KubernetesVersion = common.String(kubeVersion)
	req.Options = &containerengine.ClusterCreateOptions{
		ServiceLbSubnetIds: []string{subnet1Id, subnet2Id},
		AddOns: &containerengine.AddOnOptions{
			IsKubernetesDashboardEnabled: common.Bool(true),
			IsTillerEnabled:              common.Bool(true),
		},
	}

	fmt.Println("OKECTL :: Create Cluster :: Submitted ...")
	resp, err := client.CreateCluster(ctx, req)
	helpers.FatalIfError(err)

	return resp
}

OCI applies throttling to many API requests to prevent accidental or abusive use of resources. If you make too many requests too quickly, you might see some succeed and others fail. Oracle recommends that you implement an exponential back-off, starting from a few seconds to a maximum of 60 seconds.

The helpers package implements a number of features, including an exponential retry backoff mechanism that's used to rate limit API polling.

okectl uses this to gracefully wait for operations to complete, such as cluster or node-pool creation:

// wait for create cluster completion..
workReqRespCls := waitUntilWorkRequestComplete(c, createClusterResp.OpcWorkRequestId)
fmt.Println("OKECTL :: Create Cluster :: Complete ...")
clusterId := getResourceID(workReqRespCls.Resources, containerengine.WorkRequestResourceActionTypeCreated, "CLUSTER")

Installation

Installing the Go SDK is simple, use the go get command to download the package (and any dependencies), and automatically install:

go get -u github.com/oracle/oci-go-sdk

Authentication

Oracle Cloud Infrastructure SDKs require basic configuration information, like user credentials and tenancy OCID. You can provide this information by:

• Using a configuration file
• Declaring a configuration at runtime

See the following overview for information on how to create a configuration file.

To declare a configuration at runtime, implement the ConfigurationProvider interface shown below:

// ConfigurationProvider wraps information about the account ownertype ConfigurationProvider interface {
    KeyProvider
    TenancyOCID() (string, error)
    UserOCID() (string, error)
    KeyFingerprint() (string, error)
    Region() (string, error)
}

Debug

The SDK has a built-in logging mechanism used internally. The internal logging logic is used to record the raw HTTP requests, responses and potential errors when (un)marshalling request and responses.

Built-in logging in the SDK is controlled via the environment variable OCI_GO_SDK_DEBUG and its contents.

The below are possible values for the OCI_GO_SDK_DEBUG variable:

• info or i enables all info logging messages
• debug or d enables all debug and info logging messages
• verbose or v or 1 enables all verbose, debug and info logging messages
• null turns all logging messages off

For example:

OCI_GO_SDK_DEBUG=1

Building okectl from source

Dependencies

• Install the Go programming language
• Install the Go SDK for Oracle Cloud Infrastructure

After installing Go and the OCI Go SDK, clone the okectl repository:

git clone https://gitlab.com/byteQualia/okectl.git

Commands from this point forward will assume that you are in the ../okectl directory.

Build

Build an okectl Linux compatible binary as follows:

GOOS=linux
GOARCH=amd64
go build -v okectl.go

Further information and pre-built binaries can be found at the okectl repository.

Using okectl

okectl requires configuration data via command-line arguments & associated flags. Command-line flags provide data relating to both the OCI tenancy, and also OKE cluster configuration parameters.

okectl implements Kingpin to manage command line and flag parsing. I chose Kingpin for this project as it's a type-safe command-line parser that provides straight-forward support for flags, nested commands, and positional arguments.

The following are a subset of the usage examples that are available at the okectl repository. For further examples, head over there..

Example - Usage

./okectl
usage: OKECTL [<flags>] <command> [<args> ...]

A command-line application for configuring Oracle OKE (Container Engine for Kubernetes.)

Flags:
  --help                 Show context-sensitive help (also try --help-long and --help-man).
  --configDir=".okectl"  Path where output files are created - e.g. kubeconfig file.
  --version              Show application version.

Commands:
  help [<command>...]
    Show help.

  createOkeCluster --vcnId=VCNID --compartmentId=COMPARTMENTID --subnet1Id=SUBNET1ID --subnet2Id=SUBNET2ID --subnet3Id=SUBNET3ID [<flags>]
    Create new OKE Kubernetes cluster.

  deleteOkeCluster --clusterId=CLUSTERID
    Delete OKE Kubernetes cluster.

  getOkeNodePool [<flags>]
    Get cluster, node poool, and node details for a specified node pool.

  createOkeKubeconfig --clusterId=CLUSTERID
    Create kubeconfig authentication artefact for kubectl.

Example - Create Kubernetes Cluster

Interactive Help

./okectl createOkeCluster --help

usage: OKECTL createOkeCluster --vcnId=VCNID --compartmentId=COMPARTMENTID --subnet1Id=SUBNET1ID --subnet2Id=SUBNET2ID --subnet3Id=SUBNET3ID [<flags>]

Create new OKE Kubernetes cluster.

Flags:
  --help                              Show context-sensitive help (also try --help-long and --help-man).
  --configDir=".okectl"               Path where output files are created - e.g. kubeconfig file. Specify as absolute path.
  --version                           Show application version.
  --vcnId=VCNID                       OCI VCN-Id where cluster will be created.
  --compartmentId=COMPARTMENTID       OCI Compartment-Id where cluster will be created.
  --subnet1Id=SUBNET1ID               Cluster Control Plane LB Subnet 1.
  --subnet2Id=SUBNET2ID               Cluster Control Plane LB Subnet 2.
  --subnet3Id=SUBNET3ID               Worker Node Subnet 1.
  --subnet4Id=SUBNET4ID               Worker Node Subnet 2.
  --subnet5Id=SUBNET5ID               Worker Node Subnet 3.
  --clusterName="dev-oke-001"         Kubernetes cluster name.
  --kubeVersion="v1.10.3"             Kubernetes cluster version.
  --nodeImageName="Oracle-Linux-7.4"  OS image used for Worker Node(s).
  --nodeShape="VM.Standard1.1"        CPU/RAM allocated to Worker Node(s).
  --nodeSshKey=NODESSHKEY             SSH key to provision to Worker Node(s) for remote access.
  --quantityWkrSubnets=1              Number of subnets used to host Worker Node(s).
  --quantityPerSubnet=1               Number of Worker Nodes per subnet.
  --waitNodesActive="false"           If waitNodesActive=all, wait & return when all nodes in the pool are active.
                                      If waitNodesActive=any, wait & return when any of the nodes in the pool are active.
                                      If waitNodesActive=false, no wait & return when the node pool is active.

Create Cluster

./okectl createOkeCluster \
--clusterName=OKE-Cluster-001 \
--kubernetesVersion=v1.10.3 \
--vcnId=ocid1.vcn.oc1.iad.aaaaaaaamg7tqzjpxbbibev7lhp3bhgtcmgkbbrxr7td4if5qa64bbekdxqa \
--compartmentId=ocid1.compartment.oc1..aaaaaaaa2id6dilongtlxxmufoeunasaxuv76xxcb4ewxcxxxw5eba \
--quantityWkrSubnets=1 \
--quantityPerSubnet=1 \
--subnet1Id=ocid1.subnet.oc1.iad.aaaaaaaagq5apzuwr2qnianczzie4ffo6t46rcjehnsyoymiuunxaauq7y7a \
--subnet2Id=ocid1.subnet.oc1.iad.aaaaaaaadxr6zl4jpmcaxd4izzlvbyq2pqss3pmotx6dnusmh3ijorrpbhva \
--subnet3Id=ocid1.subnet.oc1.iad.aaaaaaaabf6k3ufcjdsdb5xfzzc3ayplhpip2jxtnaqvfcpakxt3bhmhecxa \
--nodeImageName=Oracle-Linux-7.4 \
--nodeShape=VM.Standard1.1 \
--nodeSshKey="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsHX7RR0z+JSAf+5nfTO9kS4Y6HV2pPXoXTqUJH..." \
--waitNodesActive="all"

For the above createOkeCluster request, okectl will provision:

• Kubernetes Cluster (Control Plane) - Version will be as nominated via the --kubeVersion flag.
• Node Pool - Node Pool will be created across the number of worker subnets as provided via --quantityWkrSubnets flag.
• Nodes - Worker nodes will be provisioned to each of the nominated worker subnets. Number of worker nodes per subnet is determined by the --quantityPerSubnet flag.
• Configuration Data - Provision to local filesystem a kubeconfig authentication artefact (kubeconfig) & json description of cluster configuration (nodeconfig.json).

Per the flag --waitNodesActive="all", okectl will return when cluster, node pool, and each of the nodes in the node pool are active.

Once completed, okectl will output the cluster, nodepool and node configuration data (stdout):

OKECTL :: Create Cluster :: Complete ...
--------------------------------------------------------------------------------------
{
       "id": "ocid1.nodepool.oc1.iad.aaaaaaaaae3tonjqgftdiyrxha2gczrtgu3winbtgbsdszjqmnrdeodegu2t",
       "compartmentId": "ocid1.compartment.oc1..aaaaaaaa2id6dilongtl6fmufoeunasaxuv76b6cb4ewxcw4juafe55w5eba",
       "clusterId": "ocid1.cluster.oc1.iad.aaaaaaaaae2tgnlbmzrtknjygrrwmobsmvrwgnrsmnqtmzjygc2domtbgmyt",
       "name": "oke-dev-001",
       "kubernetesVersion": "v1.10.3",
       "nodeImageId": "ocid1.image.oc1.iad.aaaaaaaajlw3xfie2t5t52uegyhiq2npx7bqyu4uvi2zyu3w3mqayc2bxmaa",
       "nodeImageName": "Oracle-Linux-7.4",
       "nodeShape": "VM.Standard1.1",
       "initialNodeLabels": [],
       "sshPublicKey": "",
       "quantityPerSubnet": 1,
       "subnetIds": [
               "ocid1.subnet.oc1.iad.aaaaaaaajvfrxxawuwhvxnjliox7gzibonafqcyjkdozwie7q5po7qbawl4a"
       ],
       "nodes": [
           {
              "id": "ocid1.instance.oc1.iad.abuwcljtayee6h7ttavqngewglsbe3b6my3n2eoqawhttgtswsu66lrjgi4q",
              "name": "oke-c2domtbgmyt-nrdeodegu2t-soxdncj6x5a-0",
              "availabilityDomain": "Ppri:US-ASHBURN-AD-3",
              "subnetId": "ocid1.subnet.oc1.iad.aaaaaaaattodyph6wco6cmusyza4kyz3naftwf6yjzvog5h2g6oxdncj6x5a",
              "nodePoolId": "ocid1.nodepool.oc1.iad.aaaaaaaaae3tonjqgftdiyrxha2gczrtgu3winbtgbsdszjqmnrdeodegu2t",
              "publicIp": "100.211.162.17",
              "nodeError": null,
              "lifecycleState": "UPDATING",
              "lifecycleDetails": "waiting for running compute instance"
           }
      ]
 }

By default, okectl will create a sub-directory named ".okectl" within the same directory as the okectl binary. okectl will create x2 files within the ".okectl" directory:

• kubeconfig - This file contains authentication and cluster connection information. It should be used with the kubectl command-line utility to access and configure the cluster.
• nodepool.json - This file contains a detailed output of the cluster and node pool configuration in json format.

Output directory is configurable via the --configDir flag. Path provided to --configDir should be provided as an absolute path.

All clusters created using okectl will be provisioned with the additional options of the Kubernetes dashboard & Helm/Tiller as installed.

Conclusion

Head over to the okectl repository for further information on accessing a cluster, and performing cluster operations using kubectl via CLI or accessing the Kubernetes dashboard.

Cover Photo by Paweł Czerwiński on Unsplash.

Welcome to this introduction to Grafeas!

First I'll introduce you to Grafeas, an open artifact metadata API designed to help audit and govern your software supply chain. Then I'll take you through a Grafeas deplyment scenario on Oracle Container Engine for Kubernetes (OKE), whereby a Kubernetes validating admission controller is configured to integrate with Grafeas - providing a mechanism to make real-time, policy-based decisioning about whether pods are authorised to run on the cluster.

This content is syndicated from this repository - references to code snippets or files herein can be found over there.

About Grafeas

At each stage of the software supply chain (code, build, test, deploy, and operate), different tools generate metadata about various software components. Examples include the identity of the developer, when the code was checked in and built, what vulnerabilities were detected, what tests were passed or failed, and so on. Grafeas’ goal is to provide the infrastructure to store and manage this metadata about artifacts associated with the software supply chain.

Grafeas (Greek word for “scribe”) provides organizations with a central source of truth for the tracking and enforcing of policies across software development teams and pipelines. The intention is that build, auditing and compliance tools can use the Grafeas API to store, query and retrieve metadata around a wide array of artifacts and components associated with the software development and deployment life cycle.

Tracking Grafeas’ metadata can give you confidence about what containers are in your environment, and also provides the ability to enforce restrictions on which containers get deployed. Deployment tooling can be configured to review Grafeas metadata for compliance with your policies before deploying. Grafeas can be used to enforce a wide variety of security policies. For example, you can configure policies to block container images with vulnerabilities from being deployed, and ensure that deployed images are built from a base image explicitly sanctioned by your security team, or to require that images go through your build pipeline.

Grafeas divides the metadata information into notes and occurrences. Notes are high-level descriptions of particular types of metadata. Occurrences are instantiations of notes, which describe how and when a given note occurs on the resource associated with the occurrence. This division allows for fine-grained access control of different types of metadata.

Tutorial Overview

Let’s consider an example of how Grafeas can provide deploy-time control for a sample MySQL implementation using a demonstration verification pipeline.

This example will use the docker.io/mysql/mysql-server:8.0.12 container for testing. We assume that you (as the QA engineer) want to create an attestation within Grafeas that will certify this image as safe for production usage. Only this image will run on our cluster, requests to create pods based on any other image that has not been approved will be rejected by the Kubernetes control plane.

The Grafeas deployment scenario illustration describes the verification pipeline that will be implemented into our cluster:

• A validating admission controller is configured via specific rules to intercept all pod creation requests to the Kubernetes API server, which occurs prior to persistence of the requested object.
• Once initiated, the admission controller is configured to call a validating admission webhook, which is responsible for checking with the Grafeas API whether our image is authorised to run (admission webhooks are HTTP callbacks that receive admission requests and do something with them).

Grafeas uses kind-specific schemas, such that each kind of metadata information adheres to a strict schema. In our Grafeas deployment we will be referncing metadata using the Grafeas kind ATTESTATION, which will certify that the MySQL image complies with our deployment policy requirements. By using the validating admission webhook, we are able to check at runtime for the expected Grafeas attestations, and block deployment when they aren’t present.

Note: It is suggested that this tutorial only be implemented into a non mission-critical environment. The deployment scenario will prevent all pods from running, apart from only those explicitly configured with the appropriate attestations within the Grafeas deployment.

Prerequisites

You will need to have deployed your OKE Kubernetes cluster before commencing to implement the deployment scenario. Follow the link to this tutorial for guidance on the process.

• Create a kubeconfig authentication artefact. This will be used later in the tutorial to connect to the Grafeas server. Follow the link to this tutorial for guidance on the process.
• Kubernetes admission controllers need to be enabled. Since we are running Kubernetes on OKE, our required admission controllers are automatically enabled. OKE implements the recommended set of admission controllers for a given Kubernetes cluster version.

Clone the oke-grafeas-tutorial repository

Clone the oke-grafeas-tutorial repository:

git clone https://gitlab.com/byteQualia/oke-grafeas-tutorial.git

Commands from this point forward will assume that you are in the oke-grafeas-tutorial directory.

Deploy Grafeas server

Run the following command to deploy your Grafeas server:

kubectl apply -f kubernetes/grafeas.yaml

In our demonstration we are using a pre-built Grafeas image based on the example Grafeas server. The demonstration Grafeas deployment is configured as a light-weight implementation, with configuration data as ephemeral only. As such, configuration data will not persist across restarts.

Generate GPG (GnuPG) keys

Next we generate a GPG keypair that will be used to sign our container image metadata. We'll then retrieve the id of the image signing key.

Assuming gpg is installed on your system, run the following command to generate a signing key:

gpg --batch --gen-key pki/gpg.keygen

Now issue the following command to retreive the ID of the gpg signing key created in the previous step:

gpg --list-keys --keyid-format short

The output from the gpg --list-keys command contains the key ID:

----------------------------
pub   2048R/89BEA918 2018-09-15
uid   signatory (example key signatory) <signatory@example.com>
sub   2048R/B5B42C98 2018-09-15

The key ID in the above output example is 89BEA918. Take note of the unique key output generated in your shell session, and store it in the GPG_KEY_ID environment variable as follows:

export GPG_KEY_ID=89BEA918

Docker uses a content-addressable image store. The image ID is a SHA256 digest covering each of the images layers. We will utilise the image ID as our unique identifier for the specific image that will be explicitly permitted to run on our cluster.

We will be using the mysql/mysql-server:8.0.12 container image as our example white-listed resource. Run the following commands to sign a text file containing the image digest for mysql/mysql-server:8.0.12 using gpg:

### ### ###
# Create our mysql-image-digest.txt file..
### ### ###
cat >mysql-image-digest.txt <<EOF
sha256:58c5d4635ab6c6ec23b542a274b9881dca62de19c793a8b8227a830a83bdbbdd
EOF
### ### ###
# Sign the file..
### ### ###
gpg -u signatory@example.com \
  --armor \
  --clearsign \
  --output=mysql-signature.gpg \
  mysql-image-digest.txt
### ### ###
# Verify the signature..
### ### ###
gpg --output - --verify mysql-signature.gpg

The output signature verification should be similar to the following:

gpg: Signature made Sat 15 Sep 2018 08:48:40 AM UTC using RSA key ID 89BEA918
gpg: Good signature from "signatory (example key signatory) <signatory@example.com>"

In order for others to verify signed images - they must both trust, and have access to the image signer's public key. Run the following command to export the image signer's public key:

gpg --armor --export signatory@example.com > ${GPG_KEY_ID}.pub

Create Grafeas objects

Now that we have a signed container image and a public key for verification, we can go ahead and create a Grafeas attestationAuthority note, and an associated pgpSignedAttestation occurrence using the Grafeas API. The pgpSignedAttestation occurrence is used to make statements about suitability for a deployment.

First, create a secure tunnel to the Grafeas API endpoint inside your cluster:

kubectl port-forward \
  (kubectl get pods -l app=grafeas -o jsonpath='{.items[0].metadata.name}') \
  8080:8080

Create Grafeas production project

Let's create a Grafeas project which will then contain our note & associated occurrence:

curl -v -X POST http://localhost:8080/v1alpha1/projects \
  -H 'Content-Type: application/json' \
  -d '{"name": "projects/image-signing"}'

Create Grafeas attestationAuthority note

Then run the following commands to create the production attestationAuthority note:

### ### ###
# Define the attestationAuthority Note content..
### ### ###
cat >prod-note.json <<EOF
{
  "name": "projects/image-signing/notes/production",
  "shortDescription": "Production image signer",
  "longDescription": "Production image signer",
  "kind": "ATTESTATION_AUTHORITY",
  "attestationAuthority": {
    "hint": {
      "humanReadableName": "production"
    }
  }
}
EOF
### ### ###
# Post the production attestationAuthority content..
### ### ###
curl -X POST \
  "http://localhost:8080/v1alpha1/projects/image-signing/notes?noteId=production" \
  -d @prod-note.json

Create Grafeas pgpSignedAttestation occurrence

Now that the project and note have been created, we can create a pgpSignedAttestation occurrence referencing our production note:

### ### ###
# Define GPG_SIGNATURE & RESOURCE_URL environment variables..
### ### ###
export GPG_SIGNATURE=$(cat mysql-signature.gpg | base64)
export RESOURCE_URL="https://docker.io/mysql/mysql-server@sha256:58c5d4635ab6c6ec23b542a274b9881dca62de19c793a8b8227a830a83bdbbdd"
### ### ###
# Define the pgpSignedAttestation Occurrance content..
### ### ###
cat > prod-occurrence.json <<EOF
{
  "resourceUrl": "${RESOURCE_URL}",
  "noteName": "projects/image-signing/notes/production",
  "attestationDetails": {
    "pgpSignedAttestation": {
       "signature": "${GPG_SIGNATURE}",
       "pgpKeyId": "${GPG_KEY_ID}"
    }
  }
}
EOF
### ### ###
# Post the pgpSignedAttestation content..
### ### ###
curl -X POST \
  'http://127.0.0.1:8080/v1alpha1/projects/image-signing/occurrences' \
  -d @prod-occurrence.json

At this point the Grafeas configuration is complete, and the hub.docker.com/r/mysql/mysql-server v8.0.12 image can be verified through the Grafeas API.

With the configuration that is currently in place - only the hub.docker.com/r/mysql/mysql-server v8.0.12 image identified by the sha256:58c5d4635ab6c6ec23b542a274b9881dca62de19c793a8b8227a830a83bdbbdd image digest can be verified by the Grafeas API. Authorisation of additional images would require a new Grafeas occurrence.

Kubernetes configuration

Next we will go ahead and create the Kubernetes Validating Admission Controller, and deploy our Validating Admission Webhook.

Deploy the Image Signature Webhook to Kubernetes

Run the following command to create the image-signature-webhook configmap, and store the image signer's public key:

kubectl create configmap image-signature-webhook \
  --from-file ${GPG_KEY_ID}.pub

Next we create the tls-image-signature-webhook secret, and store the TLS certificates:

kubectl create secret tls tls-image-signature-webhook \
  --cert=pki/image-signature-webhook.pem \
  --key=pki/image-signature-webhook-key.pem

Now create the image-signature-webhook deployment, and image-signature-webook ValidatingWebhookConfiguration:

kubectl create secret tls tls-image-signature-webhook \
  --cert=pki/image-signature-webhook.pem \
  --key=pki/image-signature-webhook-key.pem

At this point the Grafeas and Kubernetes configurations are complete. Requests to create pods in the cluster will now be intercepted by the validating admission controller.

Testing the Admission Controller and Webhook

First we attempt to run the oracle/nosql:4.3.11 container image, which doesn't have a pgpSignedAttestation occurrence in the Grafeas metadata repository. Attempt to create the pod:

kubectl apply -f kubernetes/pod/nosql-server-4.3.11.yaml

Notice the nosql-server pod was not created, and the following error was returned:

The  "" is invalid: : No matched signatures for container image: docker.io/oracle/nosql:4.3.11

The nosql-server pod wasn't created because the oracle/nosql:4.3.11 container image was not verified by the image signature webhook. No related pgpSignedAttestation occurrence exists in the Grafeas metadata repository.

Now attempt to run the docker.io/mysql/mysql-server@sha256:58c5d4635ab6c6ec23b542a274b9881dca62de19c793a8b8227a830a83bdbbdd container image - which does have a pgpSignedAttestation occurrence in the Grafeas metadata repository:

kubectl apply -f kubernetes/pod/mysql-server-8.0.12.yaml

Now we receive a more familiar, successful response:

pod "mysql-server" created

At this point you should now have the following pods running in your cluster:

kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
grafeas-7554b6bffd-6gsfq                   1/1       Running   0          58m
image-signature-webhook-6fd49f765f-8h9z9   1/1       Running   0          30m
mysql-server                               1/1       Running   0          30s

Logging

To attach to your image-signature-webhook pod and get access to more detailed logging, refer to the logging section in the following instruction.

Tidy Up

Run the following commands to remove the Kubernetes resources created during this tutorial:

kubectl delete deployments grafeas image-signature-webhook
kubectl delete pods mysql-server
kubectl delete svc grafeas image-signature-webhook
kubectl delete secrets tls-image-signature-webhook
kubectl delete configmap image-signature-webhook

Conclusion

Tracking Grafeas' metadata allows a team to understand what containers are stored within the environment, and also to enforce restrictions on which containers get deployed. This central store of metadata promises to provide a whole new layer of visibility into the software supply chain — and to enhance auditing and governance capabilities.

Cover Photo by Mitchell Luo on Unsplash

When using cloud computing services, it seems simple to say that managing costs is critical to success. As it turns out - the task of managing cloud computing costs can be easier said than done. Incurring unplanned costs can often be the result of a combination of factors, such as a lack of visibility about the current consumption patterns and past trends, non-standard deployments which can originate from an absence development and authorisation processes, a lack of organisation, or the absence of automated deployment and configuration tools.

A combination of tooling and controls is critical in empowering organisations to effectively plan, forecast and manage cloud computing costs. Factors such as consolidated visibility of cloud inventory, policy based governance, role based access control, controlled stack templates, automated alerts and notifications, and cost analytics are essential.

Oracle Cloud Infrastructure provides a compliment of native controls, billing, and payment tools that make it easy to manage your service costs. For example, OCI Budgets can be used to set and manage limits on your Oracle Cloud Infrastructure spending. You can set alerts on your budget to let you know when you might exceed your budget, and you can view all of your budgets and spending from one single place in the OCI console. Among other controls, such as Compartments and cost-tracking tags - OCI also provides a Cost Analysis dashboard, and access to Usage Reports.

An OCI Usage Report is a comma-separated value (CSV) file that can be used to get a detailed breakdown of your Oracle Cloud Infrastructure resources for audit or invoice reconciliation.

In this tutorial we'll take you through a deployment scenario on OCI, whereby an Oracle Function (i.e. a managed serverless function) will be deployed to automatically to retrieve OCI Usage Reports, and upload the data to an Autonomous Data Warehouse (ADW) instance.  

Storing the historical Usage Data in ADW provides a well managed repository for the historical account utilisation data, as well as making it available for introspection and analysis by popular analytics tooling such as Oracle Data Visualisation Desktop (DVD), or Oracle Analytics Cloud (OAC). These powerful data analysis tools will enable you to perform deep and comprehensive introspection of cloud resource utilisation, and also provide the ability to perform predictive cost analyses based on historical usage trends.

This content is syndicated from this repository - references to code snippets or files herein can be found over there.

About OCI Usage Reports

The OCI Usage Report is automatically generated daily, and is stored in an Oracle-owned object storage bucket. It contains one row per each Oracle Cloud Infrastructure resource (such as instance, object storage bucket, VNIC) per hour along with consumption information, metadata, and tags. Usage reports are retained for one year.

A comprehensive overview of the Usage Report schema is available here.

About Oracle Functions

Oracle Functions is a fully managed, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform, built on enterprise-grade Oracle Cloud Infrastructure and powered by the Fn Project open source engine. With Oracle Functions, you can deploy your code, call it directly or trigger it in response to events, and get billed only for the resources consumed during the execution.

Oracle Functions are "container-native". This means that each function is a completely self-contained Docker image that is stored in your OCIR Docker Registry and pulled, deployed and invoked when you invoke your function.

Tutorial Overview

Let’s consider an example of how we can;

• Provision an ADW instance & a database table to host our OCI tenancy Usage Report data.
• Create a custom Oracle Function to programatically retreive daily CSV data from the Object Storgae Service, and insert the data into ADW.

The Oracle Function itself is written in Python (see ../oci-usage-to-adw-function/adw-billing/func.py).
The function uses a custom container image based on oraclelinux:7-slim, and also includes oracle-instantclient19.3-basiclite, and rh-python36 (see ../oci-usage-to-adw-function/adw-billing/Dockerfile).

When invoked, the function uses a call to a 'resource principal provider' that enables the function to authenticate and access the Usage Reports Object Storage (OSS) bucket, and to also download the credentials wallet used to access the ADW instance.

The function enumerates the usage reports contained within the OSS bucket, and will insert into the oci_billing table all Usage Reports data that has not previously been insterted. This means that the first time the function is invoked, an initial bulk upload of all historical Usage Report data will occur. For subsequent function invocations, only new Usage Data will be processed.

Resources referenced in this tutorial will be named as follows:

• Compartment containing the ADW instance: Demo-Compartment
• OCI IAM Dynamic Group Name: FnFunc-Demo
• Oracle Functions Application Name: billing
• Function Name: adw-billing

Prerequisites

The following should be completed before going ahead and creating your Oracle Cloud Function:

• OCI Tenancy: If you don't already have an OCI tenancy, you can sign-up right here and experience the benefits of OCI with the included always free services, including Autonomous Data Warehouse!
• Deploy an Autonomous Data Warehouse: You will need to have deployed your Autonomous Data Warehouse instance prior to commencing implementation of the deployment scenario. Follow the link to this tutorial for guidance on the process.
• Download the DB Client Credential Package (Credentials Wallet): Information contained within the wallet will be used later in the tutorial. Follow the link to this tutorial for guidance on the process.
• Access the Usage Reports Bucket: OCI IAM policies are required to be configured in order to access Usage Reports. Follow this tutorial for guidance on the process.
• Set up your tenancy for function development: Follow the link to this tutorial for guidance on the process.
• Configure Your Client Environment for Function Development: Before you can start using Oracle Functions to create and deploy functions, you have to set up your client environment for function development. Follow the link to this tutorial for guidance on the process.

Additional IAM Policies

When a function you've deployed to Oracle Functions is running, it can access other Oracle Cloud Infrastructure resources. To enable a function to access another Oracle Cloud Infrastructure resource, you have to include the function in a dynamic group, and then create a policy to grant the dynamic group access to that resource. Follow the link to this tutorial for guidance on creating a dynamic group.

For our deployment scenario we'll require our "FnFunc-Demo" dynamic group to access both the Usage Report object storage bucket, as well as our Autonomous Database instance. To enable this, create the following dynamic group and additional IAM policies:

Dynamic group

For the below dynamic group definition, the resource.compartment.id is that of the "Demo-Compartment" where the application and associated function will be deployed:

ALL {resource.type = 'fnfunc', resource.compartment.id = 'ocid1.compartment.oc1..aaaaaaaafnaar7sww76or6eyb3j625uji3tp4cb4tosaxx4wbxvvag4dkrra'}

IAM policies

Create the additional IAM policies:

endorse dynamic-group FnFunc-Demo to read objects in tenancy usage-report
allow dynamic-group FnFunc-Demo to use autonomous-databases in compartment Demo-Compartment where request.permission='AUTONOMOUS_DATABASE_CONTENT_READ'

Create oci_billing Database Table

Next we create our database table which will store the Usage Report data. To do this we will use the ADW instance built-in SQL Developer Web client. Follow the link to this tutorial for guidance on accessing SQL Developer Web as user ADMIN from your Autonomous Data Warehouse. Once connected, run the following SQL statement to create the oci_billing table:

create table oci_billing(usage_report varchar2(150 CHAR), lineItem_referenceNo varchar2(150 CHAR), lineItem_tenantId varchar2(150 CHAR),  
lineItem_intervalUsageStart varchar2(150 CHAR), lineItem_intervalUsageEnd varchar2(150 CHAR), product_service varchar2(150 CHAR),  
product_resource varchar2(150 CHAR), product_compartmentId varchar2(150 CHAR), product_compartmentName varchar2(150 CHAR),  
product_region varchar2(150 CHAR), product_availabilityDomain varchar2(150 CHAR), product_resourceId varchar2(150 CHAR),  
usage_consumedQuantity varchar2(150 CHAR), usage_billedQuantity varchar2(150 CHAR), usage_consumedQuantityUnits varchar2(150 CHAR),  
usage_consumedQuantityMeasure varchar2(150 CHAR), lineItem_isCorrection varchar2(150 CHAR), lineItem_backreferenceNo varchar2(150 CHAR));

Create Oracle Functions Application: billing

In Oracle Functions, an application is a logical grouping of functions & a common context to store configuration variables that are available to all functions in the application.
Next, create an application named billing to host the adw-billing function. Follow the link to this tutorial for guidance on the process.

When creating applications, Oracle recommends that you use the same region as the Docker registry that's specified in the Fn Project CLI context, and be sure to select the compartment specified in the Fn Project CLI context.

Create Function: adw-billing

Now to create the actual function!

Clone the oci-adw-billing-tutorial git repository

First, let's clone the oci-adw-billing-tutorial repository:

git clone https://gitlab.com/byteQualia/oci-usage-to-adw-function.git

Commands from this point forward will assume that you are in the ../oci-usage-to-adw-function/adw-billing directory, which is the directory containing the function code, and other dependencies such as the Dockerfile used to build the container image, the func.yaml (function configuration file), and a Python requirements definition file.

Create the function

Enter the following single Fn Project command to build the function and its dependencies as a Docker image, push the image to the specified Docker registry, and deploy the function to Oracle Functions:

fn -v deploy --app billing

The Fn Project CLI will generate output similar to the following (abbreviated) detailing the steps taken to build and deploy the function.

Deploying adw-billing to app: billing
Bumped to version 0.0.1
Building image...
...
...
0.0.1: digest: sha256:71c0f9fac6164b676b781970b5d79b86a28838081c6ea88e00cc1cf07630ccc6 size: 1363
Updating function adw-billing using image iad.ocir.io/tenancy/fnbilling/adw-billing:0.0.1...

Implement function configuration parameters

Now that we have our function built and deployed - it requires the creation of a number of configuration parameters in order for it to operate successfully. User defined configuration parameters are made available to the function via key-value pairs known as custom configuration parameters.

To specify custom configuration parameters using the Fn Project CLI, the following command format is used:

fn config function <app-name> <function-name> <key> <value>

Create the following custom configuration parameters using the cofig function command:

-- Tenancy Usage Report Bucket

fn config function billing adw-billing usage_report_bucket <value>

The <value> field should contain the OCI tenancy OCID.

-- Credentials Wallet Configuration

fn config function billing adw-billing TNS_ADMIN /tmp/wallet

After invocation, the function will connect to the ADW instance and download and extract a copy of the credentials wallet (containing tsnames.ora) to the path /tmp/wallet. The TNS_ADMIN environment variable is used to specify the directory location for the tnsnames.ora file, which is used by the Oracle Instant Client when connecting to the ADW instance.

-- Database connection DSN

fn config function billing adw-billing db_dsn <value>

The <value> field should contain the preferred DSN connection string for the database.

An ODBC DSN specifies the database server name, and other database-related information required to access the Autonomous Data Warehouse instance.
Available DSNs are contained within the tsnames.ora file, which is located in the credentials wallet previously downloaded in the Prerequisites section herein.

The tsnames.ora file will contain multiple DSNs of the format name = (configuration data), e.g.

db201908233333_high = (description= (address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=aaaaaaaaaayxrey_db201908233333_high.adwc.oraclecloud.com))(security=(ssl_server_cert_dn=
        "CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US"))   )

In the case of the above example, the required <value> will be the name db201908233333_high.

-- Autonomous Database OCID

fn config function billing adw-billing db_ocid <value>

The <value> field should contain the Autonomous Database instance OCID. The ADW instance OCID can be found in the OCI console on Autonomous Database Details page for your ADW instance.

-- Autonomous Database Username

fn config function billing adw-billing db_user ADMIN

-- Autonomous Database Password

fn config function billing adw-billing db_pass <value>

The <value> field should contain the ADW ADMIN user password specified during the instance creation.

Configure function logging

When a function you've deployed to Oracle Functions is invoked, you'll typically want to store the function's logs so that you can review them later. You specify where Oracle Functions stores a function's logs by setting a logging policy for the application containing the function. Follow the link to this tutorial for guidance on the process.

Invoke the function

To invoke the function, issue the following command:

fn invoke billing adw-billing

That's it! Once completed, your function has now inserted all historical Usage Report data into your ADW instance.
Note: The current maximum run time for an Oracle Function is 120 seconds. If your tenancy has hundreds of historical Usage Reports to process (there can be up to 365), then it may take a couple of invocations to completely process the data backlog..

Function return data

The function will return a JSON array containing the name of each of the Usage Report files that were processed during the given invocation, e.g.

["0001000000010470.csv", "0001000000010480.csv"]

Inspect function logs

The function has been configured to provide some basic logging regarding it's operation.
The following excerpt illustrates the function log data relating to the download and processing of a single Usage Report file:

oci.base_client.139777842449152 - INFO -  2019-10-17 06:17:41.425456: Request: GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/bling/b/ocid1.tenancy.oc1..aaaaaaaac3l6hgyl../o/reports/usage-csv/0001000000076336.csv.gz 
oci._vendor.urllib3.connectionpool - DEBUG - https://objectstorage.us-ashburn-1.oraclecloud.com:443 "GET /n/bling/b/ocid1.tenancy.oc1..aaaaaaaac3l6hgyl../o/reports/usage-csv/0001000000076336.csv.gz HTTP/1.1" 200 256997 
...
...
root - INFO - finished downloading 0001000000076336.csv.gz
root - INFO - finished uploading 0001000000076336.csv.gz
root - INFO - report_id: ocid1.tenancy.oc1..aaaaaaaac3l6hgyl..-0001000000076776: 0
root - INFO - runtime: 06.092967748641968 seconds

Inspect uploaded Usage Data via SQL Developer Web client

Finally - let's use the ADW instance built-in SQL Developer Web client to take a look at the usage data as stored in our data warehouse. Once connected to the SQL Developer Web client, run the following SQL statement:

SELECT * FROM oci_billing;

On observation of the result set - you will note that each of the columns in the database correlate to fields as contained within Usage Report CSV files.

The only exception is the column usage_report, which has been included to help ensure records remain unique - particularly if you are hosting usage data from multiple tenancies within a single database. It's also used by the function to determine if a given Usage Report file has been previosly inserted into the database.

The usage_report field stores a value that is a concatenation of the OCI tenancy OCID and the Usage Report CSV file name from which the data was sourced, for example:

USAGE_REPORT
------------------------------------------------------------------------------------------------
ocid1.tenancy.oc1..aaaaaaaac3l6hgylozzuh2bxhf3557quavpa2v6675u2kejplzalhgk4nzka-0001000000010470
ocid1.tenancy.oc1..aaaaaaaac3l6hgylozzuh2bxhf3557quavpa2v6675u2kejplzalhgk4nzka-0001000000010470
ocid1.tenancy.oc1..aaaaaaaac3l6hgylozzuh2bxhf3557quavpa2v6675u2kejplzalhgk4nzka-0001000000010470

Cover Photo by Paweł Czerwiński on Unsplash.

Welcome to this guide to integrating Vault with Oracle Container Engine for Kubernetes (OKE).

Introduction

First I'll introduce you to Hashicorp Vault, a comprehensive tool designed for the secure management of secrets.

Then I'll provide an overview of the Vault Kubernetes Auth Method, which is used to facilitate authentication with Vault by using a Kubernetes Service Account Token. This method of authentication leverages native Kubernetes identity and access management, and makes it easy to introduce a Vault token into a Kubernetes Pod.

Finally, I'll provide an overview of a Vault deployment scenario on OKE, whereby Vault will be both deployed to OKE, and integrated with the OKE cluster control plane using the Vault Kubernetes Auth Method. This deployment scenario is fully documented in this work instruction, including a step-by-step guide and necessary configuration files.

Oracle Cloud Infrastructure Container Engine for Kubernetes (often abbreviated as `OKE`) is a fully-managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. Use OKE when your development team wants to reliably build, deploy, and manage cloud-native applications. OKE uses Kubernetes - the open-source system for automating deployment, scaling, and management of containerized applications across clusters of hosts.

Vault

Secrets management is one of the core use cases for Vault. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any type of secret, while providing important features such as an API driven framework delivering tight access control, version control, and detailed audit log.

Many organisations have credentials hard coded in source code, littered throughout configuration files and configuration management tools, and stored in plaintext in version control, wikis, and shared volumes. Vault provides a central place to store these credentials, ensuring they are encrypted, access is audit logged, and exposed only to authorized clients.

Vault provides a wide array of features across secrets management, data protection, identity-based access, collaboration & operations, and governance and compliance.

Vault is able to provide tight control over access to secrets and encryption keys by authenticating against trusted sources of identity, such as Active Directory, LDAP, Kubernetes, and cloud platforms. Vault enables fine grained authorization across which users and applications are permitted access to secrets and keys.

Kubernetes Auth Method

When working with Vault, successful authentication is a pre-requisite step for actors to store/retrieve secrets and perform cryptographic operations. Tokens are the core method for authentication within Vault which means that the secret consumer must first acquire a valid token. The Vault authentication process verifies the client's identity (secrets consumer), and then generates a token to associate with that identity.

Vault provides a range of Auth Methods to address application requirements where running on a variety of platforms. The Kubernetes Auth Method works well for Kubernetes based orchestrators such as OKE.

The Kubernetes Auth Method is used to authenticate with Vault using a Kubernetes Service Account Token. The token for a pod’s service account is automatically mounted within a pod at /var/run/secrets/kubernetes.io/serviceaccount/ token when a pod is instantiated. When using the Kubernetes Auth Method, it is this token which is sent to Vault for authentication.

Vault is configured with a service account that has permissions to access the Kubernetes TokenReview API. This service account is used by Vault to make authenticated calls to the Kubernetes API Server in order to verify service account tokens presented by pods that want to connect to Vault to access secrets.

The Vault deployment scenario illustration describes the solution architecture implemented in the referenced work instruction.

Flow 2, represented in orange provides a high level representation of the authentication workflow implemented via the Kubernetes Auth Method.

On successful completion, Vault returns a token to the application with pre-configured policies attached. From step 3 onward, the application can use the token to retrieve secrets from Vault’s Key/Value secrets engine.

Deployment Scenario

The following provides a break-down of the components provisioned in the deployment scenario:

• OKE Kubernetes cluster
• etcd and Vault Kubernetes operators
• Vault cluster
• Vault configured to use Kubernetes auth method
• Vault Key/Value (KV) store secret/testapp
• Vault role 'testapp' associated with the Kubernetes service account 'testapp' in the 'default' namespace
• Vault policy 'testapp-kv-crud' associated with 'testapp' role (providing CRUD access to the secret/testapp KV store)
• etcd cluster serving as persistent storage tier for the Vault cluster
• Test application authenticated to vault (via Kubernetes auth) using the testapp service account
• Test application creating and reading secrets from the Vault secret/testapp KV store

Both the etcd and Vault clusters will be created by their respective Kubernetes operators. The Vault operator deploys and manages Vault clusters on Kubernetes. Vault instances created by the Vault operator are highly available and support automatic failover and upgrade. For each Vault cluster, the Vault operator will also create an etcd cluster for the storage backend.

The following is a high level outline of the process described in the work instruction to install and integrate Vault with OKE:

1. Deploy the Vault & etcd operators
2. Deploy the Vault & etcd clusters
3. Configure Vault Kubernetes auth
4. Create the Vault Key/Value (KV) store & associated policy for the test application
5. Deploy the test application and authenticate to Vault using Kubernetes auth
6. Create and read secrets from the KV store

The work instruction is a great way to quickly get up and running with a comprehensive, HA deployment of Vault on OKE.

Summary

Integrating Vault authentication with Kubernetes using the Kubernetes Auth Method serves to simplify the process of authenticating Vault clients by using a Kubernetes Service Account Token.

Some challenges do remain around solving how to manage the lifecycle of tokens in a standard way, without having to write custom application logic. The Vault team are planning a number of features which are designed to address these challenges.

One of the proposed mechanisms involves integrating Vault with the Kubernetes Secrets mechanism via a periodically running sync process. Another involves a Container Storage Interface plugin to inject secrets into a running pod. There's also mention of injecting Vault secrets into Pods via a sidecar container - each very interesting approaches to solving for this complexity.

The Kubernetes and cloud native space is moving fast - watch this space as we will continue to share solutions, tutorials, best practices, and more, all designed to spark inspiration, drive hands-on experiences, and unleash our industry’s potential.

Cover Photo by JR Korpa on Unsplash.

In this blog post I'll show you how to configure a Kubernetes CronJob to run Oracle Cloud Infrastructure CLI commands automatically, on a recurring schedule.

In the example solution, we'll be scheduling the invocation of an Oracle Serverless Function.

About Kubernetes CronJobs

It's common for developers and operators to have a range of different tasks scheduled to run automatically in the background. These scheduled commands or tasks are typically known as “Cron Jobs”.

Kubernetes supports the creation of CronJobs, which is a mechanism for configuring Kubernetes to run containerised tasks on a time-based schedule. These automated jobs run like Cron tasks on a Linux or UNIX system.

Cron jobs are useful for creating periodic and recurring tasks, like running backups or sending emails. Cron jobs can also schedule individual tasks for a specific time, such as if you want to schedule a job for a low activity period.

About the OCI CLI

The Oracle Cloud Infrastructure CLI is a powerful, and easy to use tool that provides the same core capabilities as the Oracle Cloud Infrastructure Console, plus additional commands that can extend the Console's functionality. The CLI is convenient for developers or anyone who prefers the command line to a GUI.

The CLI supports orchestration and configuration of many OCI services, including Core Services (Networking, Compute, Block Volume), Database, Load Balancing, Serverless Functions, and many more. The complete list of supported services is available here.

About Oracle Functions

Oracle Functions is a fully managed, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform, built on enterprise-grade Oracle Cloud Infrastructure, and powered by the Fn Project open source engine.

With Oracle Functions, you can deploy your code, call it directly or trigger it in response to events, and get billed only for the resources consumed during the execution.

Oracle Functions are "container-native". This means that each function is a completely self-contained Docker image that is stored in your OCIR Docker Registry and pulled, deployed and invoked when you invoke your function.

Tutorial Overview

First, we're going to be building a container image containing the OCI CLI, then we'll configure a Kubernetes Secret to host your CLI configuration parameters and credentials.

Once we've pushed our container image to the OCI Registry, we'll schedule our containerised OCI CLI to invoke our Serverless Function via a Kubernetes CronJob.

Prerequisites

First, let's implement and configure the components that you need in order to deploy an OCI CLI command as a scheduled CronJob on Container Engine for Kubernetes.

You will need:

• An Oracle Container Engine for Kubernetes cluster that is up and running
• An Oracle Cloud Function provisioned in your OCI tenancy
• Oracle Cloud Infrastructure CLI and kubectl command lines installed and configured on your development workstation

Deployment Process

1. Create Kubernetes Secret

In the process of working through the prerequisites section, you will have installed and configured the OCI CLI on your development workstation. As a part of this process there was the requirement to configure the CLI config file, and OCI API private key.

We're now going to copy and store these artifacts inside a Kubernetes Secret, which will be used to authenticate the CLI to your OCI tenancy each time our scheduled task runs.

From your development workstation, run the following command:

kubectl create secret generic oci-cli-config --from-file=<oci-config-file> --from-file=<rsa-private-key>

Substitute the values <oci-config-file> and <rsa-private-key> with the appropriate path directives to the files on your development workstation, e.g.

kubectl create secret generic oci-cli-config --from-file=./.oci/config --from-file=./.oci/ssh/id_rsa_pri.pem

The OCI config file and RSA private key will be automatically mounted into the container filesystem at the path /root/.oci at runtime.

As this is the default file system location for the OCI CLI config files, with no further configuration the CLI will use these files to authenticate to your OCI tenancy each time the scheduled task is invoked.

2. Build the CLI container image

Create a directory on your development workstation named oci-fn-cron and create a file named Dockerfile in the directory with the following content:

FROM oraclelinux:7-slim
ENV OCI_CLI_SUPPRESS_FILE_PERMISSIONS_WARNING=True
ENV PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rh/rh-python36/root/usr/bin:${PATH}"
ENV LC_ALL=en_US.utf8
ENV LANG=en_US.utf8
ARG CLI_VERSION=2.6.14
RUN mkdir /oci-cli
WORKDIR /oci-cli
RUN yum -y install oracle-release-el7 && \
    yum -y install oracle-softwarecollection-release-el7 && \
    yum-config-manager --enable software_collections && \
    yum-config-manager --enable ol7_latest ol7_optional_latest ol7_addons && \
    yum-config-manager --disable ol7_ociyum_config && \
    yum -y install scl-utils && \
    yum -y install rh-python36 && \
    yum -y install gcc && \
    yum -y install wget && \
    yum -y install unzip && \
    rm -rf /var/cache/yum
RUN wget -qO- -O oci-cli.zip "https://github.com/oracle/oci-cli/releases/download/v${CLI_VERSION}/oci-cli-${CLI_VERSION}.zip" && \
    unzip -q oci-cli.zip -d .. && \
    rm oci-cli.zip && \
    pip3 install oci_cli-*-py2.py3-none-any.whl && \
    yes | oci setup autocomplete
ENTRYPOINT ["/opt/rh/rh-python36/root/usr/bin/oci"]

From the oci-fn-cron directory, run the following command to build the CLI container image:

docker build -t oci-fn-cron .

3. Push the container image to the OCI registry

In this next step, we'll push the local copy of the image up to the cloud. For a great walkthrough on how to use the OCI Registry service, check out this article.

You will need to log into your Oracle Cloud Infrastructure console. Your user will either need to be a part of the tenancy's Administrators group, or another group with the REPOSITORY_CREATE permission.

After confirming you have the proper permissions, generate an auth token for your user. Be sure to take a copy of the token as you will not be able to access it again.

In the OCI console, navigate to the Developer Services | Registry (OCIR) tab, and select the OCI region to which you would like to push the image. This should be the same region into which you provisioned your OKE cluster.

Log into the OCI registry

Log into the OCI registry in your development environment using the docker login command:

docker login <region-key>.ocir.io

<region-key> corresponds to the code for the Oracle Cloud Infrastructure region you're using. See the this reference for the available regions and associated keys.

When prompted, enter your username in the format <tenancy_name>/<username>. When prompted, enter the auth token you copied earlier as the password.

Tag the container image

Next we'll tag the OCI CLI image that we're going to push to the OCI registry:

docker tag oci-fn-cron:latest <region-code>.ocir.io/<tenancy-name>/oci-cron/oci-fn-cron:latest

Push the image to the OCI registry

And now we'll use the docker push command to push the conainer image to the OCI registry:

docker push <region-code>.ocir.io/<tenancy-name>/oci-cron/oci-fn-cron:latest

Within the OCI console Registry UI you will now be able to see the newly created repository & image.

4. Schedule the Kubernetes CronJob

On your development workstation create a file named oci-fn-cron.yaml with the following content:

kind: CronJob
apiVersion: batch/v1beta1
metadata:
  name: oci-functions-cron
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: oci-functions-cron
            image: <region-code>.ocir.io/<tenancy-name>/oci-cron/oci-fn-cron:latest
            command: ["/opt/rh/rh-python36/root/usr/bin/oci"]
            args: ["--debug", "fn", "function", "invoke", "--function-id", "<function-ocid>", "--file", "-", "--body", "-"]
            imagePullPolicy: Always
            volumeMounts:
            - name: oci-cli-config
              mountPath: "/root/.oci"
              readOnly: true
            ports:
            - containerPort: 9081
          restartPolicy: Never
          volumes:
          - name: oci-cli-config
            secret:
              secretName: oci-cli-config
              items:
              - key: config
                path: config
              - key: id_rsa_pri.pem
                path: ssh/id_rsa_pri.pem

Remember to update the following:

• Line 13 with your OCI region-code and tenancy-name
• Line 15 function-ocid with the OCID of the Oracle Function you wish to invoke

Save the file, and submit the CronJob with the following command:

kubectl apply -f cronJob.yaml

Verifying CronJob operation

You can validate that your CronJob is functioning correctly by following this procedure:

1. Obtain the job execution history by entering the command:

kubectl get jobs --watch

The output will look similar to the following:

NAME                            COMPLETIONS   DURATION   AGE
oci-functions-cron-1575886560   1/1           43s        4m45s
oci-functions-cron-1575886680   1/1           34s        2m44s
oci-functions-cron-1575886800   1/1           35s        44s

2. Enter the following command to obtain the pod name associated with the scheduled job, replacing <job-name> with the job name received via the previous command output:

kubectl get pods --selector=job-name=<job-name> --output=jsonpath={.items[*].metadata.name}

3. Enter the following command to obtain the logs associated with the executed CLI command, replacing <pod-name> with the pod name name received via the previous command output:

kubectl logs <pod-name>

If your function was invoked correctly, the output will look similar to the following - which is the log data generated by the CLI running the fn function invoke command which we defined in the Kubernetes CronJob:

INFO:oci.base_client.140263433586560: 2019-12-09 21:54:04.502876: Request: GET https://functions.us-ashburn-1.oci.oraclecloud.com/20181201/functions/ocid1.fnfunc.oc1.iad.aaaaaaaaadfmkqscppi63jistu4t7au2veexg5in6lykzovzmvaja6vqmwsa
DEBUG:oci.base_client.140263433586560: 2019-12-09 21:54:04.601030: time elapsed for request 4AFF967835C440009D15F3CFAAC404D2: 0.0978535171598196
DEBUG:oci.base_client.140263433586560: 2019-12-09 21:54:04.601216: time elapsed in response: 0:00:00.092832
DEBUG:oci.base_client.140263433586560: 2019-12-09 21:54:04.601319: Response status: 200
DEBUG:oci.base_client.140263433586560: 2019-12-09 21:54:04.603549: python SDK time elapsed for deserializing: 0.0020453811157494783
DEBUG:oci.base_client.140263433586560: 2019-12-09 21:54:04.603681: Response returned
DEBUG:oci.base_client.140263433586560:time elapsed for request: 0.1009602730628103
INFO:oci.base_client.140263433271056: 2019-12-09 21:54:04.608421: Request: POST https://newg3h4jqoq.usashburn1.functions.oci.oraclecloud.com/20181201/functions/ocid1.fnfunc.oc1.iad.aaaaaaaaadfmkqscppi63jistu4t7au2veexg5in6lykzovzmvaja6vqmwsa/actions/invoke
DEBUG:oci.base_client.140263433271056: 2019-12-09 21:54:37.288483: time elapsed for request 7C625DE5724D4C3B8E26A771D3F7F87B: 32.679952513892204
DEBUG:oci.base_client.140263433271056: 2019-12-09 21:54:37.288662: time elapsed in response: 0:00:32.676453
DEBUG:oci.base_client.140263433271056: 2019-12-09 21:54:37.288778: Response status: 200
DEBUG:oci.base_client.140263433271056: 2019-12-09 21:54:37.288893: Response returned
DEBUG:oci.base_client.140263433271056:time elapsed for request: 32.68057371187024

Altering the CronJob Schedule

With Kubernetes, one CronJob object is like one line of a crontab (cron table) file. It runs a job periodically on a given schedule, written in Cron format. For the sake of example, the above CronJob object will invoke the Oracle Function every 5 minutes.

To have your function run on a different schedule - simply modify the schedule as defined on line 6 in oci-fn-cron.yaml, and resubmit the CronJob.

Scheduling Different CLI Operations

In the example solution we've scheduled the OCI CLI to invoke an Oracle Serverless Function on a regular interval.

Now that you have the solution in place, it's actually very easy to schedule additional Kubernetes CronJobs that execute different CLI commands.

All that's required is to create additional CronJob yaml files, in each updating the job name (line 4 in the example oci-fn-cron.yaml) and the OCI CLI command to execute via args (line 15 in the example oci-fn-cron.yaml) to suit your requirements.

The complete list of  services supported by the OCI CLI is available here, and the range of possible use-cases is almost limitless!

Cover Photo by Paweł Czerwiński on Unsplash.

When working with both the Fn Project and the Oracle Functions service, it's possible to access information about the invocation, function, and execution environment from within a running function - including HTTP properties such as custom HTTP headers and HTTP query parameters.

In this blog post, I'll show you how to work with HTTP requests when building your Fn functions using the Python Function Development Kit (FDK).

Scenario

I was recently working on an IoT implementation where devices had been configured to remotely invoke an Oracle Function in order to retrieve configuration data.

When invoked by an IoT device, the Oracle Function was configured to connect to a downstream system to retrieve configuration data specific to the requesting device, and in-turn respond with a JSON payload.

In order to meet its functional requirement, the Oracle Function (in this case, implemented in Python) required at runtime, access to a custom HTTP header submitted by the requesting device.

Along the lines of the example scenario - accessing information about the invocation, function, and execution environment when working with serverless functions is a typical requirement.

The Fn Project implements a number of features, including Function Development Kits (FDKs) to simplify and standardise the developer experience when working with such data.

About the Fn Project and Oracle Functions

Oracle Functions is a fully managed, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform, built on enterprise-grade Oracle Cloud Infrastructure, and powered by the Fn Project open source engine.

With Oracle Functions, you can deploy your code, call it directly or trigger it in response to events, and get billed only for the resources consumed during the execution.

Oracle Functions are "container-native". This means that each function is a completely self-contained Docker image that is stored in your OCIR Docker Registry and pulled, deployed and invoked when you invoke your function.

Fn Project Function Development Kits

The Fn Project provides Function Development Kits (FDKs) with support for a variety of programming languages, including: Java, Python, Ruby, Node, & Go.

FDKs are designed to abstract away from developers the requirement to interact directly with underlying low-level constructs, or to perform complex work such as protocol framing.

At runtime FDKs execute in three phases, and in the following order:

• Request: with request data deserialised into a request context and request data
• Execute: with request context and request data
• Respond: with response data being rendered into a formatted response

At the time you create an Fn function, you specify a handler, which is a function in your code, that Oracle Functions can invoke when the service executes your code.

The handler accepts a callable object: fdk.handle({callable_object}), and the callable object implements a signature with the format: (context, data).

The following general syntax structure is used when creating a handler function in Python.

def handler(ctx, data):
    ...
    return response

The request data is obtained from the request used to trigger the function. In the case of an HTTP invocation, data is an HTTP request body.

When Oracle Functions invokes your function, it passes a context object ctx to the handler. This object provides methods and properties that provide information about the invocation, function, and execution environment.

FDK Request Context

The following describes the range of data exposed by the attributes of a request context object when working with the Python FDK:

Config
   Class: os._Environ
   Configuration data for the current application and current function.
Headers
   Class: dict
   HTTP headers included with the request submitted to invoke the current function.
AppID
   Class: str
   Unique Oracle Cloud ID (OCID) assigned to the application.
FnID
   Class: str
   Unique Oracle Cloud ID (OCID) assigned to the function.
CallID
   Class: str
   Unique ID assigned to the request.
Format
   Class: str
   The function’s communication format - the interaction protocol between Fn and the function.
Deadline
   Class: str
   How soon function the will be aborted, including timeout date and time.
RequestURL
   Class: str
   Request URL that was used to invoke the current function.
Method
   Class: str
   HTTP method used to invoke the current function.

Example Function

Here's an example of a simple hello world function which will include within the response data the full set of attributes exposed by the request context object:

import io
import json
from fdk import response
 
def handler(ctx, data: io.BytesIO=None):
    name = "World"
    try:
        body = json.loads(data.getvalue())
        name = body.get("name")
    except (Exception, ValueError) as ex:
        print(str(ex))
    return response.Response(
        ctx, response_data=json.dumps(
            {"Message": "Hello {0}".format(name),
            "ctx.Config" : dict(ctx.Config()),
            "ctx.Headers" : ctx.Headers(),
            "ctx.AppID" : ctx.AppID(),
            "ctx.FnID" : ctx.FnID(),
            "ctx.CallID" : ctx.CallID(),
            "ctx.Format" : ctx.Format(),
            "ctx.Deadline" : ctx.Deadline(),
            "ctx.RequestURL": ctx.RequestURL(),
            "ctx.Method": ctx.Method()},
            sort_keys=True, indent=4),
        headers={"Content-Type": "application/json"}
    )

When invoked, the hello world function response data is returned as JSON formatted:

    "Message": "Hello World",
    "ctx.AppID": "ocid1.fnapp.oc1.iad.aaaaaaaaafkiyvdtalsn6aako2i6jttllk7tgaj4v4hgpnccwggd00000000",
    "ctx.CallID": "01E3BCBYFR1BT163GZ00000000",
    "ctx.Config": {
        "FN_APP_ID": "ocid1.fnapp.oc1.iad.aaaaaaaaafkiyvdtalsn6aako2i6jttllk7tgaj4v4hgpnccwggd00000000",
        "FN_CPUS": "100m",
        "FN_FN_ID": "ocid1.fnfunc.oc1.iad.aaaaaaaaadjx7atmmfcbm6ipfw67bykoh2lniadadurqiex2p3d500000000",
        "FN_FORMAT": "http-stream",
        "FN_LISTENER": "unix:/tmp/iofs/lsnr.sock",
        "FN_MEMORY": "256",
        "FN_TYPE": "sync",
        "GPG_KEY": "0D96DF4D4110E5C43FBFB17F2D347EA600000000",
        "HOME": "/home/fn",
        "HOSTNAME": "71c2e839ca59",
        "LANG": "C.UTF-8",
        "OCI_RESOURCE_PRINCIPAL_PRIVATE_PEM": "/.oci-credentials/private.pem",
        "OCI_RESOURCE_PRINCIPAL_REGION": "us-ashburn-1",
        "OCI_RESOURCE_PRINCIPAL_RPST": "/.oci-credentials/rpst",
        "OCI_RESOURCE_PRINCIPAL_VERSION": "2.2",
        "PATH": "/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "PYTHONPATH": "/python",
        "PYTHON_GET_PIP_SHA256": "b86f36cc4345ae87bfd4f10ef6b2dbfa7a872fbff70608a1e43944d283fd0eee",
        "PYTHON_GET_PIP_URL": "https://github.com/pypa/get-pip/raw/ffe826207a010164265d9cc807978e3604d18ca0/get-pip.py",
        "PYTHON_PIP_VERSION": "19.3.1",
        "PYTHON_VERSION": "3.6.9",
        "card": "not_set"
    },
    "ctx.Deadline": "2020-03-14T02:01:58Z",
    "ctx.FnID": "ocid1.fnfunc.oc1.iad.aaaaaaaaadjx7atmmfcbm6ipfw67bykoh2lniadadurqiex2p3d500000000",
    "ctx.Format": "http-stream",
    "ctx.Headers": {
        "accept": "*/*",
        "accept-encoding": "gzip",
        "content-type": "application/octet-stream",
        "date": "Sat, 14 Mar 2020 02:01:03 GMT",
        "fn-call-id": "01E3BCBYFR1BT163GZ00000000",
        "fn-deadline": "2020-03-14T02:01:58Z",
        "fn-http-method": "GET",
        "fn-http-request-url": "/gtc/ml?card=CC50E3CCBFFF",
        "fn-intent": "httprequest",
        "fn-invoke-type": "sync",
        "host": "localhost",
        "oci-subject-id": "ocid1.apigateway.oc1.iad.amaaaaaap7nzmjiajosozyrcbvtwhtqdd4zvlojl4qn4teauugge00000000",
        "oci-subject-tenancy-id": "ocid1.tenancy.oc1..aaaaaaaac3l6hgylozzuh2bxhf3557quavpa2v6675u2kejplzal00000000",
        "oci-subject-type": "resource",
        "opc-request-id": "/7FAB0BCD835B93B731AF16E754880DA2/01E3BCBYD01BT163GZ00000000",
        "orwarded": "for=111.111.111.111",
        "ost": "b65alubkumgiuzhn3400000000.apigateway.us-ashburn-1.oci.customer-oci.com",
        "transfer-encoding": "chunked",
        "user-agent": "curl/7.47.0",
        "x-content-sha256": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZ00000000",
        "x-forwarded-for": "111.111.111.111",
        "x-forwarded-host": "ggd00000000.us-ashburn-1.functions.oci.oraclecloud.com:443",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https",
        "x-real-ip": "111.111.111.111",
        "x-device-id": "CC50E3CCB000"
    },
    "ctx.Method": "GET",
    "ctx.RequestURL": "/gtc/ml?device-id=CC50E3CCB000"

With the IoT scenario in mind, and in reference to the JSON response data - it's apparent that we're able to obtain the requesting IoT device unique ID from the custom HTTP header x-device-id, which is available via the request context object attribute Headers.

Another option for submitting custom data to the function to be available via the request context object is to include HTTP request parameters within the HTTP request submitted to invoke a function.

HTTP request parameters are exposed by the request context object attribute requestURL. Per the example response data, the IoT device ID was also submitted via HTTP request parameter device-id, and exposed by the request context object attribute requestURL.

There's a wealth of contextual and environmental data exposed by the request context object, and through the context object it's easily available for a developer to work with when constructing functions in Python.

If you're not already working with Oracle Functions on OCI, get started today by heading over to https://www.oracle.com/cloud/free/ to access a free trial, and unlock access to the always free services.

Cover Photo by Steve Johnson on Unsplash.

Welcome to the byteQualia blog!

byteQualia

The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer.

There are many definitions of qualia, which have changed over time. One of the simpler, broader definitions is: The 'what it is like' character of mental states.

This is a personal weblog. Any views or opinions represented in this weblog are personal and belong solely to the blog author, and do not represent those of people, institutions, organisations that the author may or may not be associated with in professional or personal capacity, unless explicitly stated.

Any views or opinion are not intended to malign any religion, ethnic group, club, organisation, company, or individual.

Photo by Susan Yin on Unsplash.