Friday, June 11, 2021

Creating a Self Signed Certificate with RootCA using openssl

 Introduction

Over the years I have lost count of the number of times I have had to create a self-signed certificate for one reason or another and every time I do it I have forgotten the various commands to create one so I thought I would actually write them down in a small blog post that I can refer back to without going through an internet search engine to find the answers.  :-)


The Root CA

It is handy to create a single root CA sometimes then this can be distributed to the various trust stores and all certs created using this rootCA are then trusted.  

1. Create a root CA key file

$ openssl genrsa -out rootCA.key 4096

2. Create and sign the root certificate.

$ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt

This will create a file called rootCA.crt.  This can be used to put into trust stores.
For Debian/Ubuntu copy this file to /usr/local/share/ca-certificates/ and run the command update-ca-certificates

$ sudo cp rootCA.crt /usr/local/share/ca-certificates
$ sudo update-ca-certificates


The Certificate (With SANs)

First create a file with the details you want in the certificate.  Specifically any Subject Alternative Names you want the certificate to be created with.


$ cat myhost-sans.cnf
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
countryName                = Two Letter code
stateOrProvinceName        = State or Province
localityName               = Area or City
organizationName           = Organisation
commonName                 = myhost.my-domain.com
[ req_ext ]
subjectAltName = @alt_names
[ v3_ca ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = myhost.mydomain.com
DNS.2   = myhost
DNS.3   = my-alternative-name.mydomain.com

When creating a certificate the above file can be used to create the Certificate Signing Request.  What is needed when getting a "real" certificate from an authority.  However openssl will not add the SANs to the certificate it creates unless we actively specify it to do so with another file.  

$ cat myhost-cert-sans.cnf
[v3_ca]
subjectAltName = DNS:myhost.mydomain.com, DNS:my-host, DNS: my-alternative-name.mydomain.com

Now we are ready to create the certificate.

1. Create the key file

$ openssl genrsa -out my-cert.key 2048

2. Create and Sign the certificate signing request

$ openssl req -new -sha256 -key my-cert.key -config ./myhost-sans.cnf -out my-host.csr

This will interactively prompt you for the various certificate names.    We can check the content of the certificate signing request

$ openssl req -in my-host.csr -noout -text

3. Assuming all looks OK then we can use this to generate the certificate.

$ openssl x509 -req -in my-host.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out my-host.crt -days 1024 -sha256 -extensions v3_ca -extfile myhost-cert-sans.cnf

This will create the certificate file my-host.crt which can then be used to encrypt your website, image repository, REST end point etc.

You can check the certificate in the same way that was done for the request.

$ openssl x509 -in my-host.crt -noout -text


Monday, December 21, 2020

Using Tanzu Service Manager to expose Tanzu Postgres services to Cloud Foundry

There have been a few products under the Tanzu brand which have fairly recently been made generally available.  These being Tanzu Postgres for Kubernetes and the Tanzu Service Manager.  A very useful combination of products which can be used to make the Postgres database available to applications running on cloud Foundry (Or Tanzu Application Service - TAS) environment.

So what are these products and how to they operate.  Firstly Tanzu Postgres, this is simply a deployment of the OSS Postgres database with the support of VMWare.  Under the guise of Pivotal there has been a commercial offering for Postgres on Virtual Machines for many years and in recent times the engineering effort has been expended to migrate this wonderful database such that it is containerised and can run on the Kubernetes orchestration engine.  

The distribution has been split into a couple of components, firstly the deployment of an operator which can be used to manage the lifecycle of a postgres instance.  This makes the creation of an instance as simple as applying a yaml file configuration to Kubernetes.

Tanzu Service Manager (TSMGR) is a product which runs on Kubernetes and provides an OSBAPI (Open Service Broker API) interface that is integrated with the Cloud Controller of Cloud Foundry.  It manages helm chart based services that run on Kubernetes and makes them available to applications and application developers in Cloud Foundry such that the service looks like it has a fully native integration.  

So if we put these two together we have a fully supported version of Postgres running on Kubernetes that can be made available to applications running in the Tanzu Application Service.  

For an example installation of Tanzu Postgres with the Service Manager have a look at my github repo.




Wednesday, September 9, 2020

Deploying Tanzu Application Service to Kubernetes (TKG)

 Introduction

Tanzu Application Service (TAS) is a powerful Platform as a Service abstraction  which can be deployed on many different cloud infrastructure providers.  It allows developers to write their application in any of the commonly used programming language and "push" their code to a runtime which takes care of all of the provisioning required to allow the program to run and does the plumbing to make the application accessible.

TAS has been available for many years now, based on the Open Source Cloud Foundry Application Service.   At time of writing VMWare have been moving TAS from running on Virtual Machines to run as containers on a kubernetes environment.  Currently in Beta the first GA release should be available in the fall of 2020.  (Or as I would say Autumn.)

This blog post runs through the steps involved to get it up and running, essentially summarising the information in the documentation. (Note - this is a link to beta documentation and the URL may well change soon.)

Installation

Start by downloading the TAS system which can be downloaded as a single .tar file.  When extracted will create a directory called tanzu-application-service with a number of subdirectories.  For this blog posting using the 0.4.0 Beta version of TAS, the installation process may significantly change over coming releases as the product goes GA.

Kubernetes

To get started we need to create a kubernetes cluster to deploy TAS4k8s onto, in this example I am using TKG running in an AWS environment.  (See posting http://ablether.blogspot.com/2020/08/first-experiences-with-tanzu-kubernetes.html about setting up TKG on AWS)
TAS will create quite a few containers and the current default setup is a minimal install with only an instance of most components.  According to the documentation you need at least 5 worker nodes with a minimum of 2CPU/7.5Gb RAM, this equates to worker nodes of t3.large on AWS.  I found that I could get away with 5 t3.medium sized ones but in order to deploy a few of your own containers I recommend 7 workers at t3.large size.

% tkg create cluster clust-tas --plan dev -w 7 --worker-size t3.large
Logs of the command execution can also be found at: /var/folders/nn/p0h624x937l6dt3bdd00lkmm0000gq/T/tkg-20200827T154322852129303.log
Validating configuration...
Creating workload cluster 'clust-tas'...
Waiting for cluster to be initialized...
Waiting for cluster nodes to be available...

Workload cluster 'clust-tas' created

Once our cluster is up and running we need to get the context to it for kubectl and then we can deploy a couple of additional capabilities in preparation for TAS.



% tkg get credentials clust-tas
Credentials of workload cluster 'clust-tas' have been saved
You can now access the cluster by running 'kubectl config use-context clust-tas-admin@clust-tas'
% kubectl config get-contexts
CURRENT   NAME                                      CLUSTER            AUTHINFO                 NAMESPACE
          aws-mgmt-cluster-admin@aws-mgmt-cluster   aws-mgmt-cluster   aws-mgmt-cluster-admin
          clust-tas-admin@clust-tas                 clust-tas          clust-tas-admin
          minikube                                  minikube           minikube
*         test-cluster-admin@test-cluster           test-cluster       test-cluster-admin
% kubectl config use-context clust-tas-admin@clust-tas
Switched to context "clust-tas-admin@clust-tas".


There are two things we need to do to the newly created cluster, firstly we need to install the metrics server which goes into the kube-system namespace.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
Installing the default storage class for aws gp2 based ext4 storage.

Then for AWS we want to set a default storage class.  Obviously the choice here may depend on what you feel you need for your environment, for the purposes of the test environment we will be using a standard gp2 storage from AWS.

Inside the download directory there is a config-optional directory that contains a yaml file called aws-default-storage-class.yml.  Use this to upload to your Kubernetes cluster.  This introduces another tool which is part of the suite of tools known as k14s or Carvel.  They can be downloaded from k14s.io (or get-kapp.io or for TAS specifically VMWare package a version).    kapp -  Kubernetes Application Management Tool is a CLI designed to manage resources in bulk and is used to package up and deploy/manage TAS4k8s.


$ kapp deploy -a default-storage-class -f <download dir>/tanzu-application-service/config-optional/aws-default-storage-class.yml
Target cluster 'https://clust-tas-apiserver-352766885.eu-west-2.elb.amazonaws.com:6443' (nodes: ip-10-0-0-56.eu-west-2.compute.internal, 7+)

Changes

Namespace  Name  Kind          Conds.  Age  Op      Wait to    Rs  Ri
(cluster)  gp2   StorageClass  -       -    create  reconcile  -   -

Op:      1 create, 0 delete, 0 update, 0 noop
Wait to: 1 reconcile, 0 delete, 0 noop

Continue? [yN]: y

9:43:18AM: ---- applying 1 changes [0/1 done] ----
9:43:18AM: create storageclass/gp2 (storage.k8s.io/v1) cluster
9:43:18AM: ---- waiting on 1 changes [0/1 done] ----
9:43:18AM: ok: reconcile storageclass/gp2 (storage.k8s.io/v1) cluster
9:43:18AM: ---- applying complete [1/1 done] ----
9:43:18AM: ---- waiting complete [1/1 done] ----

Succeeded

Tanzu Application Service

The first step towards configuring TAS is to decide on what domain name to use.  I have a domain donaldforbes.com which I use for dev/testing and so decided to use a system domain of sys.tas.donaldforbes.com for my deployment.  This domain points to a load balancer which I will run in AWS to handle all the ingress.  (wildcard DNS used)

Next we need to create a set of configuration values which are specific to this install.  Create a configuration-values directory and then run the tanzu-application-service/bin/generate-values.sh script to populate configuration parameters for your environment.   The script generates a file with a set of config values - it uses self signed certificates by default and the various passwords for TAS components.

$ ./bin/generate-values.sh -d sys.tas.donaldforbes.com > configuration-values/deployment-values.yml

Using AWS we will also use a load balancer to manage ingress to the platform.  This requires the presence of another file in the configuration-values directory.  Called load-balancer-values.yml

$ cat load-balancer-values.yml
#@data/values
---
enable_load_balancer: True

We also need to be able to access two different registries (they could be the same). Firstly a system registry which is used by the TAS installer to reach out for the images used to run the various components/containers of TAS itself.  This is configured in a file called system-registry-values.yml

$ cat system-registry-values.yml
#@data/values
---
system_registry:
  hostname: "registry.pivotal.io"
  username: "dforbes@pivotal.io"
  password: "<RedactedPassword>"

To gain access simply sign up to https://network.pivotal.io.

The second registry is one that TAS will use to store the containers it creates for every application that is deployed.  Most registries will be possible to used for this.  Simply create a file called app-registry-values.yml and populate it with values to enable the login to the registry and ensure that the user has read/write access.  In my example below I decided to use the docker registry on docker hub.

$ cat app-registry-values.yml
#@library/ref "@github.com/cloudfoundry/cf-for-k8s"
#@data/values
---
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "donaldf"
  username: "donaldf"
  password: "<Redacted Password>"


Once done we are ready to install TAS.  Simply run the install-tas.sh script in the bin directory of tanzu-application-service passing in to it a parameter which points to your custom configuration directory.

$ ./bin/install-tas.sh ./configuration-values

This script will do all the kubernetes configuration, upload the images from the system repository and create the necessary deployments.  Typically this process will take several minutes to complete.

Once done there is a load balancer created for the ingress.  You must point your DNS to this load balancer to access TAS.  (I actually used route 53 to handle the wildcard DNS need to point to the load balancer and used a CNAME from my domain registration.)



Thursday, August 27, 2020

First Experiences with Tanzu Kubernetes Grid on AWS

 Introduction

Kubernetes as a container orchestration system has been about for a while now and many vendors have jumped onto it to provide supported builds of it which are simple to install, update and manage when compared to the pure play open source kubernetes distribution.  VMWare currently provide a couple of fully supported distributions which are known under the Tanzu brand banner.  

  • TKG - Tanzu Kubernetes Grid (Formally known as the Heptio distribution)
  • TKGI - Tanzu Kubernetes Grid Integrated (Formally known as Pivotal Container Service - PKS)
This blog post is really to provide a simple guide to deploying TKG into an Amazon account, subsequent blog posts will dive into various features/components/use cases for such an environment.

Installation

Obviously the details of the installation may change with time, this summary is for TKG 1.1 and the docs provide the full details for step by step installation instructions.  

In summary the steps to follow are:-

  1. Download and install the tkg and clusterawsadm command lines from the VMWare Tanzu downloads.
  2. Setup environment variables to run clusterawsadm which will bootstrap the AWS account with necessary groups, profiles, roles and users.  (Only needs done once per AWS account)
    $ clusterawsadm alpha bootstrap create-stack
  3. The configuration for the first management cluster can be either done via a UI.  A good choice the first time you do this as it guides you through the setup.  The UI is available through a browser having run init -ui and then access localhost:8080/#/ui.
    $ tkg init --ui
    Or you can setup the ~/.tkg/config.yaml file with the AWS environment variables required.  (A template for this file can be created by running $ tkg get management-cluster. ). Once setup run the command to create a management cluster.
    $ tkg init --infrastructure aws --name aws-mgmt-cluster --plan dev
At this point what is happening behind the scenes is that the tkg command line actually runs a small kubernetes cluster locally using docker, this cluster includes the clusterapi custom resource and it understands the AWS API and is able to create VMs using the AWS infrastructure and ensure that the tkg kubernetes executables are deployed to it.  The number and size of the VMs are controllable via the plan used and other flags to specify the size and number of workers/masters.  

Even for a small plan this process will take several minutes to complete.  (on my laptop15mins)

% tkg init --infrastructure aws --name aws-mgmt-cluster --plan dev
Logs of the command execution can also be found at: /var/folders/nn/p0h624x937l6dt3bdd00lkmm0000gq/T/tkg-20200827T093026003038156.log

Validating the pre-requisites...

Setting up management cluster...
Validating configuration...
Using infrastructure provider aws:v0.5.4
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /Users/dforbes/.kube-tkg/tmp/config_lieTADrw
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.6" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.6" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.6" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v0.5.4" TargetNamespace="capa-system"
Start creating management cluster...
Saving management cluster kuebconfig into /Users/dforbes/.kube/config
Unable to persist management cluster aws-mgmt-cluster info to tkg config
Installing providers on management cluster...
Fetching providers
Installing cert-manager
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.6" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.6" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.6" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v0.5.4" TargetNamespace="capa-system"
Waiting for the management cluster to get ready for move...
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Context set for management cluster aws-mgmt-cluster as 'aws-mgmt-cluster-admin@aws-mgmt-cluster'.

Management cluster created!


You can now create your first workload cluster by running the following:

  tkg create cluster [name] --kubernetes-version=[version] --plan=[plan]

%

At the end of this process there will be several VMs created in (a bastion server, a master and a worker for the basic dev plan.) a load balancer to allow ingress to the services and security groups etc. to restrict direct access to the cluster.

The tkg init command will by default create a cluster context in your .kube/config file called <cluster name>-admin@<cluster name>.  If other machines/operators are to manage the cluster then the tkg option of --kubeconfig will allow the access context to be stored in a separate file.  (Yup, I deleted the context from my config file then found there was no easy way to re-generate it!)

Testing

First off try a few basic commands

% tkg get management-clusters
 MANAGEMENT-CLUSTER-NAME  CONTEXT-NAME
 aws-mgmt-cluster *       aws-mgmt-cluster-admin@aws-mgmt-cluster

Shows the management cluster we just crated.

% tkg get clusters
 NAME  NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES

No clusters created yet so not a lot of useful information here.  Lets create our first cluster.

% tkg create cluster test-cluster --plan dev
Logs of the command execution can also be found at: /var/folders/nn/p0h624x937l6dt3bdd00lkmm0000gq/T/tkg-20200827T110852116171506.log
Validating configuration...
Creating workload cluster 'test-cluster'...
Waiting for cluster to be initialized...
Waiting for cluster nodes to be available...

Workload cluster 'test-cluster' created

%

This creates a very small cluster with 3 nodes.  A bastion host which is connected to a public network and two VMs on a private network, 1 master or control plane node and 1 worker.  Typically this will take up to 10 minutes to create the VMs and configure the kubernetes cluster.



Next job is to configure kubectl to be able to connect to the cluster.  The tkg command line allows us to get an admin context for the cluster.

% tkg get credentials test-cluster
Credentials of workload cluster 'test-cluster' have been saved
You can now access the cluster by running 'kubectl config use-context test-cluster-admin@test-cluster' 
% kubectl config get-contexts
CURRENT   NAME                                      CLUSTER            AUTHINFO                 NAMESPACE
*         aws-mgmt-cluster-admin@aws-mgmt-cluster   aws-mgmt-cluster   aws-mgmt-cluster-admin
          minikube                                  minikube           minikube
          test-cluster-admin@test-cluster           test-cluster       test-cluster-admin
% kubectl config use-context test-cluster-admin@test-cluster
Switched to context "test-cluster-admin@test-cluster".
% kubectl get ns
NAME              STATUS   AGE
default           Active   10m
kube-node-lease   Active   10m
kube-public       Active   10m
kube-system       Active   10m

That's a kubernetes worker cluster up and ready for action.  An obvious next step might be to scale the cluster.  Again tkg has a very simple api for this.

% tkg get clusters
 NAME          NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES
 test-cluster  default    running  1/1           1/1      v1.18.3+vmware.1
% tkg scale cluster test-cluster -w 2
Successfully updated worker node machine deployment replica count for cluster test-cluster
workload cluster test-cluster is being scaled
% tkg get clusters
 NAME          NAMESPACE  STATUS    CONTROLPLANE  WORKERS  KUBERNETES
 test-cluster  default    updating  1/1           1/2      v1.18.3+vmware.1
.
. . . <about 5 minutes>
.
% tkg get clusters
 NAME          NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES
 test-cluster  default    running  1/1           2/2      v1.18.3+vmware.1

And finally once we have finished with the cluster we want to delete it.

% tkg delete cluster test-cluster
Deleting workload cluster 'test-cluster'. Are you sure?: y█
workload cluster test-cluster is being deleted

Note - there is also a tkg upgrade cluster command.  Makes it nice and simple to upgrade a given cluster just when you want.

Conclusion

A few things to download and startup and certainly a little more complex than using one of the public cloud providers kubernetes environment.  But if you pause for a second and think about the complexity of operation it is achieving in a short period of time it is very impressive and simple approach.  It also gives you access to cluster creation and management with the same experience/binaries running on multiple cloud providers or on-premise. Not to mention full control over the cluster lifecycle.

All in all I was suitably impressed at the ease of use and from nothing it would take no more than a morning or afternoon to have any size of k8s cluster created and ready for use.



Friday, May 18, 2018

Cloud at Customer - DNS

Introduction

In recently weeks I have been involved with a few engagements where there has been a confusion with the understanding of how DNS within a Cloud at Customer environment works.  This blog posting will be an attempt to clarify how it works at a reasonably high level.

DNS Overview

The Domain Name System is provided to enable a mapping from a fairly human readable form to an actual IP address which computers can use to route traffic.  There are essentially two components to the name, firstly the hostname of a machine and secondly a domain name.  eg. donald.mycompany.com where donald represents the hostname and mycompany.com represents what is called the domain.

On a linux system the resolution of DNS is typically specified in a file /etc/resolv.conf it will look something like this.

# Generated by NetworkManager
nameserver 194.168.4.100
nameserver 194.168.8.100


The nameserver entry specifies the IP address of a DNS server which will be used to change names into IP addresses.  We have two entries so that should the first one be unavailable for any reason then the second one will be used.  We can then use a unix utility called nslookup (part of the bind-utils package if you want to install it) which will allow us to query the DNS server.  Other utilities such as dig can also provide information from DNS.

$ nslookup oracle.com
Server:        194.168.4.100
Address:    194.168.4.100#53

Non-authoritative answer:
Name:    oracle.com
Address: 137.254.120.50



Note that this response includes the text "Non-authorative answer".   The reason for this is that all DNS servers will be in control of different domains and in our example we tried to lookup a domain called oracle.com.   From my laptop the DNS server is from my ISP so it knows nothing about the oracle.com domain.  In this situation the DNS server I talked to will forward the request on to a .com domain server which in turn is likely to delegate the request to the DNS server responsible (or authorative) for the oracle.com domain.  This returns the IP address which flows back through the DNS servers to my ISP which caches the answer and gives me the IP address to use for a lookup on oracle.com.

DNS on Oracle Cloud@Customer

Inside the control plane of an OCC environment there is a DNS server that runs.  This server is authorative over two domains of interest to a customer.  Specifically one in the format of <Subscription ID>.oraclecloudatcustomer.com and also one  of the format <Account>/<Region>.internal.  For example. S12345678.oraclecloudatcustomer.com and compute-500011223.gb1.internal.

cloudatcustomer.com domain


cloudatcustomer.com is a domain that Oracle own and run on behalf of the customer and is used to provide the URLs that the customer will interact with.  Utilise the UI for managing the rack or accessing the REST APIs for compute, PaaS, Storage etc.  So, for example the underlying compute REST API of a C@C environment will be something like https://compute.<Region>.ocm.<Subscription ID>.oraclecloudatcustomer.com.  So in our example above this would become https://compute.gb1.ocm.S12345678.oraclecloudatcustomer.com.

Clearly the domain <Subscription>.oraclecloudatcustomer.com is something that is not going to be known in any customer's DNS so normal DNS actions would take place and the customer's DNS would look up the domain chain to find a DNS server that is authorative for oraclecloudatcustomer.com.  As these are not publicly accessible this lookup would fail.  To fix this we need to change the customer's DNS to make it aware of the <Subscription ID>.oraclecloudatcustomer.com domain so it knows how to access a DNS service that will return an appropriate IP address and enable usage of the URL.  This is done by adding a forwarding rule to your customer's DNS service which forwards any request for a C@C URL to the DNS running in the C@C environment.

Once the forwarding rule is in place then the customer is able to start using the UI and REST APIs of the C@C so they can manage the platform.

internal domain

So now the customer can access the cloud at customer management screens and can create virtual machines to run their enterprise.  The next question is about what is the internal domain.  This essentially relates to configuration of the VM itself.  During the provisioning process there is a screen which requests the "DNS Hostname Prefix" as shown below. 






The name used in the DNS Hostname Prefix becomes part of the registration into the OCC DNS server to represent the host portion.  ie. When we create the VM it will have a DNS registration internally like df-web01.compute-500019369.gb1.internal which maps onto the private IP address allocated to the VM.  This means if you are creating multiple VMs together or using a PaaS service that produces multipleVMs (like JCS) then there are URLs available to enable them to talk to each other without you having to know the IP address.

Once a VM has been created it is possible to see the internal DNS name for the VM from the details page as shown below.



This can be resolved from within the VM as shown below.

[opc@df-vm01 ~]$ nslookup df-vm01
Server:        100.66.15.177
Address:    100.66.15.177#53

Name:    df-vm01.compute-500019369.gb1.internal
Address: 100.66.15.178

[opc@df-vm01 ~]$ cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search compute-500019369.gb1.internal. compute-500019369.gb1.internal.
nameserver 100.66.15.17


So we can see that the default behaviour of a VM created from one of the standard templates is to configure the VM to use the internal DNS server and the DNS Name of the VM is made available  through the DNS lookup and will return the internal or private IP address of the VM.

So what about lookups on other domains.  Provided the customer DNS allows lookups then the request will get forwarded on to the customer DNS which can provide the appropriate response.

[opc@df-vm01 ~]$ nslookup oracle.com
Server:        100.66.15.177
Address:    100.66.15.177#53

Non-authoritative answer:
Name:    oracle.com
Address: 137.254.120.50



The next obvious question is regarding how a client sitting in the customer's datacentre would find the IP address of the VM that was created.  There are no DNS records created for the "public" IP address which has been allocated (NAT) to the VM.  The answer to this is that it is the same as if you were installing a new server in your environment.  One of the steps would be to specify an IP address for the new machine and then register into the customer's DNS.   Once you have created the VM and know which IP address is being used for it take that IP and add it to your DNS with an appriate fully qualified domain name.  eg. donald.mycompany.com.   Once registered any client would be able to lookup donald.mycompany.com and start using the services running on the VM instance.

With Oracle Cloud at Customer it is possible to use an IP Reservation which will reserve an IP address from the client IP pool/shared network and once reserved it is fully under the control of the cloud administrator.  Thus the IP address can be registered in DNS before the VM is created if that suits the processes of your company better than registering the address after VM creation.

Tuesday, May 1, 2018

OCC/OCI-C Orchestration V2 Server Anti-Affinity

Introduction

The Oracle Cloud at Customer offering provides a powerful deployment platform for many different application types.  In the enterprise computing arena it is necessary to deploy clustered applications such that the app can be highly available by providing service even if there is a fault in the platform.  To this end we have to ensure that an application can be distributed over multiple virtual machines and these virtual machines have to reside on different physical hardware.  The Oracle Cloud Infrastructure - Classic (which is what runs on the Oracle Cloud at Customer racks) can achieve this by having it specified in what is called an orchestration.

Orchestration Overview

In OCI-C it is possible to create complex environments using multiple networks and apply firewall rules to the VMs that connect to these networks.  The VMs themselves have many different configuration items to specify the behaviour.  All of this is described in the documentation but the diagram below shows the interactions between various objects and a virtual machine instance.



Relationships between objects in Oracle Compute Cloud Service
OCI-C Object interactions (Shared Network only)

Using orchestrations (v2) you can define each of these objects in a single flat file (json formatted).  This file allows you to specifiy the attributes of each object and also reference each other.  By using a reference the system is then able to work out what order the objects should be created.  i.e. The dependencies.  Say a VM will use a storage volume it can use a reference to the volume meaning that the storage volume must come on-line prior to the instance being created.

For example the following snippet of json file defines a boot disk and then the VM instance definition references the name of the volume, identified by the label name.

 
 {
      "label": "df-simplevm01-bootdisk",
      "type": "StorageVolume",
      "persistent":true,
      "template":
        {
          "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm01-bootvol",
          "size": "18G",
          "bootable": true,
          "properties": ["/oracle/public/storage/default"],
          "description": "Boot volume for the simple test VM",
          "imagelist": "/oracle/public/OL_7.2_UEKR4_x86_64"
        }
...
 {
      "label": "df-simplevm01",
      "type": "Instance",
      "persistent":false,
      "name": "/Compute-500019369/don.forbes@oracle.com/df-simple_vms/df-simplevm01",
      "template": {
        "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm01",
        "shape": "oc3",
        "label": "simplevm01",
        "storage_attachments": [
          {
            "index": 1,
            "volume": "{{df-simplevm01-bootdisk:name}}"
          }
        ], 
...


There are a few other things in this that are worth pointing out.  Firstly the naming convention for all objects is essentially /<OCI-Classic account identifier>/<User Name>/<Object name>.   Thus when defining the JSON file content the first part of the name will vary according to what data centre/Cloud at Customer environment you are deploying into and the rest is determined by the users setup and the naming convention you want to utilse.  When deploying between cloud regions/Cloud at Customer these values may change.

The other thing to notice is that I have specifically made the boot disk have a persistence property of true while the VM has a persistence property of false.  The reason for this becomes clear when we consider the lifecycle of objects managed by a V2 orchestration.


Depicts the states of an orchestration
Orchestration V2 lifecycle



An orchestration can be either suspended or terminated once in the active state.  Suspension means that any non-persistent object is deleted while persistent objects remain on-line.  Termination will delete all objects defined in the orchestration.  By making the storage volume persistent we can suspend the orchestration which will stop the VM.  With the VM stopped we can update the VM to change many of its characteristics and then activate the orchestration again which will re-create the VM but using the persistent storage volume meaning the VM retains any data it has written to disk and acts as if it was simply re-booted but now happens to have more cores/memory etc.

Below is screen shot taken from OCI-Classic (18.2.2) showing some of the VM attributes that can be changed.  Essentially all configuration items of the VM can be adjusted.

Basic VM attributes that can be changed in an Orchestration V2

These attributes cannot be changed on a running VM -  it mandates a shutdown.  If the VM had been marked to be persistent then suspending the orchestration would have left the VM on-line and thus with immutable configuration so any change would mandate a termination of the orchestration which would delete things like the storage volumes.  Potentially not the desired behaviour.


Using Server Anti-Affinity

Having considered the usage of orchestrations we can now consider setting up some degree of control over the placement of a VM.    This is covered in the documentation but here we are considering two options, firstly the instruction to place VMs on "different nodes" and secondly to place them on the "same node".  Obviously the primary purpose of this blog posting is to consider the different node approach.

different_node Relationships

One of the general attributes of an object in an orchestration is its relationship to other objects.  The documentation for V2 specifies that the only relationship is the "depends" relationship and is included into an orchestration using the following format:

"relationships": [
  {
    "type": "depends",
    "targets": ["instance1"]
  }
]
 
Other relationships are possible, namely different_node and same_node. For a different_node approach we simply have the type set to the text of "different_node" and then in the targets array specify the instances that must be on different nodes.  Doing this will also setup a depends relationship as this instance placement will depend on the other instance placement.  So for example, with 4 VMs the objects in the orchestration will have the following relationships to ensure they are placed on different physical nodes.

 ...
  {
      "label": "df-simplevm01",
      "type": "Instance",
      "persistent":false,
      "name": "/Compute-500019369/don.forbes@oracle.com/df-simple_vms/df-simplevm01",
      "template": {
        "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm01",
        "shape": "oc3",
        "label": "simplevm01",
 "relationships":[
  {
  "type":"different_node",
  "instances":[ "instance:{{df-simplevm02:name}}",
         "instance:{{df-simplevm03:name}}",
         "instance:{{df-simplevm04:name}}"
       ]
   
  }
 ...
   {
      "label": "df-simplevm02",
      "name": "/Compute-500019369/don.forbes@oracle.com/df-simple_vms/df-simplevm02",
      "template": {
        "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm02",
        "shape": "oc3",
        "label": "simplevm02",
 "relationships":[
  {
  "type":"different_node",
  "instances":[ "instance:{{df-simplevm03:name}}",
         "instance:{{df-simplevm04:name}}"
       ]
   
  }
],
...
 {
      "label": "df-simplevm03",
      "type": "Instance",
      "persistent":false,
      "name": "/Compute-500019369/don.forbes@oracle.com/df-simple_vms/df-simplevm03",
      "template": {
        "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm03",
        "shape": "oc3",
        "label": "simplevm03",
 "relationships":[
  {
  "type":"different_node",
  "instances":[ "instance:{{df-simplevm04:name}}"
       ]
   
  }
],
...
   {
      "label": "df-simplevm04",
      "type": "Instance",
      "persistent":false,
      "name": "/Compute-500019369/don.forbes@oracle.com/df-simple_vms/df-simplevm04",
      "template": {
        "name": "/Compute-500019369/don.forbes@oracle.com/df-simplevm04",
        "shape": "oc3",
        "label": "simplevm04",
...
 
So in this example the 4th VM has no relationships but all the others have a dependence on it and where it is placed.  As such it is the first located VM, then the 3rd VM gets placed on a different node then the second and finally the first.  All the relationships have used the format of an object reference using the label of the object:name to identify the specific instance.

In the public cloud there is notionally an infinite compute resource so a great many VMs can get placed onto different nodes.  In the cloud at customer model there are only so many "model 40" compute units that are subscribed to which puts a physical limit on the number of VMs that can be placed on different nodes.  In the example above there are 4 VMs and in a typlical OCC starter rack there are only 3 nodes so the obvious question is what is the behaviour in this scenario.  The answer is that the orchestration will enter a "transient_error" state as the fourth VM cannot be started on the rack and the orchestration will try to start up the VM on a regular basis.  The error is reported as:-


"cause": "error_state", 
"detail": "System object is in error state: Cannot satisfy both the placement and relationship requirements." 


So in a Cloud at Customer environment you should be aware of how many physical machines are in place and if a cluster size is larger than this then split the dependencies up accordingly.  e.g. With 6 VMs and only 3 nodes make 3 VMs have a different_node relationship to each other and the other three another different_node relationship.

same_node Relationship

Same node relationships are configured in exactly the same way as the different_node relationship is, the only difference being that in this situation the VMs are placed on the same physical node.&nbsp; Obviously using this approach you need to be aware of the physical limits of a single node.&nbsp; Currently the OCC is using an X6-2 server which has 40 cores and 496Gb of memory available for use.&nbsp; Clearly trying to put more VMs than will fit in this space will result in a similar failure to place the VM that occurred when trying to place on a different node.