Cloud at Customer - DNS

Introduction

In recently weeks I have been involved with a few engagements where there has been a confusion with the understanding of how DNS within a Cloud at Customer environment works.  This blog posting will be an attempt to clarify how it works at a reasonably high level.

DNS Overview

The Domain Name System is provided to enable a mapping from a fairly human readable form to an actual IP address which computers can use to route traffic.  There are essentially two components to the name, firstly the hostname of a machine and secondly a domain name.  eg. donald.mycompany.com where donald represents the hostname and mycompany.com represents what is called the domain.

On a linux system the resolution of DNS is typically specified in a file /etc/resolv.conf it will look something like this.

# Generated by NetworkManager
nameserver 194.168.4.100
nameserver 194.168.8.100

The nameserver entry specifies the IP address of a DNS server which will be used to change names into IP addresses.  We have two entries so that should the first one be unavailable for any reason then the second one will be used.  We can then use a unix utility called nslookup (part of the bind-utils package if you want to install it) which will allow us to query the DNS server.  Other utilities such as dig can also provide information from DNS.

$ nslookup oracle.com
Server:        194.168.4.100
Address:    194.168.4.100#53

Non-authoritative answer:
Name:    oracle.com
Address: 137.254.120.50


Note that this response includes the text "Non-authorative answer".   The reason for this is that all DNS servers will be in control of different domains and in our example we tried to lookup a domain called oracle.com.   From my laptop the DNS server is from my ISP so it knows nothing about the oracle.com domain.  In this situation the DNS server I talked to will forward the request on to a .com domain server which in turn is likely to delegate the request to the DNS server responsible (or authorative) for the oracle.com domain.  This returns the IP address which flows back through the DNS servers to my ISP which caches the answer and gives me the IP address to use for a lookup on oracle.com.

DNS on Oracle Cloud@Customer

Inside the control plane of an OCC environment there is a DNS server that runs.  This server is authorative over two domains of interest to a customer.  Specifically one in the format of <Subscription ID>.oraclecloudatcustomer.com and also one  of the format <Account>/<Region>.internal.  For example. S12345678.oraclecloudatcustomer.com and compute-500011223.gb1.internal.

cloudatcustomer.com domain

cloudatcustomer.com is a domain that Oracle own and run on behalf of the customer and is used to provide the URLs that the customer will interact with.  Utilise the UI for managing the rack or accessing the REST APIs for compute, PaaS, Storage etc.  So, for example the underlying compute REST API of a C@C environment will be something like https://compute.<Region>.ocm.<Subscription ID>.oraclecloudatcustomer.com.  So in our example above this would become https://compute.gb1.ocm.S12345678.oraclecloudatcustomer.com.

Clearly the domain <Subscription>.oraclecloudatcustomer.com is something that is not going to be known in any customer's DNS so normal DNS actions would take place and the customer's DNS would look up the domain chain to find a DNS server that is authorative for oraclecloudatcustomer.com.  As these are not publicly accessible this lookup would fail.  To fix this we need to change the customer's DNS to make it aware of the <Subscription ID>.oraclecloudatcustomer.com domain so it knows how to access a DNS service that will return an appropriate IP address and enable usage of the URL.  This is done by adding a forwarding rule to your customer's DNS service which forwards any request for a C@C URL to the DNS running in the C@C environment.

Once the forwarding rule is in place then the customer is able to start using the UI and REST APIs of the C@C so they can manage the platform.

internal domain

So now the customer can access the cloud at customer management screens and can create virtual machines to run their enterprise.  The next question is about what is the internal domain.  This essentially relates to configuration of the VM itself.  During the provisioning process there is a screen which requests the "DNS Hostname Prefix" as shown below. 

The name used in the DNS Hostname Prefix becomes part of the registration into the OCC DNS server to represent the host portion.  ie. When we create the VM it will have a DNS registration internally like df-web01.compute-500019369.gb1.internal which maps onto the private IP address allocated to the VM.  This means if you are creating multiple VMs together or using a PaaS service that produces multipleVMs (like JCS) then there are URLs available to enable them to talk to each other without you having to know the IP address.

Once a VM has been created it is possible to see the internal DNS name for the VM from the details page as shown below.

This can be resolved from within the VM as shown below.

[opc@df-vm01 ~]$ nslookup df-vm01
Server:        100.66.15.177
Address:    100.66.15.177#53

Name:    df-vm01.compute-500019369.gb1.internal
Address: 100.66.15.178

[opc@df-vm01 ~]$ cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search compute-500019369.gb1.internal. compute-500019369.gb1.internal.
nameserver 100.66.15.17

So we can see that the default behaviour of a VM created from one of the standard templates is to configure the VM to use the internal DNS server and the DNS Name of the VM is made available  through the DNS lookup and will return the internal or private IP address of the VM.

So what about lookups on other domains.  Provided the customer DNS allows lookups then the request will get forwarded on to the customer DNS which can provide the appropriate response.

[opc@df-vm01 ~]$ nslookup oracle.com
Server:        100.66.15.177
Address:    100.66.15.177#53

Non-authoritative answer:
Name:    oracle.com
Address: 137.254.120.50


The next obvious question is regarding how a client sitting in the customer's datacentre would find the IP address of the VM that was created.  There are no DNS records created for the "public" IP address which has been allocated (NAT) to the VM.  The answer to this is that it is the same as if you were installing a new server in your environment.  One of the steps would be to specify an IP address for the new machine and then register into the customer's DNS.   Once you have created the VM and know which IP address is being used for it take that IP and add it to your DNS with an appriate fully qualified domain name.  eg. donald.mycompany.com.   Once registered any client would be able to lookup donald.mycompany.com and start using the services running on the VM instance.

With Oracle Cloud at Customer it is possible to use an IP Reservation which will reserve an IP address from the client IP pool/shared network and once reserved it is fully under the control of the cloud administrator.  Thus the IP address can be registered in DNS before the VM is created if that suits the processes of your company better than registering the address after VM creation.

A quick look at using Quota on the Oracle Cloud Machine

Introduction

On OCM racks prior to version 17.1.6 they have the option to place a quota limitation on their tenancies.  Without a quota the tenant is able to use compute resources up to the physical limits of the rack that has been subscribed to.  It may be appropriate to limit the resource that one tenancy can utilise, this can be achieved using the quota object which is assigned to each tenancy.

There are a number of resources that can be limited on a per-tenant basis, these are.

• Compute
• Instances
• Memory
• CPUs
• Instance Restrictions
• Storage
• Machine Image
• Number
• Megabytes
• Network
• IP reservations

Figure 1 shows an example listing of a quota that has been applied to a tenancy.   It shows both the quota limits set and what the current usage is so it becomes a fairly simple task to work out how close to the current quota the tenancy is getting.

# oc list quota -u /quota-test/administrator /quota-test -f json {  "list": [   {    "usage": {     "machineimage": {      "number": 1,      "megabytes": 2466     },     "vnet": {      "vnetreservations": 0     },     "compute": {      "instances": 7,      "ram": 24576,      "instance_restrictions": {},      "cpus": 6.0     },     "nds": {      "megabytes": 0     }    },    "uri": "https://api/quota/quota-test",    "description": "Default quota for \"quota-test\"",    "name": "/quota-test",    "allowance": {     "machineimage": {      "number": 2,      "megabytes": 4096     },     "vnet": {      "vnetreservations": 2     },     "compute": {      "instances": 8,      "ram": 41000,      "instance_restrictions": {},      "cpus": 6.0     },     "nds": {      "megabytes": 4096     }    }   }  ] }

Figure 1 - Example quota output

In general these are fairly self explanatory as to what each restriction will do but there are some subtlties.

Compute

The restrictions for compute are split into Instances, Memory, CPU and "Instance Restrictions."   The instances number in a quota directly relates to the number of  VM instances that have been created within the tenancy.  This quota is checked as soon as an orchestration attempts to start and if the quota is exceeded it will cause the orchestration to fail to startup until other instances (orchestrations) are stopped.  Note - Stopped rather than simply shutdown as a shutdown instance remains in existence.

Memory and CPU quota are very similar to instance restrictions.  The main difference is that a VM has to be actually running to consume the quota so if this limit is reached then existing VMs have to be shutdown to allow the new instance to startup but they do not have to be "stopped".

Figure 2 below shows the errors reported when the quota is exceeded in a couple of examples.

Error starting orchestration that exceeds the allowed number of instances      "status": "error",      "info": {       "errors": {        "0": "don05 is in error: {'/quota-test': u'Reservation exceeded quotas: instances exceeded by 1.0'}"       } Error starting orchestration that exceeds the allowed Memory      "status": "error",      "info": {       "errors": {        "0": "don06 is in error: {'/quota-test': u'Reservation exceeded quotas: ram exceeded by 3576.0'}" }

Figure 2 - Error message when exceeding quota for instance or memory

Instance Restrictions is a special tagging mechanism that would allow a machine image (template) to be tagged and it is then possible to restrict the number of instances created from that specific machine image.  This is currently not supported on the OCM so the instance_restrictions quota value should be left blank.

Storage

The storage figure given applies to additional storage volumes and not to the root disks.  i.e. If you need to restrict storage space used by the root disks then you need to apply a quota to the instances in the compute section.  If you hit the limit then it is necessary to actively stop any storage volumes to delete the storage and free up resource.

Machine Image

The machine image quota can be specified either in terms of the numbers of machine images held within the quota or by the storage volume space taken up.  Typically this would be used to limit the number of snapshots that a tenancy can use.  (Taking a snapshot of a VM will take a copy of the root disks at the time of snapshotting and create a new machine image.)

Network

When networks are defined for a tenancy there are several choices on their configuration which can be used as a limit in themselves.  For example if a tenant has a public or EoIB network assigned to them then the number of IP addresses the tenancy can use will be determined by the subnet and within the subnet further limited by the IP Range defined.  Private networks can also be created but again these have a defined size which cannot be exceeded.

However, in OCM releases prior to 17.1.6 the concept of a service network was available where a tenancy would be given access to a network and the same network could be handed out to other tenancies as well.  If a "vnetreservation" quota is setup then the system will prevent a tenancy using more than the quota value of IP addresses from these service networks.

An example failure to create a VM is shown in figure 3 when the tenancy has run out of IP addresses on a network.

"info": {       "errors": {        "0": "{u'/cloud/public/EoIB-OMS': u'1 IP(s) requested, but only 0 IP(s) available in the vnet.'}"       },

Figure 3 - No IP addresses available

Conclusion

If a customer has divided their rack into multiple tenancies to run different environments then the quota system is a powerful and useful mechanism to ensure that no users of the system hog the available compute resource.

What happens to attached volumes when snapshotting VMs on an OCM

Introduction

With OCM there is the capability to take a "snapshot" of a virtual machine.  As described in the documentation a snapshot will take a copy of the machine image boot disk.  Essentially there are two purposes for this action to take place, firstly if we take a machineimage then we can use that machineimage to create additional VMs from this copy - using it as a template.  Secondly it can be used as a backup copy to recreate the VM if needed.  (Really just the same as the first but using the copy to recreate rather than clone.)

This leads me to think of a couple of questions that this blog posting will be answering.

1. What happens to storage volumes that have been added to the VM.  Are these copied as well?
2. Is this a good mechanism to increase the root volume size to make more space for VMs that might want a bit more disk space?

I have tested two specific scenarios out starting from a VM based on the OL6 template with an attached volume.

1. use the new volume to add to the root disk logical volume
2. create a new logical volume and attach it to the filesystem, say from /u01.

Snapshot both cases and create a new VM from the machineimage created and look to see what happened.

 Extending root volume

In the first case I simply create a VM from the OL6 base template using a simple orchestration.  I create an additional volume then attach the new volume to the VM.  Having created the VM I log onto it and use the unix commands for LVM to extend the size of the root disk.

The steps taken are:-

1. Use fdisk to format the attached volume to LVM.
2. Use lvdisplay and vgdisplay to identify the current root volume (Prob VolGroup00)
3. Use vgextend to extend the current volume group to add the storage from the attached volume.
4. Use lvextend to make the root logical volume larger
5. Use resize2fs the device mapper to make the extra space available to the filesystem.

Some of the key commands and output are shown below.  The result of all these commands is that the root filesystem has grown from 11G to 61G using all the 50Gb in the attached volume.

#df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   11G  3.3G  6.9G  33% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance # vgextend VolGroup00 /dev/xvdb1   Volume group "VolGroup00" successfully extended # pvdisplay   --- Physical volume ---   PV Name               /dev/xvda2   VG Name               VolGroup00   PV Size               17.75 GiB / not usable 2.12 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              4543   Free PE               191   Allocated PE          4352   PV UUID               VnZ5PZ-8IrP-nggg-B7Fo-9sog-g4bw-skbPBE      --- Physical volume ---   PV Name               /dev/xvdb1   VG Name               VolGroup00   PV Size               50.00 GiB / not usable 3.31 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              12799   Free PE               12799   Allocated PE          0   PV UUID               c1m21x-2IBX-Qrsy-1AJI-3UQ7-apUe-weETeq # lvextend -L+55G /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 66.00 GiB   Insufficient free space: 14080 extents needed, but only 12990 available # lvextend -l+12990 /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 61.74 GiB   Logical volume LogVol00 successfully resized   # resize2fs /dev/mapper/VolGroup00-LogVol00 resize2fs 1.43-WIP (20-Jun-2013) Filesystem at /dev/mapper/VolGroup00-LogVol00 is mounted on /; on-line resizing required old_desc_blocks = 4, new_desc_blocks = 4 The filesystem on /dev/mapper/VolGroup00-LogVol00 is now 16185344 blocks long. # df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   61G  3.3G   55G   6% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance

I now use the UI from EMCC to create a snapshot of the VM.  This could also be done from the command line.

Snapshotting a VM

The snapshot takes a few minutes to complete and once done there is a template available that will allow the creation of new VMs.

Snapshot appearing as a template in the OCM library

Creating a VM based on the snapshot

Once the new VM is created just log on and have a look at the root disk size.  It is immediately clear that it has a root disk of 61Gb and no attached volumes so by expanding the root LVM partition and snapshotting it will effectively increase the size of the disk in the machine image.

ERRATA - This approach does not work for increasing the root disk size.  Further investigation shows that while the OS reports the increased disk size all looks good but issuing a pvdisplay command reports that a device is missing.  This was confirmed by simply filling the disk up, as soon as the used space reached the same level as the original disk then warnings about the possible loss of data were reported and some of the new writes failed to get persisted to disk.  The conclusion - The actual disk space available was not expanded.

EXT4-fs error (device dm-2): ext4_wait_block_bitmap:448: comm flush-252:2: Cannot read block bitmap - block_group = 117, block_bitmap = 3670021
EXT4-fs (dm-2): delayed block allocation failed for inode 13345 at logical offset 16665 with max blocks 1 with error -5
EXT4-fs (dm-2): This should not happen!! Data will be lost

Adding the volume as a new partition/logical volume

The same process was completed for a volume being attached but this time rather than extendingVolGroup00 I created a new physical volume, volume group and logical volume which I mounted on /u01.  (fdisk to format volume to LVM [8e type], pvcreate to create the physical volume, vgcreate to create  a volume group using the new volume, lvcreate to create the logical volume and then mount the logical volume of /u01.)  Exactly the same process was used to create a snapshot and then create a new VM from the machineimage that was created.  This time round the resulting disk space on the new VM was just 11Gb - the disk size of the original template.  i.e. The snapshot has ignored the additional volume and done what the docs say, specifically take an image of the machine's boot disk.

Conclusion

As a mechanism to increase the root disk space of a template the approach of creating a VM with an attached volume and using this volume to extend the size of the root volume group/logical volume will NOT enable a new template with larger disk space.

If you are planning to create a VM with an attached volume and think that the snapshot will backup the entire VM then think again.  It will be necessary for you to also snapshot any volumes that are not part of the machine image boot disk to create a full snapshot of your virtual machine.  

Encrypting "disks" on Oracle Cloud Machine

Introduction

The Oracle Cloud Machine, like the public cloud, is administered by Oracle.  While the Oracle staff who manage the rack are highly skilled professionals and all their actions audited there is an obvious concern about the security of customer data at rest.  On the OCM the administrators of the rack have no direct access to the customer's virtual machines.  This article demonstrates how storage volumes can be used by a tenant to mount block storage devices that are encrypted and hence further obscured from system administrators.

(As a side effect of demonstrating the security aspect this is also a useful reference for using cryptsetup to encrypt disks.)

Setup

In order to demonstrate that a storage volume is encrypted and hence not visible to cloud administrators we do a very simple setup where two storage volumes are created, one to be encrypted and the other left in plain text.  These volumes are "attached" to a virtual machine and then within the virtual machine we use the linux utility cryptsetup to encrypt one of the volumes the other is simply mounted with an ext4 filesystem on it.  Plain text files are created in both volumes and then we will switch to the cloud administration side of things to see if it is possible to read the content of the two volumes.

Virtual Machine Instance Creation

First of all we create two storage volumes.  This can be done from the command line easily.


# oracle-compute add storagevolume /osc/public/encrypt-storage-001 10G /oracle/public/storage/default --description "A test 10Gb storage volume that we will try to have encrypted" 

# oracle-compute add storagevolume /osc/public/plain-storage-001 10G /oracle/public/storage/default --description "A test 10Gb storage volume that we will try to have encrypted"


Then we create a virtual machine via an orchestration defined in a json file

# cat simple_vm_with_storage.json
{
"name": "/osc/public/encryption-vm",
"oplans": [
{
 "obj_type": "launchplan",
 "ha_policy": "active",
 "label": "encryption volume launch plan",
 "objects": [
 {
 "instances": [
 {
 "label": "encryption-vm001",
 "imagelist": "/oracle/public/linux6_16.1.2_64",
 "networking":
 {
   "net0": { "vnet": "/osc/public/vnet-eoib-1706" }
 },
 "storage_attachments": [
 { "volume": "/osc/public/encrypt-storage-001", "index": 1},{"volume": "/osc/public/plain-storage-001", "index": 2}],
 "shape": "ot1",
 "sshkeys": ["/osc/public/labkey"],
 "attributes":
 {
 "userdata":
 {
 "key1": "value 1",
 "key2": "value 2"
 }
 }
 } ]
 } ]
} ]
}


This json file will create a single instance called encryption-vm001 based on the OL6 base template, connect it to the EoIB public network and attach the two storage volumes that we created earlier.  (Storage volumes created independently of this orchestration in this case.)

We upload the orchestration and start it.  Once up and running then the instance will be listed as running and we can see the IP address assigned to it.

# oracle-compute add orchestration ./simple_vm_with_storage.json 

(see above for json)

# oracle-compute start orchestration /osc/public/encrytption-vm

# oracle-compute list instance /osc -Fname,state,ip

Configuring volumes within instance

Having created and started up our instance we can look at the attached volumes and run through the process using Oracle Linux to setup one of the volumes as an encrypted one.   To see the volumes on the instance we use the fdisk command.



# fdisk -l

Disk /dev/xvda: 19.3 GB, 19327352832 bytes
255 heads, 63 sectors/track, 2349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c520c


    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          32      256000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/xvda2              32        2349    18611318+  8e  Linux LVM



Disk /dev/xvdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000



Disk /dev/xvdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol01: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol00: 11.8 GB, 11811160064 bytes
255 heads, 63 sectors/track, 1435 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol02: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

With the OCM each volume that is attached gets an index, in the orchestration above we use indexes 1 and 2.  These numbers equate to the xvd<char> devices that appear in the fdisk output.where 1 equates to b, 2 equates to c etc.  Thus in the output above the two attached volumes are /dev/xvdb and /dev/xvdc.  The next step is to setup one of the volumes as an block device encrypted one.  To do this I used the linux command cryptsetup defining cipher information etc.  In the example shown below I show it run twice as the first time I answered the question with a lower case yes.  The command mandated uppercase YES as an answer.  Easy mistake to make!



# cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/xvdb



WARNING!
========
This will overwrite data on /dev/xvdb irrevocably.


Are you sure? (Type uppercase yes): yes
Command failed with code 22: Invalid argument

# cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/xvdb

WARNING!
========

This will overwrite data on /dev/xvdb irrevocably.
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.



Now we can open the encrypted drive such that it appears as normal.  This will create the /dev/mapper/<name> device file and allow it to be mounted by the OS.  The luksOpen command will prompt for the passphrase used earlier.

# cryptsetup luksOpen /dev/xvdb encrypted-drive

# cryptsetup -v status encrypted-drive
/dev/mapper/encrypted-drive is active.
  type:  LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/xvdb
  offset:  4096 sectors
  size:    20967424 sectors
  mode:    read/write
Command successful.

This is a new raw volume so we need to put some sort of filesystem onto it.  In this case I use the ext4 filesystem.


# mkfs.ext4 /dev/mapper/encrypted-drive
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2620928 blocks
131046 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done                           
Writing inode tables: done                           
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now simply create a directory where we can mount the encrypted drive and create a simple text file.

# mkdir /u01
# mount /dev/mapper/encrypted-drive /u01
# df -kh
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   11G  3.3G  7.0G  32% /
tmpfs                            3.8G     0  3.8G   0% /dev/shm
/dev/xvda1                       239M   55M  168M  25% /boot
/dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance
/dev/mapper/encrypted-drive      9.8G   23M  9.2G   1% /u01

 Having done this we can do a quick check to ensure that we can unmount and close the encrypted disk and re-open it providing the passphrase and mount it for use.


#umount /u01
# cryptsetup luksClose encrypted-drive
# mount /dev/mapper/encrypted-drive /u01
mount: you must specify the filesystem type



# cryptsetup luksOpen /dev/xvdb encrypted-drive
Enter passphrase for /dev/xvdb:
# mount /dev/mapper/encrypted-drive /u01
# df -kh
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   11G  3.3G  7.0G  32% /
tmpfs                            3.8G     0  3.8G   0% /dev/shm
/dev/xvda1                       239M   55M  168M  25% /boot
/dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance
/dev/mapper/encrypted-drive      9.8G   23M  9.2G   1% /u01



Using fdisk we can format the /dev/xvdc volume, create a file system on this volume and mount it into another directory.  Then create a plain text file in this volume as well.   If the encryption has all worked then cloud operations may be able to access the plain text volume and read the content but the encrypted volume content is kept secret unless the passphrase is known.

Testing

As a general rule cloud operations do not have access to the customer's virtual machines unless the customer shares the login credentials or the ssh keys with Oracle.  However, because the OCM stores the volumes as raw disk images on the internal ZFS storage appliance in the EPC_<rack>/storagepool1 filesystem it is possible for cloud operations to access these files and mount the images directly to access the content.


As a cloud operations user I have accessed the ZFS storage device and can copy the storage volume disks off the rack.  In a linux server I attempt to mount these volumes to see the content.

# file plain_storage.raw
plain_storage.raw: Linux rev 1.0 ext4 filesystem data (extents) (large files) (huge files)
# mount -o loop ./plain_storage.raw /mnt/don
# cat /mnt/don/don-plain

This text is in the unencrypted volume and hence should be readable by anyone.....
# unmount /mnt/don



So it is obviously fairly easy to access the unencrypted storage.  Now lets see what is involved in accessing the encrypted storage volume.

# file encrypted_storage.raw
encrypted_storage.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha512] UUID: edff3d80-3813-4abc-a58c-e2f1862

# mount -o loop ./encrypted_storage.raw /mnt/don
mount: unknown filesystem type 'crypto_LUKS'

# losetup /dev/loop0 ./encrypted_storage.raw
# mount /dev/loop0 /mnt/don
mount: unknown filesystem type 'crypto_LUKS'

# cryptsetup luksOpen /dev/loop0 encrypted-dev
Enter passphrase for /dev/loop0:

# mount /dev/mapper/encrypted-dev /mnt/don

# cat /mnt/don/don

some text

#

In the above I attempt to mount the encrypted filesystem using the same mechanism as previously was successful but to no effect.  The only way to mount the disk is to make use of the cryptsetup command which mandated entering the passphrase.  Obviously the passphrase is not something that is shared with cloud operations so they would be unable to access the content of the raw file.

Conclusion

Certainly using the standard linux command of cryptsetup it is a relatively simple task to encrypt any storage volume that is mounted on a VM such that the data is kept private to the end customer/tenant and cloud operations has no mechanism of seeing the content.

The down side of encrypting is that it means that the administrator of the virtual machine (end customer) has to log on and provide the passphrase to mount the volume.  Not a major problem unless you are looking at trying to automatically start up the applications deployed that use the encrypted volume, in this case it becomes necessary to have a manual startup procedure.