Expanding the root volume in an OCM template

Introduction

The OCM is delivered to you with two templates that are specifically ready for use by the Infrastructure as a Service interfaces.  These templates are OL5 and OL6 and by default the root volume has approximately 11Gb of space available.  This should be ample as a root volume for 90% of customer uses.  However from time to time it might be useful to increase this disk space in a template.  For example a customer might want to have a number of applications deployed into the base template, things like specific package installs, monitoring tools etc.  These could use up much of the disk space making management of logs etc. more space critical than would be liked.

This blog posting goes through a process that will enable the root disk structure to be increased in size.  Specifically for the OL6 template.  The process revolves around three actions.

1. Download the base template from your OCM rack.
2. Make use of "EKIT" to increase the disk space
3. Add the new template back onto the OCM and make it available for use.

Step 1 - Download the template

The first activity is to download the template.  This can be done easily from the Command line - oracle-compute.

# oracle-compute download machineimage /oracle/public/linux6_16.3.1_64

This will create a directory structure where the file is downloaded as a gziped tar file.

Step 2 - Use ModifyJeosLVM.sh to increase disk space

Handily there is a toolkit known as the "Exalogic Kinetic Infrastructure Tools" or EKIT for short which contains a script called ModifyJeosLVM.sh which has been designed to extend the root volume (or swap space) of an OL LVM based image.  EKIT is available from My Oracle Support and can be easily found using MOS document ID of 1933252.1.

Install the EKIT into an appropriate environment according to the toolkit installation instructions.  (Can be on a laptop, VM on a laptop or a VM on OCM itself although you will need to ensure that the VMs have enough disk space to handle files as large as you intend to make the root volume.)  Once installed copy the downloaded <template>.tar.gz file to your EKIT environment.   Extract the machineimage (will be a System.img file)  And run ModifyJEOSLVM.sh as shown below.  This adds space to the current image file and then tar/gzip the image file and you have a machineimage ready to upload into your OCM rack.

[root@utility ocm-ol6]# gunzip linux6_16.3.1_64.tar.gz [root@utility ocm-ol6]# ls linux6_16.3.1_64.tar [root@utility ocm-ol6]# tar xvf linux6_16.3.1_64.tar System.img [root@utility ocm-ol6]# ls -lh total 37G -rw-r--r--+ 1 root root 19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 18G Sep 26 05:31 System.img  [root@utility ocm-ol6]# ModifyJeosLVM.sh --resize -if System.img -er 20 Img File Array: System.img *********************************************** *** Modifying System.img *********************************************** Block size: 1024 Base Image Size 19327352832 Base Image 18874368 blocks Adding 20971520 blocks to root file system Adding 0 blocks to swap file system Adding 0 blocks free space to Volume Group Adding 524288 spare blocks Resizing image file to 40370176 *********************************************** *** Checking Primary Partition Status System.img *********************************************** *********************************************** *** Using loop /dev/loop0 *********************************************** Disk /dev/loop0: 19.3 GB, 19327352832 bytes 255 heads, 63 sectors/track, 2349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM 1. Found /dev/loop0p1 2. Found /dev/loop0p2 *********************************************** *** Extending System.img *********************************************** 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000268591 s, 0.0 kB/s *********************************************** *** Using loop /dev/loop0 *********************************************** Disk /dev/loop0: 41.3 GB, 41339060224 bytes 255 heads, 63 sectors/track, 5025 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM 1. Found /dev/loop0p1 2. Found /dev/loop0p2 Creating new Primary partition 3 *********************************************** (echo n; echo p; echo 3; echo ; echo ; echo t; echo 3; echo 8e; echo w) | fdisk /dev/loop0 *********************************************** WARNING: DOS-compatible mode is deprecated. It's strongly recommended to          switch off the mode (command 'c') and change display units to          sectors (command 'u'). Command (m for help): Command action    e   extended    p   primary partition (1-4) Partition number (1-4): First cylinder (2350-5025, default 2350): Using default value 2350 Last cylinder, +cylinders or +size{K,M,G} (2350-5025, default 5025): Using default value 5025 Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 3 to 8e (Linux LVM) Command (m for help): The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 22: Invalid argument. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. Disk /dev/loop0: 41.3 GB, 41339060224 bytes 255 heads, 63 sectors/track, 5025 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM /dev/loop0p3            2350        5025    21494970   8e  Linux LVM loop: can't delete device /dev/loop0: Device or resource busy =============================================== loop0p1|:|0|512000|/dev/loop0|2048 loopId: loop0p1 loop0p2|:|0|37222637|/dev/loop0|514048 loopId: loop0p2 loop0p3|:|0|42989940|/dev/loop0|37736685 loopId: loop0p3 loop|deleted|:|/dev/loop0 [root@utility ocm-ol6]# ll -h total 37G -rw-r--r--+ 1 root root 19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 39G Jan 27 08:41 System.imgloopId: loop0p3 Using loopId: loop0p3 =============================================== ################################################################################ ####                         getRunningSystemInfo                           #### ################################################################################ ******************************************************************************** *** *** utility has Volume Group *** *** RHEL Version : 6.7 *** OL Version   : 6.7 *** *** File System  : lvm *** *** Volume Group : VGEKITUtility *** Root Log Vol : LVRoot *** Swap Log Vol : LVSwap *** ******************************************************************************** ################################################################################ ####                         createLoopFileSystem                           #### ################################################################################ ################################################################################ ####                          getSystemImgFSType                            #### ################################################################################ ******************************************************************************** *** *** Mounted Image File System *** *** File System  : ext3 *** ******************************************************************************** Loop device is /dev/loop0 add map loop0p1 (252:2): 0 512000 linear /dev/loop0 2048 add map loop0p2 (252:3): 0 37222637 linear /dev/loop0 514048 add map loop0p3 (252:4): 0 42989940 linear /dev/loop0 37736685 ################################################################################ ####                           getSystemImgInfo                             #### ################################################################################ ******************************************************************************** *** *** Mounted Image has no Volume Group *** *** RHEL Version : *** OL Version   : *** *** File System  : ext3 *** ******************************************************************************** ################################################################################ ####                      validateLoopFileSystemType                        #### ################################################################################ ******************************************************************************** *** *** Specified System Image file has ext3 file system so we should be okay. *** ******************************************************************************** *********************************************** *** Scanning Volume Groups ***********************************************   Reading all physical volumes.  This may take a while...   Found volume group "VolGroup00" using metadata type lvm2   Found volume group "VGEKITUtility" using metadata type lvm2 *********************************************** *** Extending VolGroup00 *********************************************** ===============================================   Physical volume "/dev/mapper/loop0p3" successfully created   Volume group "VolGroup00" successfully extended =============================================== *********************************************** ***** Extending Root LogVol00 ***********************************************   Size of logical volume VolGroup00/LogVol00 changed from 11.00 GiB (2816 extents) to 31.00 GiB (7936 extents).   Logical volume LogVol00 successfully resized *********************************************** *** Changing VolGroup00 ***********************************************   3 logical volume(s) in volume group "VolGroup00" now active *********************************************** *** Listing Directories *********************************************** *** ls -lh /dev/mapper *********************************************** total 0 crw-rw---- 1 root root 10, 236 Jan 16 05:43 control lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p1 -> ../dm-2 lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p2 -> ../dm-3 lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p3 -> ../dm-4 lrwxrwxrwx 1 root root       7 Jan 16 05:43 VGEKITUtility-LVRoot -> ../dm-1 lrwxrwxrwx 1 root root       7 Jan 16 05:43 VGEKITUtility-LVSwap -> ../dm-0 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol00 -> ../dm-6 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol01 -> ../dm-5 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol02 -> ../dm-7 *** ls -lh /dev/V* *********************************************** /dev/VGEKITUtility: total 0 lrwxrwxrwx 1 root root 7 Jan 16 05:43 LVRoot -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 16 05:43 LVSwap -> ../dm-0 /dev/VolGroup00: total 0 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol00 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol01 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol02 -> ../dm-7 Oracle Linux 6 (6.7) *********************************************** ***** Checking Root LogVol00 *********************************************** /dev/VolGroup00/LogVol00: 154624/720896 files (0.1% non-contiguous), 925814/2883584 blocks *********************************************** ***** Resizing Root LogVol00 *********************************************** resize2fs 1.43-WIP (20-Jun-2013) Resizing the filesystem on /dev/VolGroup00/LogVol00 to 8126464 (4k) blocks. The filesystem on /dev/VolGroup00/LogVol00 is now 8126464 blocks long. *********************************************** ***** Setting Swap LogVol01 *********************************************** mkswap: /dev/VolGroup00/LogVol01: warning: don't erase bootbits sectors         on whole disk. Use -f to force. Setting up swapspace version 1, size = 4194300 KiB no label, UUID=73a92d9c-4c00-4013-9681-a8e5ae2994b6   0 logical volume(s) in volume group "VolGroup00" now active *********************************************** ***** Changing UUID of VolGroup00 ***********************************************   Volume group "VolGroup00" successfully changed ################################################################################ ####                         deleteLoopFileSystem                           #### ################################################################################ *********************************************** *** Unmounting /dev/loop0 *********************************************** Processing System.img *********************************************** *** System.img Modified *** Start Size : -rw-r--r--+ 1 root root 18G Sep 26 05:31 System.img *** End Size   : -rw-r--r--+ 1 root root 39G Jan 27 08:41 System.img *********************************************** [root@utility ocm-ol6]# tar cvf linux6-40G_16.3.1_64.tar System.img System.img [root@utility ocm-ol6]# ls -lh total 39G -rw-r--r--+ 1 root root  19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 2.5G Jan 27 08:44 linux6-40G_16.3.1_64.tar.gz -rw-r--r--+ 1 root root  39G Jan 27 08:41 System.img

Step 3 - Adding the new machineimage to your OCM

The final step in the process is to copy the new <template>.tar.gz file back to where the OCM CLI has been installed and then add it to the environment.  To make a machine image available for use there are three steps.

1. Add the machineimage file 
2. Add an imagelist to the environment
3. Add a imagelistentry to link the machine image to the imagelist.

The OCM makes use of an imagelist  when specifying which template to use as the base OS for a VM.  By abstracting the connection between the VM and its machineimage we introduce a useful mechanism for version control of the VM so VM creation scripts do not change even if we want to change the underlying machineimage.

The steps to make it available for use are shown below:-

# oc add machineimage /oracle/public/linux6-40G-16.3.1_64 /root/nimbula-machineimages-2017-01-27/linux6-40G_16.3.1_64.tar.gz [====================================================================================================] # oc add imagelist /oracle/public/linux6-40G-16.3.1_64 "Image list for base template expanded to 40G in size" uri                                                          name                                    description                                              default     entries    https://api/imagelist/oracle/public/linux6-40G-16.3.1_64     /oracle/public/linux6-40G-16.3.1_64     Image list for base template expanded to 40G in size     1                       # # oc add imagelistentry /oracle/public/linux6-40G-16.3.1_64 /oracle/public/linux6-40G-16.3.1_64 1 uri                                        imagelist                                                                                               machineimages                  versi a https://api/imagelist/oracle/public/linux6 {"default": 1, "description": null, "entries": null, "uri": "imagelist/oracle/public/linux6-40G-16.3.1_ /oracle/public/linux6-40G-16.3 1     { #

Note - In the example above I have used the /oracle tenancy to host the template.  This is a tenancy that only oracle cloud operations have access to and by placing the imagelist in this tenancy it makes the template available to all tenancies on the rack.  If you want this to be the situation then you must raise a request for cloud operations to add the objects.  However you can also do this within another tenancy in which case the imagelist will only be available for use within that tenancy.

Below is a screenshot from the self service portal showing the imagelist as available for use as a template.

What happens to attached volumes when snapshotting VMs on an OCM

Introduction

With OCM there is the capability to take a "snapshot" of a virtual machine.  As described in the documentation a snapshot will take a copy of the machine image boot disk.  Essentially there are two purposes for this action to take place, firstly if we take a machineimage then we can use that machineimage to create additional VMs from this copy - using it as a template.  Secondly it can be used as a backup copy to recreate the VM if needed.  (Really just the same as the first but using the copy to recreate rather than clone.)

This leads me to think of a couple of questions that this blog posting will be answering.

1. What happens to storage volumes that have been added to the VM.  Are these copied as well?
2. Is this a good mechanism to increase the root volume size to make more space for VMs that might want a bit more disk space?

I have tested two specific scenarios out starting from a VM based on the OL6 template with an attached volume.

1. use the new volume to add to the root disk logical volume
2. create a new logical volume and attach it to the filesystem, say from /u01.

Snapshot both cases and create a new VM from the machineimage created and look to see what happened.

 Extending root volume

In the first case I simply create a VM from the OL6 base template using a simple orchestration.  I create an additional volume then attach the new volume to the VM.  Having created the VM I log onto it and use the unix commands for LVM to extend the size of the root disk.

The steps taken are:-

1. Use fdisk to format the attached volume to LVM.
2. Use lvdisplay and vgdisplay to identify the current root volume (Prob VolGroup00)
3. Use vgextend to extend the current volume group to add the storage from the attached volume.
4. Use lvextend to make the root logical volume larger
5. Use resize2fs the device mapper to make the extra space available to the filesystem.

Some of the key commands and output are shown below.  The result of all these commands is that the root filesystem has grown from 11G to 61G using all the 50Gb in the attached volume.

#df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   11G  3.3G  6.9G  33% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance # vgextend VolGroup00 /dev/xvdb1   Volume group "VolGroup00" successfully extended # pvdisplay   --- Physical volume ---   PV Name               /dev/xvda2   VG Name               VolGroup00   PV Size               17.75 GiB / not usable 2.12 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              4543   Free PE               191   Allocated PE          4352   PV UUID               VnZ5PZ-8IrP-nggg-B7Fo-9sog-g4bw-skbPBE      --- Physical volume ---   PV Name               /dev/xvdb1   VG Name               VolGroup00   PV Size               50.00 GiB / not usable 3.31 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              12799   Free PE               12799   Allocated PE          0   PV UUID               c1m21x-2IBX-Qrsy-1AJI-3UQ7-apUe-weETeq # lvextend -L+55G /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 66.00 GiB   Insufficient free space: 14080 extents needed, but only 12990 available # lvextend -l+12990 /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 61.74 GiB   Logical volume LogVol00 successfully resized   # resize2fs /dev/mapper/VolGroup00-LogVol00 resize2fs 1.43-WIP (20-Jun-2013) Filesystem at /dev/mapper/VolGroup00-LogVol00 is mounted on /; on-line resizing required old_desc_blocks = 4, new_desc_blocks = 4 The filesystem on /dev/mapper/VolGroup00-LogVol00 is now 16185344 blocks long. # df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   61G  3.3G   55G   6% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance

I now use the UI from EMCC to create a snapshot of the VM.  This could also be done from the command line.

Snapshotting a VM

The snapshot takes a few minutes to complete and once done there is a template available that will allow the creation of new VMs.

Snapshot appearing as a template in the OCM library

Creating a VM based on the snapshot

Once the new VM is created just log on and have a look at the root disk size.  It is immediately clear that it has a root disk of 61Gb and no attached volumes so by expanding the root LVM partition and snapshotting it will effectively increase the size of the disk in the machine image.

ERRATA - This approach does not work for increasing the root disk size.  Further investigation shows that while the OS reports the increased disk size all looks good but issuing a pvdisplay command reports that a device is missing.  This was confirmed by simply filling the disk up, as soon as the used space reached the same level as the original disk then warnings about the possible loss of data were reported and some of the new writes failed to get persisted to disk.  The conclusion - The actual disk space available was not expanded.

EXT4-fs error (device dm-2): ext4_wait_block_bitmap:448: comm flush-252:2: Cannot read block bitmap - block_group = 117, block_bitmap = 3670021
EXT4-fs (dm-2): delayed block allocation failed for inode 13345 at logical offset 16665 with max blocks 1 with error -5
EXT4-fs (dm-2): This should not happen!! Data will be lost

Adding the volume as a new partition/logical volume

The same process was completed for a volume being attached but this time rather than extendingVolGroup00 I created a new physical volume, volume group and logical volume which I mounted on /u01.  (fdisk to format volume to LVM [8e type], pvcreate to create the physical volume, vgcreate to create  a volume group using the new volume, lvcreate to create the logical volume and then mount the logical volume of /u01.)  Exactly the same process was used to create a snapshot and then create a new VM from the machineimage that was created.  This time round the resulting disk space on the new VM was just 11Gb - the disk size of the original template.  i.e. The snapshot has ignored the additional volume and done what the docs say, specifically take an image of the machine's boot disk.

Conclusion

As a mechanism to increase the root disk space of a template the approach of creating a VM with an attached volume and using this volume to extend the size of the root volume group/logical volume will NOT enable a new template with larger disk space.

If you are planning to create a VM with an attached volume and think that the snapshot will backup the entire VM then think again.  It will be necessary for you to also snapshot any volumes that are not part of the machine image boot disk to create a full snapshot of your virtual machine.  

Using Reporting in Enterprise Manager

Introduction

A common question that arises when talking to customers is to have visibility on the quantity of compute resource (CPU, Memory and Storage) that is being used.  This information is fairly easy to extract from the oracle-compute command line but it also is surfaced in Enterprise Manager.  As an exercise I wanted to try and produce a report from Enterprise Manager which gives the detail on the compute resource used.  This blog posting is just a capture of my experiences and not necessarily a best practice approach to reporting using EM12c against an OCM.

EM12c Reporting Overview

Enterprise Manager, as a monitoring tool, captures a great deal of information about anything it is monitoring.  Both configuration details and of course some historical information on the usage.  Provided you are logged in to the tool as a user with permissions to access reports and use the BI Publisher to create custom reports you will be able to find the reports under the Enterprise menu which typically is in the top left corner of the screen.

There are two options under reports, Information Publisher Reports which is a list of pre-defined reports that can be run to pull out commonly used reports and the BI Publisher Enterprise Reports.  The BI Publisher approach is the preferred route to use as this is now the report generator of choice for Enterprise Manager, others are deprecated and may eventually be dropped.  Like the Information Publisher Reports there are a series of out-the-box reports you can utilise but for many cases a custom report is the way to go. 

Creating a BI Publisher Report

BI Publisher is an incredibly powerful reporting tool that can query any database (or indeed even other sources of data) and push that data into an on-line report that can then be run on a regular basis, converted into PDF e-mailed out or run ad-hoc as needed.  With Enterprise Manager the main source of data is the underlying database of EM12c.

To create a report the process is essentially a two step process.  Firstly you must create a datamodel where you specify which tables to query, what the associations are between the tables, add filters and conditions to extract the specific data of interest.  Once the model has been defined you can optionally add in additional query parameters which can tune the report at run-time.  Once done you build a report up based on the data in the model, the report is built using simple wizards to produce tables of data, total up columns, display details on various graph types etc.

Building the DataModel for OCM

The Oracle Cloud Machine makes use of an EM12c Virtual Infrastructure plugin for most of the monitoring and management functionality.  This plugin stores much of its data in the tables that are prefixed with "MGMT$VI" and using BI Publisher we can create a new data model  that will pick the data we are interested.  When creating a new data model the default data source is called EMREPOS which is the database used by Enterprise Manager.  We can then simply type in the SQL if we know it in advance or alternatively use a "Query Builder" which allows us to dynamically build up the query using a fairly intuitive web based GUI.

The query builder allows us to drag and drop the tables onto a palate and select the fields we are interested in.  We can add conditions to the query, define linkage between tables etc.

In our specific use case we are looking to understand the resource used by the virtual machines, specifically allocated CPU, Memory and the storage volumes that have been added to the VMs.  This information is available in two tables, MGMT$VI_NM_OSV_CFG_DETAILS and MGMT$VI_NM_STORAGE_CFG.

BI Publisher has a mechanism to allow the report user to specify "parameters" which can be used to filter the data returned by the model.  It seems sensible to be able to query the model by tenancy I have added into the data model a parameter which will allow the user to specify one or more tenancies to report on.  For these tables the tenancy is effectively defined in the Quota. (an alternative breakdown might be per-orchestration)  To build up a parameter we have to create a "list of values" which the user can select from, as with the main data set this is defined via SQL queries against the database.  To show all tenancies I used the following SQL query:-

select "MGMT$VI_NM_OSV_CFG_DETAILS"."QUOTA" as "QUOTA" from "MGMT_VIEW"."MGMT$VI_NM_OSV_CFG_DETAILS" "MGMT$VI_NM_OSV_CFG_DETAILS"   where "MGMT$VI_NM_OSV_CFG_DETAILS"."QUOTA" !='RANDOMTEXT'     AND VNC_URL=(SELECT MAX(VNC_URL) from MGMT_VIEW.MGMT$VI_NM_OSV_CFG_DETAILS "B" WHERE b.QUOTA=MGMT$VI_NM_OSV_CFG_DETAILS.QUOTA)

This allows me to build up a list of tenancies (quota) which has been de-duplicated via the where clause.  (Could not get select distinct to work....)  This value list (the tenancies) is used as the selection for the Parameter which will be presented on the report to allow the user to narrow the report down to specific tenancies.

As shown in the screnshot I have selected to allow the user to chose multiple tenancies to report on or all the tenancies.  If all then a comma separated list of all tenancies on the rack is passed in as the parameter to the report.

Building the report

Once done we can turn attention to the report.  The first thing to do is to click on the data tab in the data model and press View to have a look at what data is actually returned by your datamodel.  If it looks like the correct information is being returned then click on "Save as Sample Data" and the data you returned is saved and used as the basis for data shown as the report is developed.

The easiest way to create the report from here is to click the "Create Report" button on the top right of the screen, this will open up a wizard to allow you to create the report and add charts and data tables to the report.

Having completed the report design we can then run the report.  In the screenshot below I have picked out three of the tenancies I am interested in and we can see at a moments glance that the JCS Demo tenancy is using most memory and CPU while the DBCS demo account is using most storage space.  Exactly as we would expect for a relatively small application.

Conclusion

Even although the OCM is managed by Oracle as a tenant user of the OCM rack it is fairly easy to use Enterprise Manager to gain insight into the usage of the OCM and BI Publisher provides a way to extract data to put into useful management reports.

Encrypting "disks" on Oracle Cloud Machine

Introduction

The Oracle Cloud Machine, like the public cloud, is administered by Oracle.  While the Oracle staff who manage the rack are highly skilled professionals and all their actions audited there is an obvious concern about the security of customer data at rest.  On the OCM the administrators of the rack have no direct access to the customer's virtual machines.  This article demonstrates how storage volumes can be used by a tenant to mount block storage devices that are encrypted and hence further obscured from system administrators.

(As a side effect of demonstrating the security aspect this is also a useful reference for using cryptsetup to encrypt disks.)

Setup

In order to demonstrate that a storage volume is encrypted and hence not visible to cloud administrators we do a very simple setup where two storage volumes are created, one to be encrypted and the other left in plain text.  These volumes are "attached" to a virtual machine and then within the virtual machine we use the linux utility cryptsetup to encrypt one of the volumes the other is simply mounted with an ext4 filesystem on it.  Plain text files are created in both volumes and then we will switch to the cloud administration side of things to see if it is possible to read the content of the two volumes.

Virtual Machine Instance Creation

First of all we create two storage volumes.  This can be done from the command line easily.


# oracle-compute add storagevolume /osc/public/encrypt-storage-001 10G /oracle/public/storage/default --description "A test 10Gb storage volume that we will try to have encrypted" 

# oracle-compute add storagevolume /osc/public/plain-storage-001 10G /oracle/public/storage/default --description "A test 10Gb storage volume that we will try to have encrypted"


Then we create a virtual machine via an orchestration defined in a json file

# cat simple_vm_with_storage.json
{
"name": "/osc/public/encryption-vm",
"oplans": [
{
 "obj_type": "launchplan",
 "ha_policy": "active",
 "label": "encryption volume launch plan",
 "objects": [
 {
 "instances": [
 {
 "label": "encryption-vm001",
 "imagelist": "/oracle/public/linux6_16.1.2_64",
 "networking":
 {
   "net0": { "vnet": "/osc/public/vnet-eoib-1706" }
 },
 "storage_attachments": [
 { "volume": "/osc/public/encrypt-storage-001", "index": 1},{"volume": "/osc/public/plain-storage-001", "index": 2}],
 "shape": "ot1",
 "sshkeys": ["/osc/public/labkey"],
 "attributes":
 {
 "userdata":
 {
 "key1": "value 1",
 "key2": "value 2"
 }
 }
 } ]
 } ]
} ]
}


This json file will create a single instance called encryption-vm001 based on the OL6 base template, connect it to the EoIB public network and attach the two storage volumes that we created earlier.  (Storage volumes created independently of this orchestration in this case.)

We upload the orchestration and start it.  Once up and running then the instance will be listed as running and we can see the IP address assigned to it.

# oracle-compute add orchestration ./simple_vm_with_storage.json 

(see above for json)

# oracle-compute start orchestration /osc/public/encrytption-vm

# oracle-compute list instance /osc -Fname,state,ip

Configuring volumes within instance

Having created and started up our instance we can look at the attached volumes and run through the process using Oracle Linux to setup one of the volumes as an encrypted one.   To see the volumes on the instance we use the fdisk command.



# fdisk -l

Disk /dev/xvda: 19.3 GB, 19327352832 bytes
255 heads, 63 sectors/track, 2349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c520c


    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          32      256000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/xvda2              32        2349    18611318+  8e  Linux LVM



Disk /dev/xvdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000



Disk /dev/xvdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol01: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol00: 11.8 GB, 11811160064 bytes
255 heads, 63 sectors/track, 1435 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup00-LogVol02: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

With the OCM each volume that is attached gets an index, in the orchestration above we use indexes 1 and 2.  These numbers equate to the xvd<char> devices that appear in the fdisk output.where 1 equates to b, 2 equates to c etc.  Thus in the output above the two attached volumes are /dev/xvdb and /dev/xvdc.  The next step is to setup one of the volumes as an block device encrypted one.  To do this I used the linux command cryptsetup defining cipher information etc.  In the example shown below I show it run twice as the first time I answered the question with a lower case yes.  The command mandated uppercase YES as an answer.  Easy mistake to make!



# cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/xvdb



WARNING!
========
This will overwrite data on /dev/xvdb irrevocably.


Are you sure? (Type uppercase yes): yes
Command failed with code 22: Invalid argument

# cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/xvdb

WARNING!
========

This will overwrite data on /dev/xvdb irrevocably.
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.



Now we can open the encrypted drive such that it appears as normal.  This will create the /dev/mapper/<name> device file and allow it to be mounted by the OS.  The luksOpen command will prompt for the passphrase used earlier.

# cryptsetup luksOpen /dev/xvdb encrypted-drive

# cryptsetup -v status encrypted-drive
/dev/mapper/encrypted-drive is active.
  type:  LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/xvdb
  offset:  4096 sectors
  size:    20967424 sectors
  mode:    read/write
Command successful.

This is a new raw volume so we need to put some sort of filesystem onto it.  In this case I use the ext4 filesystem.


# mkfs.ext4 /dev/mapper/encrypted-drive
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2620928 blocks
131046 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done                           
Writing inode tables: done                           
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now simply create a directory where we can mount the encrypted drive and create a simple text file.

# mkdir /u01
# mount /dev/mapper/encrypted-drive /u01
# df -kh
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   11G  3.3G  7.0G  32% /
tmpfs                            3.8G     0  3.8G   0% /dev/shm
/dev/xvda1                       239M   55M  168M  25% /boot
/dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance
/dev/mapper/encrypted-drive      9.8G   23M  9.2G   1% /u01

 Having done this we can do a quick check to ensure that we can unmount and close the encrypted disk and re-open it providing the passphrase and mount it for use.


#umount /u01
# cryptsetup luksClose encrypted-drive
# mount /dev/mapper/encrypted-drive /u01
mount: you must specify the filesystem type



# cryptsetup luksOpen /dev/xvdb encrypted-drive
Enter passphrase for /dev/xvdb:
# mount /dev/mapper/encrypted-drive /u01
# df -kh
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   11G  3.3G  7.0G  32% /
tmpfs                            3.8G     0  3.8G   0% /dev/shm
/dev/xvda1                       239M   55M  168M  25% /boot
/dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance
/dev/mapper/encrypted-drive      9.8G   23M  9.2G   1% /u01



Using fdisk we can format the /dev/xvdc volume, create a file system on this volume and mount it into another directory.  Then create a plain text file in this volume as well.   If the encryption has all worked then cloud operations may be able to access the plain text volume and read the content but the encrypted volume content is kept secret unless the passphrase is known.

Testing

As a general rule cloud operations do not have access to the customer's virtual machines unless the customer shares the login credentials or the ssh keys with Oracle.  However, because the OCM stores the volumes as raw disk images on the internal ZFS storage appliance in the EPC_<rack>/storagepool1 filesystem it is possible for cloud operations to access these files and mount the images directly to access the content.


As a cloud operations user I have accessed the ZFS storage device and can copy the storage volume disks off the rack.  In a linux server I attempt to mount these volumes to see the content.

# file plain_storage.raw
plain_storage.raw: Linux rev 1.0 ext4 filesystem data (extents) (large files) (huge files)
# mount -o loop ./plain_storage.raw /mnt/don
# cat /mnt/don/don-plain

This text is in the unencrypted volume and hence should be readable by anyone.....
# unmount /mnt/don



So it is obviously fairly easy to access the unencrypted storage.  Now lets see what is involved in accessing the encrypted storage volume.

# file encrypted_storage.raw
encrypted_storage.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha512] UUID: edff3d80-3813-4abc-a58c-e2f1862

# mount -o loop ./encrypted_storage.raw /mnt/don
mount: unknown filesystem type 'crypto_LUKS'

# losetup /dev/loop0 ./encrypted_storage.raw
# mount /dev/loop0 /mnt/don
mount: unknown filesystem type 'crypto_LUKS'

# cryptsetup luksOpen /dev/loop0 encrypted-dev
Enter passphrase for /dev/loop0:

# mount /dev/mapper/encrypted-dev /mnt/don

# cat /mnt/don/don

some text

#

In the above I attempt to mount the encrypted filesystem using the same mechanism as previously was successful but to no effect.  The only way to mount the disk is to make use of the cryptsetup command which mandated entering the passphrase.  Obviously the passphrase is not something that is shared with cloud operations so they would be unable to access the content of the raw file.

Conclusion

Certainly using the standard linux command of cryptsetup it is a relatively simple task to encrypt any storage volume that is mounted on a VM such that the data is kept private to the end customer/tenant and cloud operations has no mechanism of seeing the content.

The down side of encrypting is that it means that the administrator of the virtual machine (end customer) has to log on and provide the passphrase to mount the volume.  Not a major problem unless you are looking at trying to automatically start up the applications deployed that use the encrypted volume, in this case it becomes necessary to have a manual startup procedure.

Introducing Oracle Cloud Machine

As mentioned in my exablurb blog I am now working beyond the bounds of Oracle Engineered Systems to incorporate the Oracle Cloud Machine as introduced here.  As such this blog will include postings that apply to both Exalogic and the Cloud Machine.


I'll start by quoting one of the PM's from the cloud machine to explain just what the cloud machine actually is:

"Oracle Cloud Machine is a cloud offering which gives you new choices for the Oracle Cloud Platform by bringing the Oracle Cloud to your data center. Leveraging our Public Cloud’s PaaS and IaaS capabilities, it enables the innovation that cloud provides, at the same time meeting the business and regulatory requirements behind your firewall. It provides a stepping-stone in the journey to cloud, as it allows you to get the advantages of cloud faster, easier and with less disruption. As an on- premises implementation of Oracle Cloud, Oracle Cloud Machine lets you run your applications seamlessly wherever you want, as workloads are completely portable between the public cloud and your data center. You can now leverage the latest innovations for rapid development that cloud provides, all while meeting any data sovereignty and residence requirements. It also provides subscription based pricing in your data center, managed by Oracle, with single vendor accountability."

Or to put it simply the cloud machine is a bunch of compute services that Oracle come along and install in your data center and then run it as a service such that you, as a customer, can consume IaaS and PaaS services without having to worry about building up the management infrastructure and on-going operational management of the platform.

Lots more information can be found from the public documentation.

Networks that span multiple Engineered Systems/Exalogic Accounts

This blog post is to introduce some functionality that has fairly recently (~Oct 2015) become available that allows additional infiniband shared networks to be defined.  This enables internal networks to span accounts or be extended to other Engineered Systems.

Historically an Exalogic rack is setup with two internal (IPoIB) networks that have IP addresses which can be handed out to vServers in all accounts, these are the vServer Shared Storage and the IPoIB Default networks.  Any vServers on the storage network are limited members and full members of the infiniband default network. It is possible to override the membership of a virtual machine to allow vServers to communicate to each other internally on the Infiniband storage net.

Security concerns about using the IPoIB default network to allow inter-vServer communication alongside access to the database tier has meant that this network tends not to be used to allow cross-account conversations.   The only other mechanism to allow network traffic between accounts was to use a public EoIB network which has the downside of preventing the Infiniband high performance protocols and mandating the smaller MTU sizes and thus is sub-optimal for performance based applications.

Recent changes in Exadata have introduced support for the use of non-default partitions.  Indeed, when Exadata is setup to make use of the database running in a virtual machine the normal configuration will be such that there is no use of the IPoIB_default partition (0x7fff).   This was a problem for Exalogic which historically only had access to Exadata over the IPoIB-default network.

The standard configuration of a virtualised Exadata is to have two IB partitions, one that allows the database server to talk to the storage servers and another that will connect the virtual machine to another virtual machine on the Exadata so that a distributed RAC cluster can be setup and use IB for inter-cluster communications.  Obviously if Exalogic wants to communicate to Exadata using the Infiniband Optimised protocols the Exalogic must be able to link in with the Exadata over a non-default infiniband partition.  This is depicted in figure 1 below.

Figure 1 - Connecting EL and ED using non-default Infiniband Network

This example shows a two tier application deployed to Exalogic, the web tier which has access to the EoIB client network, potentially hosting an application like Oracle Traffic Director.  This can forward requests on to an application tier over an internal private network and then the application tier is linked to another IPoIB internal network but this is what might be considered a "public private network" meaning that this network can be handed out to vServers and provide linkage to the Exadata virtual machines which have had this specific network (partition) allocated to them.  The Exadata also has two other internal IB networks, one to allow the RAC cluster to communicate between the DB servers and another to allow access to the storage cells.

The approach to creating this non-default network that spans both Exalogic and Exadata introduces a couple of potential options.  Firstly to extend a private network from an Exalogic account into the Exadata rack and secondly to create a new Exalogic custom shared IPoIB network which can span multiple Exalogic accounts.

Extending a Private Network

In this scenario we create a private network within an Exalogic account and then expand the Infiniband partition into the Exadata.  This means that access to the Exadata is kept purely within an Exalogic account.  The steps to go through are:

1. Create a private network in an Exalogic Account
2. Edit the network to reserve the IP addresses in the subnet that the Exadata will use.
3. Identify the  pkey value that this new network has been assigned
4. Using the IB command line/Subnet Manager make the new partition extend to the Exadata switches and database servers.
5. Recreate the Exadata virtual machines adding the new partition key to the virtual machine configuration file used.
6. Configure the Exadata VM to use an IP address made unavailable to the Exalogic

Creating a new "Custom Shared IPoIB Network"

This is a slightly more flexible approach than the first scenario as we create a new "public private" network and then allocate IP addresses on this network to each account that will need access to it.  This is also useful in the use cases that Exadata is not involved because it allows certain virtual machines to be setup as a service provider and others as service consumers.  A provider being an IB full member of the partition and a consumer a limited member.  Thus all consumers can access and use the service provider functions but the consumers cannot "see" each other.

This example is is for the connected Exadata that we discussed earlier.  In this case the process to follow is:-

1. Run the process to create the new IPoIB network.  It can be setup such that all vServers will be limited or full members by default, defines the IB Partition and specifies the subnet used as well as which IP addresses the Exalogic rack will use.
2. Allocated a number of IP addresses from this new network to each account that will use it.  Same process that is used for EoIB networks, storage network or the IPoIB Default network today.
3. Create vServers in the accounts with an IP address on the custom shared network.
4. Identify the pkey for the custom network and extend the partition to the Exadata switches and DB server nodes.  The primary difference here is that if the Exadata was setup first then the first step in this process would have been to specify the pKey that was originally used by the Exadata.  (i.e. Either the Exadata or the Exalogic can be the first to specify the pKey.)
1. Warning - The pKey being used is defined manually.  Make sure it will not overlap with any pKeys that Exalogic Control will assign.
5. Recreate the database virtual machines assigning the pkey to their configuration and within the VM specify the IP address you want them to use.  
6. Test

Note - The technical details on how to achieve this are fully documented in an Oracle support note.  Get in touch with your local Oracle representative find out more.

Oracle Traffic Director - Deployment options, Virtual Servers vs Configurations

Summary

Oracle Traffic Director (OTD) is a powerful software load balancing solution.  As with most good products there is a degree of flexibilty in how it can be deployed with different approaches allowing the solution to be formed.  This article discusses two options that could be used to determine different routing possiblilties.

The scenario that is being considered is a need to perform two separate load balancing activities in the same OTD environment.  For example, load balancing to an older SOA 11g deployment and to SOA 12c for recent integration deployments.  Another possible example would be two routes to the same back end service but one is designed for high priority traffic while the other route will throttle the service at a preset load.   The two options that are discussed are:

1. Using two separate configurations, one for SOA 11g and one for SOA12c.
2. Using one configuration that has two virtual servers.  The virtual servers handling the routing for each environment.

Needless to say either option can be appropriate and it will depend on the details of the overall solution and to some extent personal preference to determine the right answer for a particular customer environment.  Of course other options such as more complex routing rules within a single configuration or multiple OTD domains are also options to think about.

OTD Configuration Overview

Simple configuration

An OTD deployment, in its simplest form, consists of an administrative instance which manages the configuration and a deployed instance.  The deployed configuration specifies the HTTP(S)/TCP listening port, routing rules to one or more origin servers, logging setup etc.   In many situations there is a business need to use OTD to manage requests to different business applications or even just to different environments/versions of an application.  It is obviously possible to split these out by using independent deployments of OTD however to minimise the resources required and keep the number of deployed components to a minimum there are options to use one administration server.

The base configuration options

The minimum configuration that will appear for a configuration is a setup which defines things like the listening ports, SSL certificates, logging setup and critically at least one origin server pool and a virtual server.  The origin server pool is a simple enough concept in that it defines the back end services to actually fulfil the client requests. 

Using Virtual Servers

The virtual servers provide a mechanism to isolate traffic sent to the software load balancer.  Each virtual server contains its own set of routing rules which can determine the origin servers to send requests to, caching rules, traffic shaping and overrides for logging and the layer 7 firewall rules.  The virtual server to be used for subsequent processing is identified by either the listening port or the hostname used to send the request.

Virtual Server example - Routing based on otrade-host

Virtual Server example - Routing based on websession-host

So in the above example both hostnames otrade-host and websession-host resolve to the same IP address in DNS (or in the clients local /etc/hosts file).   In this case two virtual servers also use the same listener.  If the client makes a request to access otrade-host then the first virtual server is used and if they request websession-host then the second's rules are used.

There is always at least one virtual server.  By default this is created and the hosts field left blank such that it is used if any traffic hits the listening port.

Solution Variations

Multiple Configurations

Overview

In this setup two configurations can be defined and deployed.  It is quite possible to have both configurations deployed to the same OS instances. (Admin node in OTD talk.)  The result of deploying the configuration to the admin nodes is the creation of another running instance of OTD.

Running multiple configurations

Thus in the example shown above we have three OS instances, one to host the admin server which could be co-located with the actual instances.  There are two OS instances which host two OTD servers, one for each configuration.  I have shown two OS instances to run the configuration to indicate that they can be setup in a failover group to provide HA, each config can utilise a different VIP.

Advantages

• Each configuration is managed independently of each other.  (within the one administration server) 
• The settings are independent of each other.
• The running instances for each configuration are independent of each other. i.e. Can be stopped and started without impacting the other configuration instances running.
• Simple to understand

Disadvantages

• Care must be taken to ensure that the configurations do not have clashes with each other.  (eg. Same listenting ports)
• Results in more processes running on each OS instance.

Multiple Virtual Servers

Overview

In this situation there is one configuration with multiple virtual servers which result in different routing rules being applied to send requests on.   In the diagram below we have deployed a single configuration to two OS instances with the configuration containing two virtual servers.  As per the multi-config option I have shown two OS instances to indicate that the failover group can be used for HA.

OTD Using Two Virtual Servers

Advantages

• One configuration that provides visibility of all configuration in the environment.
• Minimal running processes
• Simplifying the monitoring
• Reducing resources required to run the system

Disadvantages

• Introduces dependencies between the environments
• eg. Can share listeners, origin server pools, logging config etc.  Thus one change can impact all instances
• eg. Some changes mandate a restart of an instance.  A change for one config may have an impact on load balancing for the other environment.
• Complexity of a single configuration
• Dependencies on external factors.  (DNS resolution of hostnames/firewalls for port access.)

Conclusions

There are no hard and fast rules to figure out which approach is the best one for you.  It will ultimately depend on the requirements for the load balancing.  If a configuration is changing frequently and is functionally independent then I would tend to go for the multiple configuration route.  If on the other hand simplicity of monitoring and minimal resource footprint alongside a fairly static configuration was the situation I would tend to use the multiple virtual server approach.

Essentially the classic IT answer of "it depends" will apply.  Only a good understanding of the requirements will clarify which way to go.  (Although if you are using OTD 11.1.1.6 then you might be better with the virtual server approach as there are a few limitations to the VIPs using keepalive for the failover groups)