Expanding the root volume in an OCM template

Introduction

The OCM is delivered to you with two templates that are specifically ready for use by the Infrastructure as a Service interfaces.  These templates are OL5 and OL6 and by default the root volume has approximately 11Gb of space available.  This should be ample as a root volume for 90% of customer uses.  However from time to time it might be useful to increase this disk space in a template.  For example a customer might want to have a number of applications deployed into the base template, things like specific package installs, monitoring tools etc.  These could use up much of the disk space making management of logs etc. more space critical than would be liked.

This blog posting goes through a process that will enable the root disk structure to be increased in size.  Specifically for the OL6 template.  The process revolves around three actions.

1. Download the base template from your OCM rack.
2. Make use of "EKIT" to increase the disk space
3. Add the new template back onto the OCM and make it available for use.

Step 1 - Download the template

The first activity is to download the template.  This can be done easily from the Command line - oracle-compute.

# oracle-compute download machineimage /oracle/public/linux6_16.3.1_64

This will create a directory structure where the file is downloaded as a gziped tar file.

Step 2 - Use ModifyJeosLVM.sh to increase disk space

Handily there is a toolkit known as the "Exalogic Kinetic Infrastructure Tools" or EKIT for short which contains a script called ModifyJeosLVM.sh which has been designed to extend the root volume (or swap space) of an OL LVM based image.  EKIT is available from My Oracle Support and can be easily found using MOS document ID of 1933252.1.

Install the EKIT into an appropriate environment according to the toolkit installation instructions.  (Can be on a laptop, VM on a laptop or a VM on OCM itself although you will need to ensure that the VMs have enough disk space to handle files as large as you intend to make the root volume.)  Once installed copy the downloaded <template>.tar.gz file to your EKIT environment.   Extract the machineimage (will be a System.img file)  And run ModifyJEOSLVM.sh as shown below.  This adds space to the current image file and then tar/gzip the image file and you have a machineimage ready to upload into your OCM rack.

[root@utility ocm-ol6]# gunzip linux6_16.3.1_64.tar.gz [root@utility ocm-ol6]# ls linux6_16.3.1_64.tar [root@utility ocm-ol6]# tar xvf linux6_16.3.1_64.tar System.img [root@utility ocm-ol6]# ls -lh total 37G -rw-r--r--+ 1 root root 19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 18G Sep 26 05:31 System.img  [root@utility ocm-ol6]# ModifyJeosLVM.sh --resize -if System.img -er 20 Img File Array: System.img *********************************************** *** Modifying System.img *********************************************** Block size: 1024 Base Image Size 19327352832 Base Image 18874368 blocks Adding 20971520 blocks to root file system Adding 0 blocks to swap file system Adding 0 blocks free space to Volume Group Adding 524288 spare blocks Resizing image file to 40370176 *********************************************** *** Checking Primary Partition Status System.img *********************************************** *********************************************** *** Using loop /dev/loop0 *********************************************** Disk /dev/loop0: 19.3 GB, 19327352832 bytes 255 heads, 63 sectors/track, 2349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM 1. Found /dev/loop0p1 2. Found /dev/loop0p2 *********************************************** *** Extending System.img *********************************************** 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000268591 s, 0.0 kB/s *********************************************** *** Using loop /dev/loop0 *********************************************** Disk /dev/loop0: 41.3 GB, 41339060224 bytes 255 heads, 63 sectors/track, 5025 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM 1. Found /dev/loop0p1 2. Found /dev/loop0p2 Creating new Primary partition 3 *********************************************** (echo n; echo p; echo 3; echo ; echo ; echo t; echo 3; echo 8e; echo w) | fdisk /dev/loop0 *********************************************** WARNING: DOS-compatible mode is deprecated. It's strongly recommended to          switch off the mode (command 'c') and change display units to          sectors (command 'u'). Command (m for help): Command action    e   extended    p   primary partition (1-4) Partition number (1-4): First cylinder (2350-5025, default 2350): Using default value 2350 Last cylinder, +cylinders or +size{K,M,G} (2350-5025, default 5025): Using default value 5025 Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 3 to 8e (Linux LVM) Command (m for help): The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 22: Invalid argument. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. Disk /dev/loop0: 41.3 GB, 41339060224 bytes 255 heads, 63 sectors/track, 5025 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c520c       Device Boot      Start         End      Blocks   Id  System /dev/loop0p1   *           1          32      256000   83  Linux Partition 1 does not end on cylinder boundary. /dev/loop0p2              32        2349    18611318+  8e  Linux LVM /dev/loop0p3            2350        5025    21494970   8e  Linux LVM loop: can't delete device /dev/loop0: Device or resource busy =============================================== loop0p1|:|0|512000|/dev/loop0|2048 loopId: loop0p1 loop0p2|:|0|37222637|/dev/loop0|514048 loopId: loop0p2 loop0p3|:|0|42989940|/dev/loop0|37736685 loopId: loop0p3 loop|deleted|:|/dev/loop0 [root@utility ocm-ol6]# ll -h total 37G -rw-r--r--+ 1 root root 19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 39G Jan 27 08:41 System.imgloopId: loop0p3 Using loopId: loop0p3 =============================================== ################################################################################ ####                         getRunningSystemInfo                           #### ################################################################################ ******************************************************************************** *** *** utility has Volume Group *** *** RHEL Version : 6.7 *** OL Version   : 6.7 *** *** File System  : lvm *** *** Volume Group : VGEKITUtility *** Root Log Vol : LVRoot *** Swap Log Vol : LVSwap *** ******************************************************************************** ################################################################################ ####                         createLoopFileSystem                           #### ################################################################################ ################################################################################ ####                          getSystemImgFSType                            #### ################################################################################ ******************************************************************************** *** *** Mounted Image File System *** *** File System  : ext3 *** ******************************************************************************** Loop device is /dev/loop0 add map loop0p1 (252:2): 0 512000 linear /dev/loop0 2048 add map loop0p2 (252:3): 0 37222637 linear /dev/loop0 514048 add map loop0p3 (252:4): 0 42989940 linear /dev/loop0 37736685 ################################################################################ ####                           getSystemImgInfo                             #### ################################################################################ ******************************************************************************** *** *** Mounted Image has no Volume Group *** *** RHEL Version : *** OL Version   : *** *** File System  : ext3 *** ******************************************************************************** ################################################################################ ####                      validateLoopFileSystemType                        #### ################################################################################ ******************************************************************************** *** *** Specified System Image file has ext3 file system so we should be okay. *** ******************************************************************************** *********************************************** *** Scanning Volume Groups ***********************************************   Reading all physical volumes.  This may take a while...   Found volume group "VolGroup00" using metadata type lvm2   Found volume group "VGEKITUtility" using metadata type lvm2 *********************************************** *** Extending VolGroup00 *********************************************** ===============================================   Physical volume "/dev/mapper/loop0p3" successfully created   Volume group "VolGroup00" successfully extended =============================================== *********************************************** ***** Extending Root LogVol00 ***********************************************   Size of logical volume VolGroup00/LogVol00 changed from 11.00 GiB (2816 extents) to 31.00 GiB (7936 extents).   Logical volume LogVol00 successfully resized *********************************************** *** Changing VolGroup00 ***********************************************   3 logical volume(s) in volume group "VolGroup00" now active *********************************************** *** Listing Directories *********************************************** *** ls -lh /dev/mapper *********************************************** total 0 crw-rw---- 1 root root 10, 236 Jan 16 05:43 control lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p1 -> ../dm-2 lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p2 -> ../dm-3 lrwxrwxrwx 1 root root       7 Jan 27 08:41 loop0p3 -> ../dm-4 lrwxrwxrwx 1 root root       7 Jan 16 05:43 VGEKITUtility-LVRoot -> ../dm-1 lrwxrwxrwx 1 root root       7 Jan 16 05:43 VGEKITUtility-LVSwap -> ../dm-0 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol00 -> ../dm-6 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol01 -> ../dm-5 lrwxrwxrwx 1 root root       7 Jan 27 08:41 VolGroup00-LogVol02 -> ../dm-7 *** ls -lh /dev/V* *********************************************** /dev/VGEKITUtility: total 0 lrwxrwxrwx 1 root root 7 Jan 16 05:43 LVRoot -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 16 05:43 LVSwap -> ../dm-0 /dev/VolGroup00: total 0 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol00 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol01 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jan 27 08:41 LogVol02 -> ../dm-7 Oracle Linux 6 (6.7) *********************************************** ***** Checking Root LogVol00 *********************************************** /dev/VolGroup00/LogVol00: 154624/720896 files (0.1% non-contiguous), 925814/2883584 blocks *********************************************** ***** Resizing Root LogVol00 *********************************************** resize2fs 1.43-WIP (20-Jun-2013) Resizing the filesystem on /dev/VolGroup00/LogVol00 to 8126464 (4k) blocks. The filesystem on /dev/VolGroup00/LogVol00 is now 8126464 blocks long. *********************************************** ***** Setting Swap LogVol01 *********************************************** mkswap: /dev/VolGroup00/LogVol01: warning: don't erase bootbits sectors         on whole disk. Use -f to force. Setting up swapspace version 1, size = 4194300 KiB no label, UUID=73a92d9c-4c00-4013-9681-a8e5ae2994b6   0 logical volume(s) in volume group "VolGroup00" now active *********************************************** ***** Changing UUID of VolGroup00 ***********************************************   Volume group "VolGroup00" successfully changed ################################################################################ ####                         deleteLoopFileSystem                           #### ################################################################################ *********************************************** *** Unmounting /dev/loop0 *********************************************** Processing System.img *********************************************** *** System.img Modified *** Start Size : -rw-r--r--+ 1 root root 18G Sep 26 05:31 System.img *** End Size   : -rw-r--r--+ 1 root root 39G Jan 27 08:41 System.img *********************************************** [root@utility ocm-ol6]# tar cvf linux6-40G_16.3.1_64.tar System.img System.img [root@utility ocm-ol6]# ls -lh total 39G -rw-r--r--+ 1 root root  19G Jan 27 08:29 linux6_16.3.1_64.tar -rw-r--r--+ 1 root root 2.5G Jan 27 08:44 linux6-40G_16.3.1_64.tar.gz -rw-r--r--+ 1 root root  39G Jan 27 08:41 System.img

Step 3 - Adding the new machineimage to your OCM

The final step in the process is to copy the new <template>.tar.gz file back to where the OCM CLI has been installed and then add it to the environment.  To make a machine image available for use there are three steps.

1. Add the machineimage file 
2. Add an imagelist to the environment
3. Add a imagelistentry to link the machine image to the imagelist.

The OCM makes use of an imagelist  when specifying which template to use as the base OS for a VM.  By abstracting the connection between the VM and its machineimage we introduce a useful mechanism for version control of the VM so VM creation scripts do not change even if we want to change the underlying machineimage.

The steps to make it available for use are shown below:-

# oc add machineimage /oracle/public/linux6-40G-16.3.1_64 /root/nimbula-machineimages-2017-01-27/linux6-40G_16.3.1_64.tar.gz [====================================================================================================] # oc add imagelist /oracle/public/linux6-40G-16.3.1_64 "Image list for base template expanded to 40G in size" uri                                                          name                                    description                                              default     entries    https://api/imagelist/oracle/public/linux6-40G-16.3.1_64     /oracle/public/linux6-40G-16.3.1_64     Image list for base template expanded to 40G in size     1                       # # oc add imagelistentry /oracle/public/linux6-40G-16.3.1_64 /oracle/public/linux6-40G-16.3.1_64 1 uri                                        imagelist                                                                                               machineimages                  versi a https://api/imagelist/oracle/public/linux6 {"default": 1, "description": null, "entries": null, "uri": "imagelist/oracle/public/linux6-40G-16.3.1_ /oracle/public/linux6-40G-16.3 1     { #

Note - In the example above I have used the /oracle tenancy to host the template.  This is a tenancy that only oracle cloud operations have access to and by placing the imagelist in this tenancy it makes the template available to all tenancies on the rack.  If you want this to be the situation then you must raise a request for cloud operations to add the objects.  However you can also do this within another tenancy in which case the imagelist will only be available for use within that tenancy.

Below is a screenshot from the self service portal showing the imagelist as available for use as a template.

What happens to attached volumes when snapshotting VMs on an OCM

Introduction

With OCM there is the capability to take a "snapshot" of a virtual machine.  As described in the documentation a snapshot will take a copy of the machine image boot disk.  Essentially there are two purposes for this action to take place, firstly if we take a machineimage then we can use that machineimage to create additional VMs from this copy - using it as a template.  Secondly it can be used as a backup copy to recreate the VM if needed.  (Really just the same as the first but using the copy to recreate rather than clone.)

This leads me to think of a couple of questions that this blog posting will be answering.

1. What happens to storage volumes that have been added to the VM.  Are these copied as well?
2. Is this a good mechanism to increase the root volume size to make more space for VMs that might want a bit more disk space?

I have tested two specific scenarios out starting from a VM based on the OL6 template with an attached volume.

1. use the new volume to add to the root disk logical volume
2. create a new logical volume and attach it to the filesystem, say from /u01.

Snapshot both cases and create a new VM from the machineimage created and look to see what happened.

 Extending root volume

In the first case I simply create a VM from the OL6 base template using a simple orchestration.  I create an additional volume then attach the new volume to the VM.  Having created the VM I log onto it and use the unix commands for LVM to extend the size of the root disk.

The steps taken are:-

1. Use fdisk to format the attached volume to LVM.
2. Use lvdisplay and vgdisplay to identify the current root volume (Prob VolGroup00)
3. Use vgextend to extend the current volume group to add the storage from the attached volume.
4. Use lvextend to make the root logical volume larger
5. Use resize2fs the device mapper to make the extra space available to the filesystem.

Some of the key commands and output are shown below.  The result of all these commands is that the root filesystem has grown from 11G to 61G using all the 50Gb in the attached volume.

#df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   11G  3.3G  6.9G  33% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance # vgextend VolGroup00 /dev/xvdb1   Volume group "VolGroup00" successfully extended # pvdisplay   --- Physical volume ---   PV Name               /dev/xvda2   VG Name               VolGroup00   PV Size               17.75 GiB / not usable 2.12 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              4543   Free PE               191   Allocated PE          4352   PV UUID               VnZ5PZ-8IrP-nggg-B7Fo-9sog-g4bw-skbPBE      --- Physical volume ---   PV Name               /dev/xvdb1   VG Name               VolGroup00   PV Size               50.00 GiB / not usable 3.31 MiB   Allocatable           yes   PE Size               4.00 MiB   Total PE              12799   Free PE               12799   Allocated PE          0   PV UUID               c1m21x-2IBX-Qrsy-1AJI-3UQ7-apUe-weETeq # lvextend -L+55G /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 66.00 GiB   Insufficient free space: 14080 extents needed, but only 12990 available # lvextend -l+12990 /dev/VolGroup00/LogVol00   Extending logical volume LogVol00 to 61.74 GiB   Logical volume LogVol00 successfully resized   # resize2fs /dev/mapper/VolGroup00-LogVol00 resize2fs 1.43-WIP (20-Jun-2013) Filesystem at /dev/mapper/VolGroup00-LogVol00 is mounted on /; on-line resizing required old_desc_blocks = 4, new_desc_blocks = 4 The filesystem on /dev/mapper/VolGroup00-LogVol00 is now 16185344 blocks long. # df -kh Filesystem                       Size  Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00   61G  3.3G   55G   6% / tmpfs                            873M     0  873M   0% /dev/shm /dev/xvda1                       239M   55M  168M  25% /boot /dev/mapper/VolGroup00-LogVol02  2.0G  3.0M  1.9G   1% /opt/emagent_instance

I now use the UI from EMCC to create a snapshot of the VM.  This could also be done from the command line.

Snapshotting a VM

The snapshot takes a few minutes to complete and once done there is a template available that will allow the creation of new VMs.

Snapshot appearing as a template in the OCM library

Creating a VM based on the snapshot

Once the new VM is created just log on and have a look at the root disk size.  It is immediately clear that it has a root disk of 61Gb and no attached volumes so by expanding the root LVM partition and snapshotting it will effectively increase the size of the disk in the machine image.

ERRATA - This approach does not work for increasing the root disk size.  Further investigation shows that while the OS reports the increased disk size all looks good but issuing a pvdisplay command reports that a device is missing.  This was confirmed by simply filling the disk up, as soon as the used space reached the same level as the original disk then warnings about the possible loss of data were reported and some of the new writes failed to get persisted to disk.  The conclusion - The actual disk space available was not expanded.

EXT4-fs error (device dm-2): ext4_wait_block_bitmap:448: comm flush-252:2: Cannot read block bitmap - block_group = 117, block_bitmap = 3670021
EXT4-fs (dm-2): delayed block allocation failed for inode 13345 at logical offset 16665 with max blocks 1 with error -5
EXT4-fs (dm-2): This should not happen!! Data will be lost

Adding the volume as a new partition/logical volume

The same process was completed for a volume being attached but this time rather than extendingVolGroup00 I created a new physical volume, volume group and logical volume which I mounted on /u01.  (fdisk to format volume to LVM [8e type], pvcreate to create the physical volume, vgcreate to create  a volume group using the new volume, lvcreate to create the logical volume and then mount the logical volume of /u01.)  Exactly the same process was used to create a snapshot and then create a new VM from the machineimage that was created.  This time round the resulting disk space on the new VM was just 11Gb - the disk size of the original template.  i.e. The snapshot has ignored the additional volume and done what the docs say, specifically take an image of the machine's boot disk.

Conclusion

As a mechanism to increase the root disk space of a template the approach of creating a VM with an attached volume and using this volume to extend the size of the root volume group/logical volume will NOT enable a new template with larger disk space.

If you are planning to create a VM with an attached volume and think that the snapshot will backup the entire VM then think again.  It will be necessary for you to also snapshot any volumes that are not part of the machine image boot disk to create a full snapshot of your virtual machine.