Monday, March 27, 2017

My experiences with migrating from ESXi 6.5 to SmartOS build: 20170315T210557Z (Part 1)

Greetings tech junkies,

After being a long-time ESXi user, I have decided to migrate to SmartOS.  In a nutshell, my reasons are:
  1. Great respect for the Illumos platform (true enterprise UNIX, ZFS, zones, dtrace, etc.)
  2. Wanting to use containers primarily as opposed to only fully-virtualized HW VMs
  3. Preferring VM storage to be stored in local ZFS pool (vmfs kinda sucks by comparison and I have less regard for the idea of scaling to external iSCSI devices)
  4. Wanting to try something different
  5. Needing a good challenge
  6. Geekius Maximus

With that being said, ESXi is a great hypervisor for beginners - it's easy to use, has a great GUI, has free versions that are perfectly adequate for most users, etc.  I can't say enough good things about it, honestly.

That being said, I tihink SmartOS is going to serve me better.  I want to run more containers and ESXi doesn't have a container platform.  It virtualizes hardware, and that's pretty much it. 

With SmartOS, I can import LX container datasets that are premade and run them with virtually bare-metal performance, or delegate certain resources to them for resource sharing reasons, etc.  What's more, I can use a  container to load LXC, OpenVZ or Docker and not have to virtualize the hardware on which to run another container platform (I think I'm going to load Docker on Alpine, myself... surprised?)

I also like that SmartOS runs in memory off of a USB drive, which makes it snappy, easy to upgrade, and not use up any of my local storage space (one of the things I really like about ESXi)

So anyway, SmartOS is pretty awesome if you've ever read anything about it.  In fact, if you're here I'm sure you've heard of it before.  There's not a whole lot of documentation about it on the web though, or real-life experiences, hence my decision to write up this basic experience migrating from ESXi to SmartOS and all the steps I had to go through to get there. 

Here's what I did:

I had two VMs I really use a lot that I want to keep and don't want to have to re-build (at least not at first). They are:

Debian 8 64-bit (my central Dropbox machine that automatically polls and downloads files from an FTP server for work)
XPEnology 5.622 (basically a Synology Diskstation on different hardware that I use for all my home cloud goodness)

To start, I followed this little guide here:

https://wiki.smartos.org/display/DOC/Migrating+from+ESXi+4.x

First, download a copy of ovftool from the VMWare website - it's available for Linux, Mac and Windows. Install it once you've downloaded it.  I'm using Debian 8 (Stretch), so I:

$ chmod +x VMware-ovftool-4.0.0-2301625-lin.x86_64.bundle

$ sudo ./VMware-ovftool-4.0.0-2301625-lin.x86_64.bundle

and follow the helpful prompts.  

Then, make sure if you do this you have plenty of room left on your current machine:

$ df -h 

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           788M  9.4M  778M   2% /run
/dev/sdb5       126G   66G   54G  56% /
tmpfs           3.9G  326M  3.6G   9% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sdb2        96M   25M   72M  26% /boot/efi
tmpfs           788M  4.0K  788M   1% /run/user/117
tmpfs           788M   28K  788M   1% /run/user/1000
/dev/sdc1        15G   32K   15G   1% /media/avery/4321-9E9E


OK, looks like I've got plenty of room, so I invoke:

$ ovftool vi://yourESXiIPorFQDN/VMname BackupName.vmx

Enter login information for source vi:///
Username: ESXiUserName
Password:**************
Opening VI source: vi://ESXiUserName@:443/Debian
Opening VMX target: Debian.vmx
Writing VMX file: Debian.vmx
Transfer Completed                    
Completed successfully


(insert pitfalls here!) Please note, there are a few caveats: 

VMWare paravirtualized SCSI controller gives the following error:

Opening VMX target: Debian.vmx
Error:
 - Line 48: Unsupported virtual hardware device 'VirtualSCSI'.
 - The guest operating system 'debian8_64Guest' is not supported.
Completed with errors

I solved this by deleting my SCSI controller after assigning the hard disk to another SCSI controller (LSI parallel worked fine).  This can be really annoying because the attempted clone prevents the VM from being able to be edited.  I'm not sure if there's a better way to abort the migration process, but I started the VM and then powered it off and that got me back to a place where I could then edit it. 

Here's another thing - if you're using an unsupported version of Debian, like I was (Stretch isn't out yet), be sure to downgrade to Debian 7 64-bit in edit-->VM options-->General before you try and migrate using ovftool (v8 still works fine under v7).  Otherwise you'll end up with something like this: 

Opening VMX target: Debian.vmx
Error:
 - The guest operating system 'debian8_64Guest' is not supported.
Completed with errors

After you wait for a while for your VM's used hard drive space to get migrated to your local drive (vmdk file should be only used space and not entire allocated virtual space), you can grab qemu-img (which is readily available in Debian/Ubuntu apt repositories) and check out the specs of the VM.  This will be useful for creating your VM in SmartOS (you have to make a VM with the stated space of the VM to be able to migrate the VM - not the size of the resulting vmdk file). 

$  qemu-img info -f vmdk Debian-disk1.vmdk 
image: Debian-disk1.vmdk
file format: vmdk
virtual size: 30G (32212254720 bytes)
disk size: 20G
cluster_size: 65536
Format specific information:
    cid: 1391262983
    parent cid: 4294967295
    create type: monolithicSparse
    extents:
        [0]:
            virtual size: 32212254720
            filename: Debian-disk1.vmdk
            cluster size: 65536
            format: 

So now we're going to create a VM on SmartOS that's got a 30G vHD ... (part 2 coming soon!!)