Setting up a virtual development environment

I'm back in the LAMP development game, and for almost as long as I've been doing LAMP development, I've been setting up local development environments for my code. This way I can monkey with my code to my heart's desire, and if I break something, it doesn't affect the production site. (In theory, anyway). In the past, I've installed MySQL, Apache, and PHP/Perl directly on my workstation, and when a project has ended, I simply deleted the old database, Apache configs, etc.

Now, however, I've discovered virtualization and fallen in love. I have a Windows XP VM on my Linux desktop for the (rare) times I need a Windows box, and I've just created a VM to use as a development environment for the contract I'm currently working on.

To begin with, I created a base system using Debian Lenny (the current "stable" release) with qemu. This involved creating a disk image file 5 GB in size. That's awfully small these days, but this particular image file isn't going to have a lot of writing done to it. Instead I'll create a copy-on-write image that will have all the client-specific data in it. I do this by:

sudo kvm-img create /opt/qemu/lampsystem.img 5G

This creates a sparse 5G file in "raw" format.

Next, I want to install Debian on it:

sudo kvm -localtime -m 512 -cdrom debian-500-i386-businesscard.iso -boot d /opt/qemu/lampsystem.img

This mounts the Debian 5.0 network install ISO as a virtual CD-ROM and attaches it as the "D" drive (the tendrils of DOS are everywhere!) on the VM. Installing Debian is well-documented elsewhere, so I won't go into it here. I will note, however, that I'm installing everything in a single partition on the virtual disk because I don't see any point in setting up multiple partitions or LVM here.

After installing Debian, I'll install mysql-client, mysql-server, libapache2-mod-php5, openssh-server, and php5-cli. That should take care of the basic LAMP stuff. If I need Catalyst, Django, or something else, I'll tack those on later. I can always resize the root filesystem if I need to.

Note that after installing Debian, an automatic reboot of the VM won't work - it will reboot, but the boot process won't finish. Instead, you'll see this:


Booting from CD-Rom...
CDROM boot failure code : 0003
Boot from CD-Rom failed: could not read the boot disk
FATAL: No bootable device.


That's not a big deal. Just close qemu, then start it again. I'll use:

sudo kvm -localtime -net nic,vlan=0 -net tap,vlan=0,ifname=tap0 /opt/qemu/lampsystem.img

Which gives my VM an IP I can ping from my workstation. I've also got IP masquerading set up for the VM, but that's another post for another time. I'll also generate an SSH key and copy it over to the VM so that I can SSH in without a password.

After the base VM image is ready, it's time to create the copy-on-write image:

sudo kvm-img create -b /opt/qemu/lampsystem.img -f qcow2 /opt/qemu/client_a.qcow2

Now I have a copy-on-write image just for "Client A"'s files. I can boot it by:

sudo kvm -localtime -m 512 -net nic,vlan=0 -net tap,vlan=0,ifname=tap0 /opt/qemu/client_a.qcow2

Zimbra 5.0.13 and Samba

Zimbra have released version 5.0.13. I figured this was a good excuse to set up a test Zimbra server and try and get Samba set up to authenticate via Zimbra, rather than our current method (a separate LDAP server) at work. So to do this I'm setting up a VM running Ubuntu Server 8.04.2 (supported by Zimbra) on which I'll install ZCS 5.0.13, and a VM running Debian Etch and Samba. I'll be using the instructions from the Zimbra wiki for most of this.

My original thought was to set this up using a backup of our existing Zimbra configuration. However, I think that might overcomplicate things. For the time being, I'll just set it up as an entirely new installation, and once I have that going, I'll try getting a configuration set up from our existing setup.

So, to begin with, there's the Ubuntu VM. 64-bit virtual processor, 4G of RAM, 40G of hard disk. I expect that will be plenty of space. I'll install ZCS 5.0.13 via the quick install guide. Next, I'll install the "zimbra_posixaccount" and "zimbra_samba" extensions. However, rather than download the zip files to my desktop, unpack them, modify them, then upload them back to the server, I'll just unpack them and modify them on the server (in /root). Thus in /root/zimbra_posixaccount/config_template.xml, I'll set the ldapSuffix property to "dc=zcstest,dc=company,dc=com". The uidBase and gidBase properties are both set to 10000, which should be fine for my purposes. I'll then zip up all the files in /root/zimbra_posixaccount (excluding the directory itself) into /root/zimbra_posixaccount_company.zip. Similarly, I'll unpack /opt/zimbra/zimlets-admin-extra/zimbra_samba.zip into /root/zimbra_samba, modify config_template.xml the same way I did for zimbra_posixaccount, then pack up the files into /root/zimbra_samba.zip.

Now that I have the extensions configured, it's time to install them. I go to https://zcstest.company.com:7071/ (example URL only) and log in as "admin" with the admin password I set during installation. Interestingly enough, it tells me that my trial license expired 596 days ago. I'm not sure what to make of that, but that's a problem for another time. In the Zimbra Administration UI, I select "Admin Extensions" under "Configuration", then hit "Deploy" near the top of the window. And discover why the instructions said to do the file modifications on your desktop, as the "Deploy" dialog looks for a zip file on the user's desktop. Oops. No matter, though. I'll just copy the files off the server and upload them... no, that won't work either. I attempted to deploy "zimbra_posixaccount_company.zip", and got a message: "Failed to deploy the zimlet!". Ah, I see the problem. The name of the zip file is important. Renaming "zimbra_posixaccount_company.zip" to "zimbra_posixaccount.zip" did the trick. I'll deploy "zimbra_samba" similarly, and reload Zimbra Admin (i.e. hit the "refresh" button in my browser). This gives me two error dialogs:

Warning! Failed to create ou=groups,dc=zcstest,dc=stdbev,dc=com for Samba groups!
Warning! Failed to create ou=machines,dc=zcstest,dc=stdbev,dc=com for Samba machine accounts!

Perhaps this has something to do with my expired license file. I found another license file that's valid until August of this year; perhaps that will do the trick... no. Looking at the installation guide again, I see this bug. Per the bug report, I'll modify /opt/zimbra/bin/amavisdctl... no, that's not it. The file already contains the necessary LD_LIBRARY_PATH setting, which is to be expected, I suppose. The relevant error in /opt/zimbra/log/mailbox.log:

javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; remaining name 'dc=zcstest,dc=company,dc=com'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3010)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2931)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2737)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1808)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1731)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:257)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.searchDir(ZimbraLdapContext.java:551)
at com.zimbra.ldaputils.GetLDAPEntries.searchObjects(GetLDAPEntries.java:188)
at com.zimbra.ldaputils.GetLDAPEntries.searchObjects(GetLDAPEntries.java:138)
at com.zimbra.ldaputils.GetLDAPEntries.handle(GetLDAPEntries.java:87)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:429)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:286)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:160)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:190)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:716)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:406)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:350)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:313)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:844)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:644)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Just for fun, I'll snapshot the VM, uninstall the system libldap, and see where that gets me. It's a test machine, after all.

(time passes)

No, that didn't do a thing. Time to restore the snapshot and consult the Zimbra forums.

EDIT I found out my problem. My server name was "zcstest.company.corp", but I set the LDAP suffix to "dc=zcstest,dc=company,dc=com". So I had the wrong LDAP suffix. Oops.

Testing AoE over SCSI

In a previous post, I demonstrated that it was possible to serve an LVM logical volume over ATA-over-Ethernet. That particular LV was backed by a parallel ATA physical device. Now I want to see if ATA-over-Ethernet is a misnomer of sorts, and if it is in fact possible to serve an LV backed by a SCSI physical device. I suspect this to be the case.

The test system:

Intel(R) Pentium(R) 4 CPU 2.53GHz
1.0 GB RAM
Adaptec AIC-7892A U160/m (rev 02) SCSI controller
80G PATA hard disk (root file system)
5 Seagate ST373207LW 73GB U320 disks (Yes, I know I only have a U160 controller installed. I don't really care about the speed of the thing. It's a waltzing bear.)
SOYO mainboard

The SCSI disks will be configured as a RAID5 array using Linux software RAID.

mdadm --create /dev/md/0 -f --level=raid5 --raid-devices=5 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

As I am impatient:

echo 100000 > /proc/sys/dev/raid/speed_limit_max
echo 100000 > /proc/sys/dev/raid/speed_limit_min

Which brings my construction speed up to ~10,600 k/s. It will still take a good two hours to build, though.

Once the RAID has been constructed, I will then create a physical volume for LVM on it:

pvcreate /dev/md0

And create a volume group named "scsi":

vgcreate scsi /dev/md0

And create a logical volume named "raid":

lvcreate --name raid -l 70001 scsi

Note that the "scsi" VG had 70001 free extents.

Next, I'll create an ext3 filesystem on the LV:

mkfs.ext3 /dev/scsi/raid

And now, the fun part - sharing the LV via AoE:

vbladed 0 1 eth0 /dev/scsi/raid

Syslog on the test server shows that vbladed is running. Let's see what I get when I run aoe-discover and aoe-stat on my desktop (which is connected to the same network the AoE test server is on):

aoe-discover
aoe-stat

e0.1 293.605GB eth0 up

That looks pretty good, except that lvdisplay on the server says the LV is only 273.44 GB in size. However, 273.44 * 1024³ (273.44 gibibytes) == 293.604 gigabytes. So it's likely the disparity is just two different definitions of "GB". Still worth testing, though.

Anyway, now it's time to mount the AoE drive:

mount /dev/etherd/e0.1 /mnt/

No errors reported on either server. But here's the real test: 1 GiB of random data.


dd if=/dev/urandom of=/mnt/random bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 169.715 s, 6.3 MB/s


No errors. Next I'll take an MD5 sum of the file, unmount the AoE share, then mount the LV on the test server. In point of fact, I'm not required to unmount the AoE share first, but I want to ensure that there is no possible way both systems would be attempting to write to the test file. Sure enough, the MD5 sums are identical. This shows that you can, in fact, export any LVM logical volume via ATA over Ethernet, no matter what the physical medium backing it. It remains to be seen, however, if it is possible to export a single partition on a SCSI drive via AoE.