Setting up a virtual development environment

I'm back in the LAMP development game, and for almost as long as I've been doing LAMP development, I've been setting up local development environments for my code. This way I can monkey with my code to my heart's desire, and if I break something, it doesn't affect the production site. (In theory, anyway). In the past, I've installed MySQL, Apache, and PHP/Perl directly on my workstation, and when a project has ended, I simply deleted the old database, Apache configs, etc.

Now, however, I've discovered virtualization and fallen in love. I have a Windows XP VM on my Linux desktop for the (rare) times I need a Windows box, and I've just created a VM to use as a development environment for the contract I'm currently working on.

To begin with, I created a base system using Debian Lenny (the current "stable" release) with qemu. This involved creating a disk image file 5 GB in size. That's awfully small these days, but this particular image file isn't going to have a lot of writing done to it. Instead I'll create a copy-on-write image that will have all the client-specific data in it. I do this by:

sudo kvm-img create /opt/qemu/lampsystem.img 5G

This creates a sparse 5G file in "raw" format.

Next, I want to install Debian on it:

sudo kvm -localtime -m 512 -cdrom debian-500-i386-businesscard.iso -boot d /opt/qemu/lampsystem.img

This mounts the Debian 5.0 network install ISO as a virtual CD-ROM and attaches it as the "D" drive (the tendrils of DOS are everywhere!) on the VM. Installing Debian is well-documented elsewhere, so I won't go into it here. I will note, however, that I'm installing everything in a single partition on the virtual disk because I don't see any point in setting up multiple partitions or LVM here.

After installing Debian, I'll install mysql-client, mysql-server, libapache2-mod-php5, openssh-server, and php5-cli. That should take care of the basic LAMP stuff. If I need Catalyst, Django, or something else, I'll tack those on later. I can always resize the root filesystem if I need to.

Note that after installing Debian, an automatic reboot of the VM won't work - it will reboot, but the boot process won't finish. Instead, you'll see this:


Booting from CD-Rom...
CDROM boot failure code : 0003
Boot from CD-Rom failed: could not read the boot disk
FATAL: No bootable device.


That's not a big deal. Just close qemu, then start it again. I'll use:

sudo kvm -localtime -net nic,vlan=0 -net tap,vlan=0,ifname=tap0 /opt/qemu/lampsystem.img

Which gives my VM an IP I can ping from my workstation. I've also got IP masquerading set up for the VM, but that's another post for another time. I'll also generate an SSH key and copy it over to the VM so that I can SSH in without a password.

After the base VM image is ready, it's time to create the copy-on-write image:

sudo kvm-img create -b /opt/qemu/lampsystem.img -f qcow2 /opt/qemu/client_a.qcow2

Now I have a copy-on-write image just for "Client A"'s files. I can boot it by:

sudo kvm -localtime -m 512 -net nic,vlan=0 -net tap,vlan=0,ifname=tap0 /opt/qemu/client_a.qcow2

Zimbra 5.0.13 and Samba

Zimbra have released version 5.0.13. I figured this was a good excuse to set up a test Zimbra server and try and get Samba set up to authenticate via Zimbra, rather than our current method (a separate LDAP server) at work. So to do this I'm setting up a VM running Ubuntu Server 8.04.2 (supported by Zimbra) on which I'll install ZCS 5.0.13, and a VM running Debian Etch and Samba. I'll be using the instructions from the Zimbra wiki for most of this.

My original thought was to set this up using a backup of our existing Zimbra configuration. However, I think that might overcomplicate things. For the time being, I'll just set it up as an entirely new installation, and once I have that going, I'll try getting a configuration set up from our existing setup.

So, to begin with, there's the Ubuntu VM. 64-bit virtual processor, 4G of RAM, 40G of hard disk. I expect that will be plenty of space. I'll install ZCS 5.0.13 via the quick install guide. Next, I'll install the "zimbra_posixaccount" and "zimbra_samba" extensions. However, rather than download the zip files to my desktop, unpack them, modify them, then upload them back to the server, I'll just unpack them and modify them on the server (in /root). Thus in /root/zimbra_posixaccount/config_template.xml, I'll set the ldapSuffix property to "dc=zcstest,dc=company,dc=com". The uidBase and gidBase properties are both set to 10000, which should be fine for my purposes. I'll then zip up all the files in /root/zimbra_posixaccount (excluding the directory itself) into /root/zimbra_posixaccount_company.zip. Similarly, I'll unpack /opt/zimbra/zimlets-admin-extra/zimbra_samba.zip into /root/zimbra_samba, modify config_template.xml the same way I did for zimbra_posixaccount, then pack up the files into /root/zimbra_samba.zip.

Now that I have the extensions configured, it's time to install them. I go to https://zcstest.company.com:7071/ (example URL only) and log in as "admin" with the admin password I set during installation. Interestingly enough, it tells me that my trial license expired 596 days ago. I'm not sure what to make of that, but that's a problem for another time. In the Zimbra Administration UI, I select "Admin Extensions" under "Configuration", then hit "Deploy" near the top of the window. And discover why the instructions said to do the file modifications on your desktop, as the "Deploy" dialog looks for a zip file on the user's desktop. Oops. No matter, though. I'll just copy the files off the server and upload them... no, that won't work either. I attempted to deploy "zimbra_posixaccount_company.zip", and got a message: "Failed to deploy the zimlet!". Ah, I see the problem. The name of the zip file is important. Renaming "zimbra_posixaccount_company.zip" to "zimbra_posixaccount.zip" did the trick. I'll deploy "zimbra_samba" similarly, and reload Zimbra Admin (i.e. hit the "refresh" button in my browser). This gives me two error dialogs:

Warning! Failed to create ou=groups,dc=zcstest,dc=stdbev,dc=com for Samba groups!
Warning! Failed to create ou=machines,dc=zcstest,dc=stdbev,dc=com for Samba machine accounts!

Perhaps this has something to do with my expired license file. I found another license file that's valid until August of this year; perhaps that will do the trick... no. Looking at the installation guide again, I see this bug. Per the bug report, I'll modify /opt/zimbra/bin/amavisdctl... no, that's not it. The file already contains the necessary LD_LIBRARY_PATH setting, which is to be expected, I suppose. The relevant error in /opt/zimbra/log/mailbox.log:

javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; remaining name 'dc=zcstest,dc=company,dc=com'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3010)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2931)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2737)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1808)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1731)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:257)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.searchDir(ZimbraLdapContext.java:551)
at com.zimbra.ldaputils.GetLDAPEntries.searchObjects(GetLDAPEntries.java:188)
at com.zimbra.ldaputils.GetLDAPEntries.searchObjects(GetLDAPEntries.java:138)
at com.zimbra.ldaputils.GetLDAPEntries.handle(GetLDAPEntries.java:87)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:429)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:286)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:160)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:190)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:716)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:406)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:350)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:313)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:844)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:644)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Just for fun, I'll snapshot the VM, uninstall the system libldap, and see where that gets me. It's a test machine, after all.

(time passes)

No, that didn't do a thing. Time to restore the snapshot and consult the Zimbra forums.

EDIT I found out my problem. My server name was "zcstest.company.corp", but I set the LDAP suffix to "dc=zcstest,dc=company,dc=com". So I had the wrong LDAP suffix. Oops.

Testing AoE over SCSI

In a previous post, I demonstrated that it was possible to serve an LVM logical volume over ATA-over-Ethernet. That particular LV was backed by a parallel ATA physical device. Now I want to see if ATA-over-Ethernet is a misnomer of sorts, and if it is in fact possible to serve an LV backed by a SCSI physical device. I suspect this to be the case.

The test system:

Intel(R) Pentium(R) 4 CPU 2.53GHz
1.0 GB RAM
Adaptec AIC-7892A U160/m (rev 02) SCSI controller
80G PATA hard disk (root file system)
5 Seagate ST373207LW 73GB U320 disks (Yes, I know I only have a U160 controller installed. I don't really care about the speed of the thing. It's a waltzing bear.)
SOYO mainboard

The SCSI disks will be configured as a RAID5 array using Linux software RAID.

mdadm --create /dev/md/0 -f --level=raid5 --raid-devices=5 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

As I am impatient:

echo 100000 > /proc/sys/dev/raid/speed_limit_max
echo 100000 > /proc/sys/dev/raid/speed_limit_min

Which brings my construction speed up to ~10,600 k/s. It will still take a good two hours to build, though.

Once the RAID has been constructed, I will then create a physical volume for LVM on it:

pvcreate /dev/md0

And create a volume group named "scsi":

vgcreate scsi /dev/md0

And create a logical volume named "raid":

lvcreate --name raid -l 70001 scsi

Note that the "scsi" VG had 70001 free extents.

Next, I'll create an ext3 filesystem on the LV:

mkfs.ext3 /dev/scsi/raid

And now, the fun part - sharing the LV via AoE:

vbladed 0 1 eth0 /dev/scsi/raid

Syslog on the test server shows that vbladed is running. Let's see what I get when I run aoe-discover and aoe-stat on my desktop (which is connected to the same network the AoE test server is on):

aoe-discover
aoe-stat

e0.1 293.605GB eth0 up

That looks pretty good, except that lvdisplay on the server says the LV is only 273.44 GB in size. However, 273.44 * 1024³ (273.44 gibibytes) == 293.604 gigabytes. So it's likely the disparity is just two different definitions of "GB". Still worth testing, though.

Anyway, now it's time to mount the AoE drive:

mount /dev/etherd/e0.1 /mnt/

No errors reported on either server. But here's the real test: 1 GiB of random data.


dd if=/dev/urandom of=/mnt/random bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 169.715 s, 6.3 MB/s


No errors. Next I'll take an MD5 sum of the file, unmount the AoE share, then mount the LV on the test server. In point of fact, I'm not required to unmount the AoE share first, but I want to ensure that there is no possible way both systems would be attempting to write to the test file. Sure enough, the MD5 sums are identical. This shows that you can, in fact, export any LVM logical volume via ATA over Ethernet, no matter what the physical medium backing it. It remains to be seen, however, if it is possible to export a single partition on a SCSI drive via AoE.

Setting up Zimbra BES Connector

My organization has several employees with BlackBerry devices, and as we run Zimbra for our mail server, we'd like those employees to be able to sync their BlackBerries with our mail server. To do this requires two things: BlackBerry Enterprise Server ("BES" for short) and Zimbra BES Connector ("ZBC" for shoft).

Zimbra BES Connector is designed to run on the same logical machine as BlackBerry Enterprise Server, and as it turns out, ZBC's requirements (as stated in the ZBC Admin Guide) are considerably more strict than BES. To wit:

• ZCS 5.0.7 or later
  
• Microsoft Windows Server 2003 operating system
  
• Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1 (MAPI/CDO)
  
• BlackBerry Enterprise Server 4.1 or BlackBerry Express 4.1
  
• Zimbra Collaboration Suite Connector for BlackBerry Enterprise Server .msi executable file
  
• Outlook® 2007 messaging and collaboration client
  

BES's requirements:

• Minimum requirements for up to 500 users: Intel® Pentium® IV processor (2GHz or greater), 1.5 GB RAM, MSDE 2000 or higher, 20 GB disk space in addition to Windows® requirements


• Windows 2000 Server (Server or Advanced Server Editions) with Service Pack 4 or Windows Server 2003 with Service Pack 1


• Integrates with Microsoft® Exchange 5.5 (Service Pack 4 or later), Microsoft Exchange 2000 (Service Pack 2 or later) or Microsoft Exchange 2003 mail server environments (mixed mode or native installation).


• Microsoft Exchange 5.5 Administrator, Microsoft Exchange 2000 System Manager or Microsoft Exchange 2003 System Manager, Microsoft Exchange 2007 MAPI Client


• Microsoft Internet Explorer® version 6.0 or later

I'm ignoring the Exchange server requirements per the ZBC documentation, but everything else still stands.

Having recently become very fond of setting up things on virtual machines, and since this installation is still in the testing stage right now, I'll set this up on a VM with:

• Windows Server 2003


• 1.5 GB RAM


• Outlook 2007


• MAPI/CDO 1.2.1


• Internet Explorer 7


• BlackBerry Enterprise Server 4.1 for Exchange (60-day, 20-user trial version)


• 30GB HDD

Note that you must set up a separate user account (see here for details) that BES and associated services will run under. It will not work if you simply install BES as Administrator and set all the services to run as system services. I learned this the hard way. :(

Once the VM is set up, the next step is to install the Zimbra BES Connector by means of the MSI file. This is a quick matter. Now I'll create an Administrator account for BES on Zimbra using the Admin Console. If you're following along, feel free to choose a username and password to your liking; I recommend diceware.com and a bunch of D6's for password generation.

The next step is to make sure that the Zimbra server's mail port has SSL enabled. To allow both SSL and non-SSL, execute (on the mail server, as the Zimbra user) zmtlsctl both. To allow only SSL, execute zmtlsctl https.

Now I'll delete and recreate the BES mail profiles "BlackBerryServer" and "BlackBerryManager". To do this, I go to the Start Menu on the BES server, then choose Control Panel, then Mail. This brings up the BlackBerryServer Mail Setup dialog. Here, I choose "Show Profiles".

This dialog only shows the BlackBerryServer profile. No big deal - if the other profile was there, I'd delete it anyway, so this just saves me a step. After deleting the BlackBerryServer profile, I hit "Add" to create a new profile, and I get dialog titled "Add New E-mail Account". This dialog is trying to set up an email account for me automatically, but I don't want that, so I click "Manually configure server settings or additional server types" at the bottom and hit Next. From here, I select "Other", which allows me to select "Zimbra Collaboration Server" from the box below. I hit Next, and am presented with a "Zimbra Server Configuration Settings" dialog. I put in the name of my mail server and the port that the Zimbra admin service is listening on ¹, check "use secure connection", and put in the BES Administrator account credentials that I set up in the Zimbra Admin Console. After that, I hit "OK". I follow the same process to create the BlackBerryServer profile.

Now I can start the BlackBerry Controller service, which should start any other services that BES requires².

¹ZCB connects to the Zimbra administrative interface, so in addition to putting in the name of your Zimbra server, you'll have to put the port (typically 7071) in as well, e.g. "mail.example.com:7071" (source)

²The first time I did this, I got an error dialog: "Error 1069: The service did not start due to a logon Failure." Turns out this was not an error in my BES configuration, but the Windows service configuration (details). I switched the service from running as the Administrator account to the system account, and after that was able to start the service successfully.

From here, the connector is set up. I can see the users on my mail server in the Global Address List. I haven't figured out how to provision anyone in BES yet, but that's a BES issue, not a Zimbra issue.

Setting up an LDAP replica server using Zimbra

We run Zimbra at work, as well as an OpenLDAP server. It occurred to us that it would be great if we could standardize on a single LDAP solution and reduce our administrative overhead. Since Zimbra has such nice management tools, I want to go with Zimbra. So my plan is to build an LDAP replication server that will initially be slaved off of the main Zimbra mail server, but eventually will be the LDAP master that the mail server is slaved to. It occurs to me that one thing I'll have to do some digging into is using that LDAP server with Samba (which we use for a lot of file sharing) - I don't know if Zimbra's LDAP setup has the Samba stuff in it by default or not.

The official documentation for this process is in the Zimbra Connection Suite Multi-Server Installation Guide, specifically the Configuring LDAP Replication section. There are also instructions in the LDAP topic of the Zimbra wiki, but the instructions in the official docs are more detailed.

To begin with, I went over to the mail server and enabled replication:


ssh mail.company.com
su -
su - zimbra
/opt/zimbra/libexec/zmldapenablereplica


Output from zmldapenablereplica:


Enabling sync provider...succeeded
Stopping LDAP on mail.company.com...done
Starting LDAP on mail.company.com...done


I then built a VM using Ubuntu Server 8.04 LTS, one of the distributions supported by Zimbra. I then downloaded (download location) the proper version of the Zimbra Network Edition installer, unpacked the installation files into /tmp, and started the installer:


cd /tmp
tar xzf /home/kit/zcs-NETWORK-5.0.11_GA_2695.UBUNTU8_64.20081117023527.tgz
cd zcs-NETWORK-5.0.11_GA_2695.UBUNTU8_64.20081117023527
./install.sh


It informed me that I had to fiddle /etc/hosts (see this howto for details). I did so, and re-ran the installer. This time, I got:


Operations logged to /tmp/install.log.26576
Checking for existing installation...
zimbra-ldap...NOT FOUND
zimbra-logger...NOT FOUND
zimbra-mta...NOT FOUND
zimbra-snmp...NOT FOUND
zimbra-store...NOT FOUND
zimbra-apache...NOT FOUND
zimbra-spell...NOT FOUND
zimbra-proxy...NOT FOUND
zimbra-archiving...NOT FOUND
zimbra-convertd...NOT FOUND
zimbra-cluster...NOT FOUND
zimbra-core...NOT FOUND


PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.
ZIMBRA, INC. ("ZIMBRA") WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU
FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING
THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY
THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS
AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.

License Terms for the Zimbra Collaboration Suite:
http://www.zimbra.com/license/zimbra_network_eval_license.pdf


Press Return to continue

Checking for prerequisites...
FOUND: NPTL
FOUND: sudo-1.6.9p10-1ubuntu3.3
FOUND: libidn11-1.1-1
MISSING: libpcre3
MISSING: libgmp3c2
FOUND: libexpat1-2.0.1-0ubuntu1
FOUND: libstdc++6-4.2.3-2ubuntu7
MISSING: libstdc++5
MISSING: libltdl3
Checking for suggested prerequisites...
FOUND: perl-5.8.8

###ERROR###

One or more prerequisite packages are missing.
Please install them before running this installer.

Installation cancelled.


Easy enough to fix, just install those packages. The installer is nice enough to use Ubuntu's package names, so with a simple cut-and-paste, I can do:


apt-get install libpcre3 libgmp3c2 libstdc++5 libltdl3


Now, for the third time, I run ./install.sh, and this time it starts the installation. As the documentation suggests, I install only the zimbra-core and zimbra-ldap packages, and am presented with:


Main menu

1) Common Configuration:
2) zimbra-ldap: Enabled
3) Enable default backup schedule: yes
r) Start servers after configuration yes
s) Save config to file
x) Expand menu
q) Quit

*** CONFIGURATION COMPLETE - press 'a' to apply
Select from menu, or press 'a' to apply config (? - help)


As per the instructions, I go into "Common Configuration":


Common configuration

1) Hostname: hostname.company.com
2) Ldap master host: hostname.company.com
3) Ldap port: 389
4) Ldap Admin password: set
5) Require secure interprocess communications: yes
6) TimeZone: (GMT-08.00) Pacific Time (US & Canada)

Select, or 'r' for previous menu [r]


I choose option 2 here and set my LDAP master host to "mail.company.com". I then choose option 4 to set the LDAP Admin password to the Zimbra LDAP password set on mail.company.com (find this by executing "zmlocalconfig -s zimbra_ldap_password" as the Zimbra user on the master LDAP server) and then choosing "r" to return to the previous menu.

From the main menu, I choose option 2, "zimbra-ldap":


Ldap configuration

1) Status: Enabled
2) Create Domain: yes
3) Domain to create: hostname.company.com
4) Ldap Root password: set
5) Ldap Replication password: set
6) Ldap Postfix password: set
7) Ldap Amavis password: set
8) Ldap Nginx password: set

Select, or 'r' for previous menu [r]


Here, I choose option 2 to set "Create Domain" to "no", then set the LDAP replication password to the LDAP replication password on the mail server (find this by executing "zmlocalconfig -s ldap_replication_password" as the Zimbra user on the master LDAP server). I then choose "r" to return to the main menu, and "a" to apply my changes.

Now, all that remains is to test the setup. As the Zimbra user, I execute "zmprov gaa", which displays all the accounts set up on the mail server. But, just to be sure, I create a user on the mail server:


zmprov ca foo.mcbarson@company.com password


Now, when I run "zmprov gaa" on the replication server, I see "foo.mcbarson@company.com" at the bottom of the list. LDAP replication is a success. But that's only half of what needs to be done. I still need to set up the mail server to replicate from the LDAP server I've just set up. However, as it's nearly time to go home, I think I'll save that for another day.

Proof-of-concept AoE on Linux

One thing I've been wanting to play with at work is ATA over Ethernet. Seems like a pretty neat trick - stick a bunch of drives in a box somewhere and mount them from somewhere else, like a VMWare image.

There's a nice article on Debian Administration on how to do this. I'll mostly be parroting that article, but I figure it might be useful to show how I did it.

I took an old P4 box that we had laying around with a 20G hard drive and put Debian 4.0 on it. I set up the disk using LVM2, because I was curious what would happen if I did that. My LVM setup:

5G root, formatted ext3
500M swap
12.92G "files", unformatted. I figure since I'll be using this thing as an AoE volume, I'll let the system that actually mounts this volume do the formatting. I don't think that's strictly necessary, though.

Also, per the article, I installed the "aoetools" and "vblade" packages. "aoetools" provides various useful tools for managing AoE volumes. "vblade" is described as a "virtual AoE blade emulator", which will allow me to export a local disk (or in this case, LV) over AoE. In this case, the command is:

vbladed 0 1 eth0 /dev/mapper/aoetest-files

And, sure enough, I see in my syslog:


... vbladed: ioctl returned 0
... vbladed: 13870562238 bytes
... vbladed: pid 2306: e0.1, 27090944 sectors


So now I need to access the AoE volume. Before I get into that, though, I'll note that both my AoE proof-of-concept machine and the machine I'll be mounting the AoE volume from have a dedicated network interface that I'll be using for AoE (connected via a crossover cable, in production I'd have a dedicated switch). I'd do similarly in production so that my AoE traffic wasn't sharing the network with regular network traffic. While I think this is good practice, it's not strictly necessary, and I'm pretty sure it's possible to run AoE over the regular network if you have to.

On my desktop (which I'll be using to mount the AoE volume) I've installed the "aoetools" package and loaded the AoE kernel module with modprobe aoe. Next I do aoe-discover, and I see:

"aoe-discover: /dev/etherd/discover does not exist or is not writeable."

Well, that's not good. What did I do wrong? Nothing, as it turns out. This is a bug in Ubuntu 8.10, and as yet, there has been no fix posted. But maybe I can fix it myself.

grep etherd /etc/udev/rules.d/* on the Debian box gives me:


/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="discover", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="err", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="interfaces", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="revalidate", NAME="etherd/%k"


The same command on my Ubuntu box gives me nothing. However, I can't just tack those lines on to /etc/udev/rules.d/udev.rules on my desktop, because apparently Ubuntu doesn't use that file. Instead I'll create a special file just for AoE, and I'll put it in /etc/udev/rules.d/25-aoe.rules. Restart udev, and viola! The devices are there!

Now, when I run "aoe-discover", I see nothing. That's OK. aoe-discover doesn't have any output. It's aoe-stat that will tell me what's there, and when I run that, I get:

e0.1 13.870GB eth1 up

Hooray! I create a filesystem with: mkfs.ext3 /dev/etherd/e0.1, then, as a test, create a 100M file: dd if=/dev/urandom of=/mnt/test1 bs=1M count=100. Takes 15.1 seconds. Creating a similar file locally? 14.8 seconds, so not too bad for speed. Of course, the two boxen are connected via a crossover - I might well see some slowdown using a switch.

So here we have it. A proof-of-concept Linux-based AoE appliance using commodity hardware. Since the AoE volume is an LVM logical volume on the appliance, you can use LVM tools to change the size of that LV, should you need to. I wouldn't recommend it, though.

Setting up LVM on an already-setup box

I have this box at work that someone else was nice enough to set up with Debian Lenny and a great big honkin' RAID5 array. We've already got the basic filesystem structure set up on the box, but we'd like to add the RAID as well.

I'm going to set up the RAID as its own volume group under LVM. This allows us, should the OS drive fail, to slap another drive with an OS on it into the machine, boot, and remount the RAID. It also allows us to dynamically add storage, say some sort of SAN, to the physical volume, then resize the logical volumes on the fly. It also simplifies things a little by keeping the OS volume group and the file storage volume group separate.

Being an LVM newbie, I'll be referencing A simple introduction to working with LVM and The LVM HOWTO. You can assume that most any LVM-specific command syntax I pulled out of one of those two sources.

Now, the first thing to do is set up a partition on the RAID array, as LVM runs on top of physical partitions. I do this with fdisk, because that's the way I learned it. :) If I were a bit more clever, or if I felt like it, I'd do this with a single call to sfdisk.

Next I create a physical volume for the RAID: pvcreate /dev/sda1, and then a volume group: vgcreate file-storage /dev/sda1. Checking my work with vgscan, I see:


Reading all physical volumes. This may take a while...
Found volume group "os" using metadata type lvm2
Found volume group "file-storage" using metadata type lvm2


Now I want to create a logical volume (LV) that encompasses the entire volume group. I do this by first examining the output of vgdisplay, where I see the line: "Free PE / Size 357375 / 1.36 TB" (I told you it was a big honkin' RAID. Also note here that "PE" means "Physical Extent", the size of one quantum of storage in LVM. One PE is exactly the same size as one LE, so here one LE is about 4 MB). I will thus create an LV of 357375 extents, lvcreate -n files file-storage -l 357375. With this done, it's time to format the LV.

After consulting with my colleagues, I've decided to use ext4 for the filesystem on the RAID. I like what I read about ext4, both on Wikipedia and from IBM, and as this box is slated to become a backup server, it seems like a good place to play with it. Before I begin, though, I'll update the kernel to the latest version in (Debian) testing: 2.6.26-1, so as to have the latest ext4 fixes that have been included in Debian kernels. Even with that, though, I'll want to add "nodealloc" to the line in my fstab for the RAID:

It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using tune2fs -E test_fs -o extents /dev/DEV, you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)

At any rate, per the ext4 HOWTO, I'll create an ext4 filesystem on the "files" LV with: mke2fs -t ext4dev /dev/file-storage/files. And wait. And wait. And wait some more, because 1.36 TiB is a lot of space.

From here, the RAID is like any other filesystem. Pick a mount point, make sure to mount it "-o nodelalloc", and off you go.

CORRECTION: Debian kernel 2.6.26-1 does not support the "nodelalloc" mount option. I ended up installing kernel 2.6.27.7 from http://kernel.org/. As the "nodelalloc" option was only recommended for 2.6.26-based kernels, I am no longer mounting the ext4 filesystem with the "nodelalloc" option.