Proof-of-concept AoE on Linux

One thing I've been wanting to play with at work is ATA over Ethernet. Seems like a pretty neat trick - stick a bunch of drives in a box somewhere and mount them from somewhere else, like a VMWare image.

There's a nice article on Debian Administration on how to do this. I'll mostly be parroting that article, but I figure it might be useful to show how I did it.

I took an old P4 box that we had laying around with a 20G hard drive and put Debian 4.0 on it. I set up the disk using LVM2, because I was curious what would happen if I did that. My LVM setup:

5G root, formatted ext3
500M swap
12.92G "files", unformatted. I figure since I'll be using this thing as an AoE volume, I'll let the system that actually mounts this volume do the formatting. I don't think that's strictly necessary, though.

Also, per the article, I installed the "aoetools" and "vblade" packages. "aoetools" provides various useful tools for managing AoE volumes. "vblade" is described as a "virtual AoE blade emulator", which will allow me to export a local disk (or in this case, LV) over AoE. In this case, the command is:

vbladed 0 1 eth0 /dev/mapper/aoetest-files

And, sure enough, I see in my syslog:


... vbladed: ioctl returned 0
... vbladed: 13870562238 bytes
... vbladed: pid 2306: e0.1, 27090944 sectors


So now I need to access the AoE volume. Before I get into that, though, I'll note that both my AoE proof-of-concept machine and the machine I'll be mounting the AoE volume from have a dedicated network interface that I'll be using for AoE (connected via a crossover cable, in production I'd have a dedicated switch). I'd do similarly in production so that my AoE traffic wasn't sharing the network with regular network traffic. While I think this is good practice, it's not strictly necessary, and I'm pretty sure it's possible to run AoE over the regular network if you have to.

On my desktop (which I'll be using to mount the AoE volume) I've installed the "aoetools" package and loaded the AoE kernel module with modprobe aoe. Next I do aoe-discover, and I see:

"aoe-discover: /dev/etherd/discover does not exist or is not writeable."

Well, that's not good. What did I do wrong? Nothing, as it turns out. This is a bug in Ubuntu 8.10, and as yet, there has been no fix posted. But maybe I can fix it myself.

grep etherd /etc/udev/rules.d/* on the Debian box gives me:


/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="discover", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="err", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="interfaces", NAME="etherd/%k"
/etc/udev/rules.d/udev.rules:SUBSYSTEM=="aoe", KERNEL=="revalidate", NAME="etherd/%k"


The same command on my Ubuntu box gives me nothing. However, I can't just tack those lines on to /etc/udev/rules.d/udev.rules on my desktop, because apparently Ubuntu doesn't use that file. Instead I'll create a special file just for AoE, and I'll put it in /etc/udev/rules.d/25-aoe.rules. Restart udev, and viola! The devices are there!

Now, when I run "aoe-discover", I see nothing. That's OK. aoe-discover doesn't have any output. It's aoe-stat that will tell me what's there, and when I run that, I get:

e0.1 13.870GB eth1 up

Hooray! I create a filesystem with: mkfs.ext3 /dev/etherd/e0.1, then, as a test, create a 100M file: dd if=/dev/urandom of=/mnt/test1 bs=1M count=100. Takes 15.1 seconds. Creating a similar file locally? 14.8 seconds, so not too bad for speed. Of course, the two boxen are connected via a crossover - I might well see some slowdown using a switch.

So here we have it. A proof-of-concept Linux-based AoE appliance using commodity hardware. Since the AoE volume is an LVM logical volume on the appliance, you can use LVM tools to change the size of that LV, should you need to. I wouldn't recommend it, though.