Log in

Previous Entry | Next Entry

Low power liliputing...

We've been thinking a lot about power consumption, and it turns out one of our big power drains is a server I have sitting in the living room. It's a fairly ancient amd64 machine, and it's been chugging along like a champ for years, running various versions of Ubuntu.

The problem is that it runs pretty warm, and the processor is a typical Intel instruction set processor, meaning that it's pretty inefficient in terms of power. Plus, it's quite noisy. So my goal for a while has been to replace it with something that consumes less power. Recently, I bought a Dreamplug, which is a computer about the size of a large power supply brick with an ARM cpu and a claimed maximum draw of 15 watts.

I spent some time this weekend configuring the dreamplug. I wanted to run Ubuntu on it, but for some reason Ubuntu has dropped support for kirkwood-based ARM CPUs. So I wound up installing gentoo on it instead. This wasn't trivial, and it isn't done, but it was totally do-able, and I wanted to write down what I'd done here so that people could benefit from it.

The first thing to say is that plug computers typically use flash memory for their root filesystem. I have no particular need to be constrained by flash memory's restricted space, so I bought a 2.5" 1 terabyte USB drive to use as my system disk. This makes things a lot easier: support for the NOR flash in the Dreamplug is somewhat problematic in the Linux base distribution, although I am confident that it can be made to work if you care enough, and I may get around to that at some point.

One problem with the 1T drive is that it wouldn't run off the power supplied by the Dreamplug's USB ports. I had to hook up a powered USB hub to make it work. I don't know if a smaller drive with a lower power draw would have worked; it'd probably be worth a try, though.

Another thing to mention is that this system is supported only to a limited degree—it's really more of an eval board than a real product. This is unfortunate: an open source ARM server board would be a very good thing to have, and it would be nice if it were really supported. Personally I'd like to see it made with one of the very nice chipsets from TI or nVidia, rather than the Marvell CPU. I chose this particular system over a Beagle Board because the Beagle Board doesn't have a very good answer for how to be a WiFi base station; the Dreamplug does. Similarly, performance of WiFi drivers on the Gumstix Overo is reputed to be poor. Of course, I don't have any experience with the WiFi on the Dreamplug either, so it may also be poor; time will tell.

The support that's available for the dreamplug currently includes a somewhat outdated kernel source tree that's been hacked to support the dreamplug. This is pretty cool, but I really don't want to use a hacked tree from a google code project. I want a mainline kernel. Otherwise, this is kind of an academic exercise: sure, it works, but where's your change control? How do you update when there's a zero-day hack in the wild? So I have the hacked kernel, and I might use some of the hacks, but I want as minimal a changeset as possible; currently I'm investigating how much of what's in the hacked kernel is in the head of the Linux source tree.

Having said that, the kernel that comes with gentoo works, so I started with that instead of the "official" dreamplug kernel that's in Google Code. The Gentoo install instructions for the SheevaPlug work just fine for the Dreamplug as well. The install root is slightly broken; they managed not to set up the inittab correctly, so the process group for the install shell doesn't have the console as its controlling terminal, meaning that when you hit Control/C, nothing happens. This complicated the install because whenever I accidentally did something (e.g., ping) that doesn't exit, I had to power cycle the computer to get back control. I didn't feel that this was worth the couple of hours it would have taken me to fix it (I'm not an expert on generating install uImages). The easiest way around this is to configure an IP address, set a root password, and then ssh in. It's nice that the install uImage includes an ssh server.

Where I diverged from the Gentoo instructions for Sheevaplug is that I attached my 1T external USB drive instead of installing to a flash drive. This showed up in the install kernel as /dev/sdd; however, the custom kernel I built according to the instructions did not support nor flash, so after disabling support for MTD devices (which was necessary to get the kernel to boot at all), the 1T drive showed up as /dev/sdc. I made three partitions: a 32M vfat partition, a 512M swap partition, and used the rest for an ext3 partition. My fstab looks like this:

/dev/sdc1		/boot		vfat		noauto,noatime	1 2
/dev/sdc3		/		ext3		noatime		0 1
/dev/sdc2		none		swap		sw		0 0
shm			/dev/shm	tmpfs		nodev,nosuid,noexec	0 0

You have to have fstab set up correctly before you can boot. It's probably better to use the disk's UUID instead of its device name, but I haven't done that yet. You can get the uuid with the blkid command as long as the disk's not mounted:

blkid /dev/sda3

If it's mounted, you can get the UUID thusly:

ls -l /dev/disk/by-uuid

This is what a UUID-based fstab entry ought to look like:

UUID=6b640dda-a5aa-432f-af8f-bf31de99d92 / ext3 noatime 0 1

To drop back a step, the gentoo install instructions have you load the kernel and ramdisk using TFTP. I am doing all this from my Macbook, so it was easiest to set up tftp on my Macbook. I did this by typing (as root):

launchctl load -w /System/Library/LaunchDaemons/tftp.plist

I then copied the two images into /private/tftpboot. This is probably the easiest time I've ever had using tftp to load images into a device.

One other thing: in order to do the install, you need access to a console. This means that you need to buy the jtag board that is available with the Dreamplug. It's better to buy it when you buy the Dreamplug; you get a discount, and you only have to pay to ship one box. They charge a lot for shipping. You can probably guess how I know this. I'm currently using my other linux box to run the console, but it should work with MacOS as well. I use screen to connect to the console: screen /dev/ttyUSB00 115200. I've found that the console doesn't work reliably at that speed, so I slow it down to 19200 baud, but uboot doesn't honor the baud rate setting when it starts, so I have to switch back and forth between 19200 and 115200 when I'm trying to access the uboot console.

Make sure you set a root password.

Anyway, that's it for now. I'm sure there's more excitement coming...


( 15 comments — Leave a comment )
May. 12th, 2011 06:13 pm (UTC)
Why dreamplug
Can you say why you chose dreamplug over other similar servers? And why you are running Linux on it rather than, say, *BSD?
May. 12th, 2011 06:32 pm (UTC)
Re: Why dreamplug
I've been thinking of installing NetBSD on it, actually. My main reason for not oing so is that they don't have support for recoverable filesystems, and when your filesystem is 1T, that's a real drag: checking the filesystem after a crash can take a very long time. I'm pretty sure FreeBSD has the same problem, plus, I prefer the NetBSD userland to the FreeBSD userland.

As for why the Dreamplug, I haven't been able to identify a better alternative. The Gumstix devices are too slow; the BeagleBoard uses USB as its sole I/O bus, and is really more aimed at being a workstation than a server—lots of video hardware, crappy I/O.

Do you know of an ARM board that's a better choice?
May. 12th, 2011 06:53 pm (UTC)
Re: Why dreamplug
What is a 'recoverable filesystem' ? FreeBSD has had softupdates (on by default, I think) for some years, which gives you filesystem consistency guarantees, and also background fsck on boot. We have some Quite Large filesystems on some of our FreeBSD boxes, and they come right up out of power outages. I don't know enough about the other BSDs to comment on pros and cons, except that FreeBSD is my native tongue so is preferable to me.
I also don't have much of a clue about low-power systems, except for knowing that there are quite a few different ones, including others from the same manufacturer; I was curious about why you picked this one.
May. 12th, 2011 06:57 pm (UTC)
Re: Why dreamplug
(e.g. www.tonidoplug.com)
May. 12th, 2011 07:05 pm (UTC)
Re: Why dreamplug
The tonidoplug has no built-in WiFi, and only one ethernet port. It's not clear how you get a console on it, and it appears not to be an open source product. Most ARM solutions are like that--there are a lot of interesting ARM-based server products, but the software isn't particularly open or hackable; to me, that's an absolute requirement.
May. 12th, 2011 07:23 pm (UTC)
Re: Why dreamplug
I thought tonidoplug was Ubuntu Jaunty. As I say, I don't know much about the subject.
May. 12th, 2011 07:51 pm (UTC)
Re: Why dreamplug
They aren't bragging about it, if so. In any case, jaunty is so hopelessly ancient as to be useless. In particular, IPv6 support is quite antiquated. This is why I had to go to gentoo.
May. 12th, 2011 07:08 pm (UTC)
Re: Why dreamplug
A recoverable filesystem is a filesystem with a journal that can be played back in order to restore it to a consistent state. NetBSD also has update ordering, but this still requires post-crash cleanup.

I've been looking into ARM servers for a long time--a couple of years. The Dreamplug was literally the first one to trigger my "okay, I can buy this" response. I really, really wanted to use a Gumstix, but the crappy WiFi performance was a dealbreaker. I've been tracking the Beagleboard for a long time too, and it's improved in a variety of ways since it first came out. But it still lacks usable I/O. The Dreamplug has an ESATA port, which I intend to eventually use. Right now I'm just using USB, because I don't have an ESATA cable, but that's not my long-term plan.
May. 12th, 2011 07:27 pm (UTC)
Re: Why dreamplug
You said "checking the filesystem after a crash can take a very long time". I'm reporting that this is not in fact a problem with FreeBSD. As I recall, softupdates is not the same as update ordering (although I'm rusty on this: it was interesting before we decided to upgrade to softupdates, then got very boring afterwards). And if you want journaling, you can run gjournal.

Your comments on Gumstix, Beagleboard, and Tonidoplus are exactly the sort of thing I was fishing for with my original question - they make this page more useful, as a guide for others interested in doing the same thing.
May. 12th, 2011 07:53 pm (UTC)
Re: Why dreamplug
It's certainly encouraging to hear that you've having this experience with FreeBSD; if this is also characteristic of NetBSD, it removes one of my main concerns about running it. I haven't run NetBSD on a server in a long time because I've had no reason to do so—it compares poorly with Ubuntu in terms of maintainability. The same is not true for gentoo—gentoo is very NetBSD-like.

The big win about both NetBSD and gentoo is that everything is oriented toward source code. The big win about Ubuntu is that things generally Just Work. But Ubuntu is way too quick to abandon useful hardware platforms in new releases.
Mar. 17th, 2012 11:50 pm (UTC)
NetBSD has WAPBL (journaling for UFS).
Mar. 18th, 2012 12:46 am (UTC)
Yup, I subsequently installed NetBSD with WAPBL. Pretty sweet.
May. 17th, 2011 02:09 pm (UTC)
Re: Why dreamplug
NetBSD acutally have journaling thanks to code donated from Wasabi. See the log option in mount(8) and also wapbl(4). It is still considered experimental though.
May. 12th, 2011 06:29 pm (UTC)
Make sure you set a root password
As for that, make sure you set:

PasswordAuthentication no
PermitRootLogin no
May. 12th, 2011 06:34 pm (UTC)
Yes. Of course, make sure you have a user account with a working ssh key in the authorized_keys file first. And make sure you can get a root shell when logged in as that user.
( 15 comments — Leave a comment )