How to encrypt an ODROID-C1’s Ubuntu root filesystem with DM-crypt LUKS

The starting point is a running ORDOID-C1 with the Ubuntu minimal image. Make sure the Ubuntu installation has been dist-upgraded and to use the latest linux-image-c1 kernel image available (3.10.72-78 as of this writing). The ODROID has to be running the lastest kernel available, check with uname -r. Some of the early Ubuntu minimal images may suffer from a dist-upgrade problem, see here how to fix it. I’m using an 8 GB eMMC card but if you change the /dev ids throughout the installation process, it may work with an SD-card too.

root@c1:/# apt-get -y install lvm2 cryptsetup parted nano rsync

Since we can’t convert a running root partition, we’re going to add a 3rd partition which will be our encrypted root filesystem. Check with parted -l where the 2nd partition ends because we’re going to add the new partition right there.

root@c1:~# parted -l
Model: MMC 008G92 (sd/mmc)
Disk /dev/mmcblk0: 7818MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 135MB 134MB primary fat16 lba
2 135MB 4295MB 4160MB primary ext4

In this case, the 2nd partition ends at 4295MB, which will be the starting value for the new partition.

root@c1:~# parted
(parted) mkpart primary ext4
Start? 4295MB
END? 100%
q

Now we’re going to tag the new partition for LVM:
root@c1:~# fdisk /dev/mmcblk0
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e

Command (m for help): w

The next step is to setup LVM and to encrypt our new root partition. I’m going to name the physical volume (PV) lvm, the volume group vg and the logical root volume root.

root@c1:~# cryptsetup -c aes-xts-plain -y -s 512 luksFormat /dev/mmcblk0p3
root@c1:~# cryptsetup luksOpen /dev/mmcblk0p3 lvm
root@c1:~# pvcreate /dev/mapper/lvm
root@c1:~# vgcreate vg /dev/mapper/lvm
root@c1:~# lvcreate -l 100%FREE -n root vg
root@c1:~# mkfs.ext4 /dev/mapper/vg-root

Let’s try to mount the new, encrypted root volume:

root@c1:~# mount /dev/mapper/vg-root /mnt

Copy the existing root volume to the new, enrypted root volume:

root@c1:~# rsync -av --exclude=/media --exclude=/mnt --exclude=/proc --exclude=/dev --exclude=/sys / /mnt

These are the commands to chroot our new root volume from the old root partition. You can always use them to gain access to the encrypted root volume from the old root partition.

mkdir -p /mnt/dev
mkdir -p /mnt/mnt
mkdir -p /mnt/proc
mkdir -p /mnt/sys
mkdir -p /mnt/media
cryptsetup luksOpen /dev/mmcblk0p3 lvm
mount /dev/mapper/vg-root /mnt
mount -o rbind /dev /mnt/dev
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -t vfat /dev/mmcblk0p1 /mnt/media
chroot /mnt

The following actions take place in the chrooted environment on the new, encrypted root LV. Make sure you don’t modify the existing root partition!

Register the encrypted volume in /etc/crypttab:

echo lvm UUID=$(cryptsetup luksUUID /dev/mmcblk0p3) none luks|tee /etc/crypttab

Add a line in /etc/fstab for the root volume:

/dev/mapper/vg-root / ext4 errors=remount-ro 0 1

Being chrooted to the encrypted volume, we can now regenerate the initrd.img using

root@c1:~# update-initramfs -u -k $(uname -r)

Since support for dm-crypt has already been built into the kernel of the minimal image, we don’t have to add any crypto-modules to /etc/initramfs-tools/modules/. initramfs also does a pretty good job (thanks to the MODULES=most setting) determining what to add for lvm and dm-crypt support. Just to make sure, let’s have a look what has been placed into the new initrd.img:

root@c1:~# lsinitramfs /boot/initrd.img-$(uname -r) | grep crypt
sbin/cryptsetup
conf/conf.d/cryptroot
scripts/local-top/cryptopensc
scripts/local-top/cryptroot
scripts/local-bottom/cryptopensc
lib/libcryptsetup.so.4
lib/modules/3.10.72-78/kernel/arch/arm/crypto
lib/modules/3.10.72-78/kernel/arch/arm/crypto/aes-arm.ko
lib/modules/3.10.72-78/kernel/crypto
lib/modules/3.10.72-78/kernel/crypto/rmd160.ko
lib/modules/3.10.72-78/kernel/crypto/seed.ko
lib/modules/3.10.72-78/kernel/crypto/michael_mic.ko
lib/modules/3.10.72-78/kernel/crypto/rmd320.ko
lib/modules/3.10.72-78/kernel/crypto/cast6_generic.ko
lib/modules/3.10.72-78/kernel/crypto/rmd128.ko
lib/modules/3.10.72-78/kernel/crypto/rmd256.ko
lib/modules/3.10.72-78/kernel/crypto/lzo.ko
lib/modules/3.10.72-78/kernel/crypto/blowfish_generic.ko
lib/modules/3.10.72-78/kernel/crypto/khazad.ko
lib/modules/3.10.72-78/kernel/crypto/serpent_generic.ko
lib/modules/3.10.72-78/kernel/crypto/fcrypt.ko
lib/modules/3.10.72-78/kernel/crypto/camellia_generic.ko
lib/modules/3.10.72-78/kernel/crypto/twofish_generic.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_tx.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_pq.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_raid6_recov.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_xor.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_memcpy.ko
lib/modules/3.10.72-78/kernel/crypto/cast5_generic.ko
lib/modules/3.10.72-78/kernel/crypto/tea.ko
lib/modules/3.10.72-78/kernel/crypto/xts.ko
lib/modules/3.10.72-78/kernel/crypto/xor.ko
lib/modules/3.10.72-78/kernel/crypto/ansi_cprng.ko
lib/modules/3.10.72-78/kernel/crypto/wp512.ko
lib/modules/3.10.72-78/kernel/crypto/cast_common.ko
lib/modules/3.10.72-78/kernel/crypto/tgr192.ko
lib/modules/3.10.72-78/kernel/crypto/zlib.ko
lib/modules/3.10.72-78/kernel/crypto/lrw.ko
lib/modules/3.10.72-78/kernel/crypto/anubis.ko
lib/modules/3.10.72-78/kernel/crypto/blowfish_common.ko
lib/modules/3.10.72-78/kernel/crypto/sha512_generic.ko
lib/modules/3.10.72-78/kernel/crypto/twofish_common.ko
lib/modules/3.10.72-78/kernel/crypto/salsa20_generic.ko
lib/modules/3.10.72-78/kernel/crypto/gf128mul.ko
lib/cryptsetup
lib/cryptsetup/askpass
lib/arm-linux-gnueabihf/libgcrypt.so.11

Looking good, the scripts/local-top/cryptroot script is now part of initrd.img. It will be called during the first phase of the Linux boot and ask for the root volume passphrase once initramfs is executed.

To update the uInitrd file in the FAT32 boot partition we first have to recreate the image using the new initrd.img containing lvm and crypto support:

root@c1:~# mkimage -A arm -O linux -T ramdisk -C none -a 0 -e 0 -n "uInitrd $(uname -r)" -d /boot/initrd.img-$(uname -r) /tmp/uInitrd-$(uname -r)

If everything went well, copy the uInitrd to /boot and /media:

root@c1:~# cp /tmp/uInitrd-$(uname -r) /boot
root@c1:~# cp /tmp/uInitrd-$(uname -r) /media/uInitrd

Edit /media/boot.ini (make a backup copy first) and replace the root UUID and add cryptdevice in the setenv booargs line:

# Boot Arguments
setenv bootargs "root=/dev/mapper/vg-root cryptdevice=/dev/mmcblk0p3:lvm ...and so on

Important: leave the rest of the setenv bootargs line intact!

That’s it.

root@c1:~# shutdown -r now

initramfs should now aks for the passphrase to decrypt the root volume during boot. Make sure to have a keyboard nearby ;-)

Free multi-domain SSL certificates from WoSign and HAProxy OCSP stapling

Since everyone now can get free 2-year multi-domain certificates from WoSign, I grabbed one for one of my web sites. However, WoSign’s OCSP server is located in China which may, depending on your and your server’s location, increase latency once the web browser is verifying the certificate’s revocation status. In my case from Europe:

PING ocsp6.wosign.com (111.206.66.61) 56(84) bytes of data.
64 bytes from 111.206.66.61: icmp_seq=1 ttl=53 time=428 ms
64 bytes from 111.206.66.61: icmp_seq=2 ttl=53 time=347 ms
64 bytes from 111.206.66.61: icmp_seq=3 ttl=53 time=312 ms
64 bytes from 111.206.66.61: icmp_seq=4 ttl=53 time=328 ms
64 bytes from 111.206.66.61: icmp_seq=5 ttl=53 time=313 ms

OCSP stapling comes in handy to reduce the latency for the revocation status check, again, depending on your clients and your server’s location.

Here’s the all-in-one shell script in /etc/cron.daily I’m using…

  1. to create the domain’s OCSP file for HAProxy
  2. to inject the latest OCSP data into a running HAProxy instance using its stats socket
#!/bin/sh
ROOT_CERT_FILE=/etc/ssl/private/wosign-root-bundle.crt
SERVER_CERT_FILE=/etc/haproxy/certs.d/domain.crt
HAPROXY_SOCKET=/var/run/haproxy.socket
OCSP_URL=`/usr/bin/openssl x509 -in $SERVER_CERT_FILE -text | grep -i ocsp | cut -d":" -f2-2,3`
OCSP_FILE=${SERVER_CERT_FILE}.ocsp

/usr/bin/openssl ocsp -noverify -issuer $ROOT_CERT_FILE -cert $SERVER_CERT_FILE -url "$OCSP_URL" -respout $OCSP_FILE -header Host `echo "$OCSP_URL" | cut -d"/" -f3`
echo "set ssl ocsp-response $(/usr/bin/base64 -w 10000 $OCSP_FILE)" | socat stdio $HAPROXY_SOCKET

To check if OCSP stapling works:

openssl s_client -connect mydomain.xyz:443 -tls1 -tlsextdebug -status

or for SNI-only configurations:

openssl s_client -connect mydomain.xyz:443 -servername mydomain.xyz -tls1 -tlsextdebug -status

If it works, there should be an OCSP section in the response like this:

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = CN, O = WoSign CA Limited, CN = WoSign Free SSL OCSP Responder(G2)
    Produced At: Mar  8 14:01:14 2015 GMT
    .
    .
    .

A few notes:

  1. HAProxy’s stats socket needs to be enabled
  2. wosign-root-bundle.crt was taken from the Apache bundle in the certificate .zip file I received from WoSign
  3. /etc/haproxy/certs.d/domain.crt contains the private key and the certificate bundle from the “for Other Server” directory, however you could remove the last certificate since it’s the root CA cert.
  4. Requires HAProxy >= 1.5
  5. If socat is missing: apt-get install socat in Debian/Ubuntu
  6. Always aim for an A or A+ grade: SSL Server Test

How to set up a transparent VPN Internet gateway tunnel using OpenVPN

I created a transparent VPN Internet gateway tunnel (sorry, couldn’t come up with a better name for it) using OpenVPN and my new Odroid-C1 Linux mini computer. However, this will work with any Linux PC (including the Raspberry Pi). The beauty of a transparent VPN gateway is that a device in the LAN doesn’t have to know anything about the VPN. I don’t have to remember to turn on the VPN nor does it drain the battery on mobile devices to encrypt and decrypt the packets. The VPN is just “there”. On the other hand, mostly for performance reasons, I don’t want to encrypt all traffic leaving my home LAN, that’s why I didn’t set up the VPN in the existing router. vpn-gatewayI wanted to be able to choose, on a per-device basis, which devices will route their traffic unencrypted to my ISP and which devices will get their traffic encrypted and forwarded to the remote VPN server using a second gateway in my LAN. And all this without additional subnets in my LAN, VLANs or additional WiFi or Ethernet-adapters. This may not look like the brightest idea to everyone but it works for me and I wanted to document it to save time if I have to set it up again. This is not a step-by-step tutorial but should provide enough pointers to get started.

Continue reading

How to install Kodi on an Odroid-C1 as a standalone mediacenter

The Odroid-C1 ist just too cool not to have. This feature-packed ARM7 quad core Linux mini computer comes with an incredible price tag of $35. However, with all the accessories (RTC-battery, power supply, case, mini-HDMI cable, eMMC card, remote control…) and shipping from Korea, the final price is around $100. It’s going to replace my Raspberry Pi which I initially intended to use as a media center but it always felt a little too slow for the task, even with the highly tuned Raspbmc.

Since I wanted the Odroid-C1 to run Kodi without a desktop manager (but with an Ubuntu repository), I started off with the Ubuntu 14.04 minimal image provided by Hardkernel. Don’t forget to resize the root partition to its true size once the Odroid is up and running (and reboot again!). I’m recommending the Odroid-Utility for doing this. And while you’re at it, make sure to “Update udev rules for ODROID subdevices” in the “Update your Kernel/Firmware” menu. If you forget this step, Kodi might abort with ERROR: failed to initialize egl display.

odroid-c1-kodi-summary

I’m assuming here that the Odroid-C1 has network connectivity and you’re logged in as root.

With a few adaptations, this information was taken from my existing post on how to install Kodi on an Ubuntu 14.04 server. Continue reading

How to set up a virtual KVM/VNC console on your OVH server

Want to install your own image on a OVH Kimsufi or SoYouStart server? Want to install an official image on your server instead of the pre-built OVH OS templates? Want to encrypt the home directory at install-time? Want to use RAID 5 using mdadm on one of those SSD equipped SoYouStart servers? Or do you want to use a more refined, custom partition layout which is not supported by the OVH partitioner? And you want this without having access to or having to pay for a KVM console?
Continue reading

KVM VM waits forever / stuck at kernel selection screen

If a KVM virtual machine has not been properly shutdown, it may wait forever at the boot up kernel selection screen for user input. You won’t see any output when trying to virsh console into the VM from the host. You have to fire up a remote VNC session in order to press enter.

In Ubuntu Server you can override this behaviour by adding the following line to /etc/default/grub:

GRUB_RECORDFAIL_TIMEOUT=2

Don’t forget to run update-grub afterwards.

How to install Kodi on Ubuntu Server 14.04

I have a small Shuttle Barbebone computer which I’m mainly using as a KVM hypervisor on top of Ubuntu Server 14.04 to run a few VMs. Since the Barebone also sports a HDMI port and the CPU comes with an integrated Intel HD GPU I thought it would be a great Kodi (ex XBMC) mediacenter as well. However, I’ve been unable to find a working walk-through on how to install it on Ubuntu Server. Most likely because nobody ever does this on a server OS. Anyway, here’s how to install the latest Kodi release on Ubuntu Server 14.04 including hardware acceleration for the Intel HD GPU. Continue reading

QEMU on Haswell causes spurious MCE events

A few dozen times each day, the Xeon E3-1275 v3 CPU on my SuperMicro X10SLM-F board generates a Machine Check Event (MCE). The Linux kernel logs all MCEs in /var/log/syslog:

mce: [Hardware Error]: Machine check events logged
mce: [Hardware Error]: Machine check events logged
CMCI storm detected: switching to poll mode
CMCI storm subsided: switching to interrupt mode
mce_notify_irq: 14 callbacks suppressed
mce: [Hardware Error]: Machine check events logged
mce: CPU supports 9 MCE banks
mce: [Hardware Error]: Machine check events logged

After installing mcelog I was able to pull some more detailed information about the check events:

Hardware event. This is not a software error.
MCE 0
CPU 3 BANK 0
TIME 1415087019 Tue Nov  4 08:43:39 2014
MCG status:
MCi status:
Corrected error
Error enabled
MCA: Internal parity error
STATUS 90000040000f0005 MCGSTATUS 0
MCGCAP c09 APICID 6 SOCKETID 0
CPUID Vendor Intel Family 6 Model 60

The MCEs all look the same (affected is always BANK 0), just the CPU and the APICID may differ. I updated the BIOS, replaced the ECC RAM, replaced the mainboard but the errors kept showing up. Continue reading

OS X Yosemite installer shows blank/black screen when using Clover

This is a heads up for everybody with an Nvidia GTX 760 (other Nvidia cards may be affected as well) trying to install OS X Yosemite 10.10 using Clover on a Hackintosh. If you’re getting a blank/black screen at the start of the installation, try to add the boot flag nv_disable=1. My screen was getting dark just after the installer displayed DSMOS has arrived when using the -v verbose boot flag. It always happened right after the installer was switching from text mode to graphics mode.

        <key>Boot</key>
        <dict>
                <key>Arguments</key>
                <string>dart=0 -v kext-dev-mode=1 nv_disable=1</string>
        ...(more)

Once OS X Yosemite has been installed, the nv_disable boot flag is no longer required and should be removed.

Ubuntu release upgrade says ‘no new release found’ on IPv6-only server

I’m running some sort of an experimental KVM guest with IPv6 connectivity only. Since it still had Ubuntu Server 13.10 installed I tried to run a do-release-upgrade on it to upgrade it to the latest Ubuntu Server release – which at the time of this writing is 14.10. However, the do-release-upgrade command kept saying that no new release could be found:

root@ipv6lab:~# do-release-upgrade
Checking for a new Ubuntu release
No new release found

I verified the /etc/update-manager/release-upgrades configuration file but it already contained the Prompt=normal line. After doing some digging I found out that the do-release-upgrade tries to connect to http://changelogs.ubuntu.com but there is no AAAA DNS record for this host. Essentially, this means that an Ubuntu server can’t be upgraded to a newer release over IPv6 because it can’t connect to the update info site over IPv6.

root@ipv6lab:~# dig +short changelogs.ubuntu.com A
91.189.95.36
root@ipv6lab:~# dig +short changelogs.ubuntu.com AAAA
root@ipv6lab:~#

Interestingly, the Ubuntu APT repository update site is accessible over IPv6, which is why something like apt-get update runs fine on IPv6-only Ubuntu servers.

I solved the problem by creating an IPv6 to IPv4 HTTP proxy using HAProxy on a IPv4/IPv6 dual stack server. The proxy listens on an IPv6 address and “tunnels” all requests to changelogs.ubuntu.com using the IPv4 address of the changelogs server. I was able to upgrade to a newer Ubuntu release this way on an IPv6-only Ubuntu server. Continue reading

How to comfortably mount Clover’s EFI partition

I’ve been using command-line commands or the Clover Configurator to mount Clover’s EFI partition to edit Clover’s main configuration file.

However, I find it easiest to mount the hidden EFI volume in Disk Utility:

diskutility-efi

The hidden partitions will only be shown if Disk Utility’s debug mode has been enabled. In a shell, type:
defaults write com.apple.DiskUtility DUDebugMenuEnabled 1

Start Disk Utility and enable the option to show all partitions:

diskutility-debug

How to auto-boot an OS X Fusion Drive using Clover

Ever since I fusioned a SSD and a HDD into an OS X Fusion Drive, Clover has been unable to auto-boot the new logical Fusion Drive volume. Clover was just sitting on its startup volume selection screen and was waiting for me to select the volume to boot. I’ve found some hints that using an UUID should make Clover autoboot the Fusion drive but I’ve been unable to make it work with any of the UUIDs of the logical/physical volume.

What finally worked was using the system ID (or whatever this is called) of the volume. Here’s an excerpt from my Clover configuration:

	<key>Boot</key>
	<dict>
		<key>Arguments</key>
		<string>dart=0</string>
		<key>DefaultVolume</key>
		<string>HD(3,GPT,17337FC1-A0F7-4C73-DEA1-363BA11AB811,0x3A346008,0x40000)</string>
		<key>Timeout</key>
		<integer>5</integer>

With this ID, Clover auto-boots my Fusion Drive volume just fine after waiting for 5 seconds for user input.

The full IDs can be found in Clover’s log file in /Library/Logs/CloverEFI/ and look like this:
system.log:0:837 0:000 PciRoot(0x0)\Pci(0x1F,0x2)\Sata(0x0,0xFFFF,0x0)\HD(3,GPT,17337FC1-A0F7-4C73-DEA1-363BA11AB811,0x3A346008,0x40000)

You have to strip the PciRoot/Sata part for Clover.

How to rename an OS X Fusion Drive

Since OS X Yosemite, the CoreStorage service allows you to rename the logical volume name of a Fusion Drive if you wish to do so.

sudo diskutil cs rename "Macintosh HD" "Fusion Drive"

The Fusion Drive now shows up as “Fusion Drive” instead of “Macintosh HD” which was the name I’ve chosen initially. The OS X main volume is still named “Macintosh HD”.

diskutil-fusiondrive

Could not load host key: /etc/ssh/ssh_host_ed25519_key

Sep 29 19:19:41 wopr sshd[11801]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key

If you’re getting this error message in the log file, you most likely have the ed25519 HostKey enabled in your sshd_config file but for some reason, no host key was generated for it.

Since openssh-6.4 you can run the ssh-keygen command to generate any missing host keys:

$ ssh-keygen -A
ssh-keygen: generating new host keys: ED25519

Enable KVM guest console access in Ubuntu using the virsh console command

Usually, my first step after setting up a new Ubuntu guest is to enable console access in order gain shell access on the newly created VM.

Step 1 – Activate the serial console in the guest

nano /etc/default/grub

Change the GRUB_CMDLINE_LINUX_DEFAULT to:

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,38400n8 console=tty0"

Don’t forget to update Grub

update-grub

Step 2 – Create the serial console in the guest

cp /etc/init/tty1.conf /etc/init/ttyS0.conf
nano /etc/init/ttyS0.conf

Edit ttyS0.conf and replace the tty1 with ttyS0 in the last line so it will read something like “exec /sbin/getty -8 38400 ttyS0″.
Reboot the VM.

Step 3 – Log in from the host

virsh console myvm

This is it! You just gained console access to your VM.

Tip: To exit the console, hit CTRL-]. It doesn’t matter where the ] is located on your keyboard, you have to press the key below the <BACKSPACE> key and on the left side of the <ENTER> key.