Auto-restart crashed mining processes in ethOS 1.2.9

ethOS 1.2.9 brings a few changes which break my auto-restart script for ethOS 1.2.7. Since 1.2.9 contains improved GPU crash detection, I rewrote the restart script to use the built-in detection mechanisms. For the required cron job please see my initial post which is available here.

As long as the DRY_RUN variable is set to false, the script won’t take any action, it just logs what it would do.

 

Auto-restart crashed mining processes in ethOS 1.2.7

I have been running a crypto-currency mining rig on the Linux based ethOS distro for quite some time now. While I realize that ethOS is problematic license-wise, it’s still a great distro to get a mining rig up and running in almost no time. The Nvidia GPUs in my rig are well tuned to operate at their optimum cost/hashrate ratio. However, due to bugs in the miner and/or the GPU drivers, every few days one of the GPUs stops mining. Sometimes ethOS is able to recover the GPU and gets it back to mining but sometimes it doesn’t seem to detect the crashed/hanging mining process at all. This is why I added a cron job that runs every 15 minutes and checks if all GPUs are still mining. If not, the miners will be re started using the minestop and minestart commands provided by ethOS.

The cron job starts the Bash script below. If it detects a problem, it writes to the console and additionally to /tmp/rigcheck.log. It’s been running smoothly on my ethOS v1.2.7 mining rig. I recommend putting it in /home/ethos/rigcheck.sh and don’t forget to add execute permissions using chmod +x /home/ethos/rigcheck.sh

The cron job can be created like this:

cat << EOF | sudo tee /etc/cron.d/rigcheck
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
*/15 * * * *   root    /home/ethos/rigcheck.sh
EOF

Thanks to this script, crashed or hanging miners will be restarted fairly quickly and my rig’s pool-reported hashrate stopped dropping in such situations.

Random delay for cron.daily, cron.weekly, cron.monthly

cron-logoWouldn’t it be nice if cron’s daily, weekly and monthly jobs could be run with a slight offset? At least that’s what I thought when 20+ servers were hitting my backup infrastructure at once. The scripts in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly are triggered directly from crontab at fixed times. Here’s what /etc/crontab looks like in Ubuntu Server 16.04:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

I’ve found several tips which suggested to use a RANDOM_DELAY variable in crontab. Unfortunately, this variable doesn’t seem to be implemented in Debian/Ubuntu’s version of crontab at this time. I even checked the source code, there’s no RANDOM_DELAY variable to be found.

Here’s the solution I came up with. I’m using a combination of sleep and numrandom with a time range between 0 and 30 minutes.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

In order to use the numrandom command, you have to apt-get -y install num-utils it first.

I didn’t delay the cron.hourly execution but the same sleep/numrandom combo could be used for it as well, just maybe replace the m (minutes) with s (seconds).

How to generate daily PowerDNS statistics reports

daae8fe19d3ab224f3a104f987acd8bfPowerDNS has been powering authoritative DNS lookups to this web site for quite a while now. It’s such a remarkable piece of software. Here’s how to create a daily statistics report for PowerDNS.

  1. Put the lines below in /etc/cron.daily/powerdns-report:
    #!/bin/sh
    /usr/bin/curl -s localhost:8081 | mail -s "$(echo "PowerDNS Daily Report\nMIME-Version: 1.0\nContent-Type: text/html")" root
  2. Make sure the file is executable: chmod +x /etc/cron.daily/powerdns-report
  3. Enable the internal web server (defaults to 127.0.0.1:8081) in the PowerDNS configuration file:
    webserver=yes
  4. Restart PowerDNS
  5. Make sure all Mails to root are forwarded to your e-mail account (or replace the recipient in the powerdns-report script)

This is how the PowerDNS statistics report will look like:

PowerDNS 4.1.0
Uptime: 3.94 days
Queries/second, 1, 5, 10 minute averages: 0.402, 0.212, 0.186. Max queries/second: 1.17
Cache hitrate, 1, 5, 10 minute averages: 45.0%, 33.4%, 30.9%
Backend query cache hitrate, 1, 5, 10 minute averages: 55.1%, 41.7%, 38.3%
Backend query load, 1, 5, 10 minute averages: 0.343, 0.317, 0.32. Max queries/second: 2.37
Total queries: 52665. Question/answer latency: 1.25ms

Log Messages
gmysql Connection successful. Connected to database 'pdns' on 'localhost'.	422	27.8%
AXFR of domain 'nogo.com.' initiated by 231.91.120.148	70	4.6%
AXFR of domain 'nogo.com.' failed: 231.91.120.148 cannot request AXFR	70	4.6%
AXFR of domain 'nogo.com' denied: client IP 231.91.120.148 has no permission	70	4.6%
Rest:	527	34.7%
Total:	1519	100%

Queries for existing records, but for type we don't have
wopr.com./AAAA	2193	47.1%
wopr.me./AAAA	503	10.8%
drfalken.me./AAAA	482	10.4%
mail.wopr.net./AAAA	281	6.0%
wopr.net./AAAA	253	5.4%
Rest:	685	14.7%
Total:	4653	100%

Queries for non-existent records within existent domains
dn42-fr.wopr.me./A	359	24.9%
_sip._udp.sbc.sipmly.com.wopr.net./SRV	335	23.2%
_sip._udp.sip.ovh.fr.wopr.net./SRV	214	14.8%
dn42-us.wopr.me./A	75	5.2%
_adsp._domainkey.woprs.net./TXT	14	1.0%
Rest:	290	20.1%
Total:	1443	100%

UDP Queries Received
wopr.com./A	2307	23.1%
wopr.net./A	2227	22.3%
trick77.com./AAAA	702	7.0%
wheniskeynote.com./AAAA	578	5.8%
www.trick77.com./A	293	2.9%
Rest:	3135	31.4%
Total:	10000	100%

Queries that could not be answered due to backend errors
Total:	0	100%

Queries for domains that we are not authoritative for
nogo.com./SOA	190	84.4%
cpsc.gov./ANY	16	7.1%
dnsscan.shadowserver.org./A	4	1.8%
./ANY	3	1.3%
cpsc.gov./A	2	0.9%
defcon.org./ANY	2	0.9%
./NS	1	0.4%
4caa2d31.openresolvertest.net./A	1	0.4%
c.afekv.com./A	1	0.4%
doc.gov./ANY	1	0.4%
Rest:	4	1.8%
Total:	225	100%

Remote server IP addresses
137.24.55.110	289	2.9%
164.8.230.15	108	1.1%
8.91.147.135	106	1.1%
8.91.150.198	103	1.0%
28.1.240.64	100	1.0%
163.19.151.12	87	0.9%
15.16.90.42	84	0.8%
141.28.51.41	73	0.7%
169.16.124.226	70	0.7%
164.891.213.199	63	0.6%
Rest:	8917	89.2%
Total:	10000	100%

Remote hosts sending corrupt packets
219.116.116.12	1	50.0%
116.125.42.131	1	50.0%
Total:	2	100%

Remote hosts querying domains for which we are not auth
8.194.50.198	70	31.1%
8.194.47.135	68	30.2%
163.19.151.12	52	23.1%
185.59.233.131	15	6.7%
88.65.195.128	2	0.9%
64.32.96.66	2	0.9%
41.12.122.91	1	0.4%
45.20.24.195	1	0.4%
15.52.206.174	1	0.4%
89.146.222.158	1	0.4%
Rest:	12	5.3%
Total:	225	100%

Variables
corrupt-packets	2	Number of corrupt packets received
deferred-cache-inserts	1042	Amount of cache inserts that were deferred because of maintenance
deferred-cache-lookup	250	Amount of cache lookups that were deferred because of maintenance
dnsupdate-answers	0	DNS update packets successfully answered.
dnsupdate-changes	0	DNS update changes to records in total.
dnsupdate-queries	1	DNS update packets received.
dnsupdate-refused	1	DNS update packets that are refused.
incoming-notifications	0	NOTIFY packets received.
packetcache-hit	16206	
packetcache-miss	36486	
packetcache-size	1252	
query-cache-hit	52387	Number of hits on the query cache
query-cache-miss	97378	Number of misses on the query cache
rd-queries	212	Number of recursion desired questions
recursing-answers	0	Number of recursive answers sent out
recursing-questions	0	Number of questions sent to recursor
recursion-unanswered	0	Number of packets unanswered by configured recursor
security-status	0	Security status based on regular polling
servfail-packets	0	Number of times a server-failed packet was sent out
signatures	377	Number of DNSSEC signatures made
tcp-answers	497	Number of answers sent out over TCP
tcp-answers-bytes	434810	Total size of answers sent out over TCP
tcp-queries	299	Number of TCP queries received
tcp4-answers	485	Number of IPv4 answers sent out over TCP
tcp4-answers-bytes	432435	Total size of answers sent out over TCPv4
tcp4-queries	287	Number of IPv4 TCP queries received
tcp6-answers	12	Number of IPv6 answers sent out over TCP
tcp6-answers-bytes	2375	Total size of answers sent out over TCPv6
tcp6-queries	12	Number of IPv6 TCP queries received
timedout-packets	0	Number of packets which weren't answered within timeout set
udp-answers	52665	Number of answers sent out over UDP
udp-answers-bytes	4955023	Total size of answers sent out over UDP
udp-do-queries	42490	Number of UDP queries received with DO bit
udp-queries	52665	Number of UDP queries received
udp4-answers	40096	Number of IPv4 answers sent out over UDP
udp4-answers-bytes	3508260	Total size of answers sent out over UDPv4
udp4-queries	40096	Number of IPv4 UDP queries received
udp6-answers	12569	Number of IPv6 answers sent out over UDP
udp6-answers-bytes	1446763	Total size of answers sent out over UDPv6
udp6-queries	12569	Number of IPv6 UDP queries received
key-cache-size	12	Number of entries in the key cache
latency	1251	Average number of microseconds needed to answer a question
meta-cache-size	61	Number of entries in the metadata cache
qsize-q	0	Number of questions waiting for database attention
real-memory-usage	56033280	Actual unique use of memory in bytes (approx)
signature-cache-size	179	Number of entries in the signature cache
sys-msec	37296	Number of msec spent in system time
udp-in-errors	0	UDP 'in' errors
udp-noport-errors	45	UDP 'noport' errors
udp-recvbuf-errors	0	UDP 'recvbuf' errors
udp-sndbuf-errors	0	UDP 'sndbuf' errors
uptime	340341	Uptime of process in seconds
user-msec	57664	Number of msec spent in user time
© 2013 - 2016 PowerDNS.COM BV.

How to migrate a LVM-based KVM guest to another host

kvm-logo_300dpiIn the past, I have been using the immensely useful virt-backup.pl script to migrate a LVM-based (raw volume) Linux KVM guest from one host to another. However, there is an even easier way to cold-migrate a KVM guest. This approach is particularly helpful if there’s not enough disk space on the host to create a gzipped backup of the logical volume using the virt-backup.pl script.

Here’s how it works:

  1. Use lvcreate to create the new logical volume on the destination host with the same size as the source logical volume. Use the lvdisplay command to find out the required size.
  2. virsh shutdown the source KVM guest
  3. On the source host: screen dd if=/dev/vg_ssd/lv_vm_wopr | pv | ssh -C root@desthost dd of=/dev/vg0/lv_vm_wopr
  4. Wait until finished

I’m using the screen command so it will continue running in the background once I close the ssh session on the source host. Use CTRL-A-D to background a screen session and screen -dr to bring it back up.

Using ssh makes sure the entire transfer is encrypted. The -C parameter makes sure the content will be compressed which may speed up the transfer considerably (or not, on a slow CPU). Obviously, the new KVM guest has to be virsh define‘d on the destination based on the virsh dumpxml configuration data from the source host.

Adding a DS3231 Real Time Clock to the Raspberry Pi 3

ds3231-rtcSince the Raspberry Pi 3 doesn’t come with a battery-powered real time clock, it will only show the correct time once it has Internet connectivity (thanks to the NTP daemon). If the Raspberry Pi 3 is not connected to the Internet, you might want to add a hardware clock to set the current date. Here’s how to add a DS3231 real time clock GPIO module to the Raspberry Pi 3 in Raspbian Jessy Lite:

  1. Get a DS3231 real time clock module and install it on the GPIO header of the Raspberry Pi 3 on pin 1
  2. Add the following line at the end of /boot/config.txt in Raspbian Jessy:
    dtoverlay=i2c-rtc,ds3231
  3. We don’t need fake-hwclock anymore:
    apt-get purge fake-hwclock
  4. Check/set the current system time and write the system time to the RTC module using:
    hwclock -w
  5. Set the correct time zone using:
    dpkg-reconfigure tzdata
  6. Edit /etc/rc.local and add the hwclock command above the line that says “exit 0”:
    /sbin/hwclock -s
  7. The /etc/init.d/hwclock.sh shell scripts tends to corrupt this RTC clock module. In my case, the RTC clock was set to 2066/01/01 after every reboot. To prevent this from happening, edit /etc/default/hwclock and set HWCLOCKACCESS to no:
    HWCLOCKACCESS=no
  8. Reboot
  9. Done! Raspbian will now set the time from the RTC clock during boot even if there is no Internet connectivity available.
  10. If RTC corruption is still happening, you may have to get rid of the NTP daemon as well using:
    apt-get purge ntp
    apt-get install ntpdate
  11. After the NTP daemon has been removed, you can still sync the system clock using ntpdate-debian which you might add to /etc/rc.local as well (after the hwclock command though) – just in case there is an Internet connection available during boot. And/or add it to /etc/cron.daily for example.

Raspbian Jessy Lite will detect the DS3231 real time clock module automatically (as a DS1307 module but nevermind), there’s no need to whitelist or blacklist any I2C modules. There’s no need to run the i2cdetect command from the i2c-tools package. Once the clock module is detected, this line should be visible using dmesg:

# dmesg | grep rtc
[    6.640799] rtc-ds1307 1-0068: rtc core: registered ds3231 as rtc0

Check /proc/driver/rtc for more data on the RTC:

# cat /proc/driver/rtc
rtc_time : 19:26:18
rtc_date : 2016-03-25
alrm_time : 00:00:00
alrm_date : 1970-01-01
alarm_IRQ : no
alrm_pending : no
update IRQ enabled : no
periodic IRQ enabled : no
periodic IRQ frequency : 1
max user IRQ frequency : 64
24hr : yes

Query status information from Huawei’s HiLink 3G/LTE modems

While Huawei provides status information for its HiLink modems via a web page, this is hardly useful when using the modem on a headless Linux server. I just published a small Python-based command-line tool on Github which displays some useful information about the modem’s status.

root@wopr~#: python ./hstatus.py
Huawei E3372 LTE Modem (IMEI: 121032526613216)
 Hardware version: CL1D3271AM Ver.A
 Software version: 22.286.53.01.161
 Web UI version: 16.100.05.00.03-Mod1.4
 Serial: L8FDW11512114431
 MAC address (modem): 00:0D:87:12:1C:1D
 Connection status: Connected
   Network type: UMTS (3G)
   Signal level: ▁▃▄▆█
   Roaming: Enabled
   Modem WAN IP address: 10.197.75.231
   Public IP address: 185.13.106.181
   DNS IP addresses: 212.113.0.5, 66.28.0.62
   Network operator: Swisscom
   Connected for: 03:15:15 (hh:mm:ss)
   Downloaded: 615.17 KB
   Uploaded: 258.69 KB
 Total downloaded: 14.69 MB
 Total uploaded: 1.34 MB
 Unread SMS: 1

The tool has been tested on a Huawei E3276 and a E3372 modem. For the newer E3372 modem I had to add some code to supply a RequestVerificationToken in the HTTP header.

Feel free to send a pull request on Github with your own tweaks!

The repository is available here: github.com/trick77/huawei-hilink-status

How to compile Quagga with SNMP support

Since the default Quagga package in Ubuntu doesn’t have SNMP support enabled, the Quagga package has to be compiled locally. The following instructions may work for Debian as well but I only tested it in Ubuntu Server 14.04 LTS.

Now, edit /etc/quagga/daemons and enable at least zebra and bgpd and let’s create some empty config files for Quagga:
touch /etc/quagga/bgpd.conf ; touch /etc/quagga/zebra.conf

To enable SNMP support in Quagga, the line agentx has to be inserted into bgpd.conf and zebra.conf:

drfalken@wopr:/etc/quagga# head bgpd.conf
hostname AS65535
log file /var/log/quagga/bgpd.log

agentx
debug bgp events
debug bgp filters
debug bgp updates

router bgp 65535
bgp router-id 1.2.3.4

I wont dwelve into how to setup the SNMP daemon but don’t forget to add the following lines to the snmpd.conf configuration file and restart the SNMP daemon afterwards:
master agentx
agentxsocket /var/agentx/master
agentxperms 777 777

Make sure to set proper permissions for the agentx directory with a
chmod 755 /var/agentx/
or you will get error messages like snmp[warning]: Warning: Failed to connect to the agentx master agent ([NIL])

Once Quagga is able to connect to the local SNMP daemon, a message like this will show up in Quagga’s log file:
snmp[info]: NET-SNMP version 5.7.2 AgentX subagent connected

Monitoring Quagga BGP sessions using SNMP

Monitoring BGP sessions works fantastically using LibreNMS. You can chose to receive push notifications and/or emails if a BGP session goes down/up or is flapping. However, there’s some tinkering involved to display 32-bit ASNs properly in LibreNMS (let me know in the comments if you’re interested) because the MIB only handles 16-bit integers. Unfortunately, there’s no IPv6 support in Quagga’s current SNMP implementation as well.

quagga-librenms-bgp-graphs

quagga-librenms-bgp

Waking up a NAS from OS X at boot time using Wake-on-LAN (WOL)

Since OS X 10.11 El Capitan protects certain system directories from modifications, my NASwake solution to wake a NAS once the Mac starts up published back in 2010 is no longer working.

I decided against building another .pkg installer since it requires root permissions and I also prefer using Homebrew instead of some obscure binary for sending the magic WOL packet. Here are the four steps to start your Wake-on-Lan-capable NAS once your Mac starts up:

  1. Install “Homebrew” (required for the wakeonlan script)
  2. Install the wakeonlan script using the Homebrew package manager
  3. Save the naswake plist to /Library/LaunchDaemons
  4. Save the naswake shell script to /usr/local/bin and set your NAS’s MAC-address

1. Install Homebrew

See http://brew.sh for instructions. Once installed, check with brew doctor if Homebrew was installed properly.

2. Install the wakeonlan script using the Homebrew package manager

brew install wakeonlan

This will install the wakeonlan Perl script. See man wakeonlan for details.

3. Save the naswake plist to /Library/LaunchDaemons

sudo nano /Library/LaunchDaemons/com.trick77.wol2.plist

Paste the XML below into the editor.

4. Save the naswake shell script to /usr/local/bin

sudo nano /usr/local/bin/naswake.sh
sudo chmod +x /usr/local/bin/naswake.sh

Paste the script below into the editor. Don’t forget to set your NAS’ MAC address in the last line of the script!

That’s it! Make sure WOL is enabled in the NAS.

Setting up a Huawei E3276-150 4G/LTE USB modem on Ubuntu Server/Desktop

I just received an unlocked Huawei E3276s-150 4G/LTE USB modem/surfstick I bought on eBay the other day. I went for the E3276s-150 because the 150 seemed to be the most compatible option for European 4G mobile networks. There are even cheaper Huawei E3276 models like the E3276-920 which you can buy for less than 20 bucks. However, the 920 seems to be frequency-optimized for Asian mobile networks and may not perform as well as a E3276s-150 in Western Europe.
huawei_e3276_lte_cat4_usb_dongle
To my great surprise, setting up the Huawei E3276 on Ubuntu 15.04 Desktop was literally plug & play. After a few seconds after plugging it in, I was greeted with a “Connection Established” message. Nicely done, Canonical!

On Ubuntu Server, like most Huawei modems, the stick is recognised as a memory card reader. It has to be switched to a USB modem device first using the usb_modeswitch command in order to establish a mobile network connection. If it’s not already installed, usb_modeswitch can be installed using apt-get -y install usb-modeswitch.

Memory card reader mode:

drfalken@wopr:~# lsusb
Bus 001 Device 007: ID 12d1:1f01 Huawei Technologies Co., Ltd.

To turn the E3276 into a modem:

drfalken@wopr:~# usb_modeswitch -v 12d1 -p 1f01 -M '55534243123456780000000000000011062000000101000100000000000000'

If the change was successful, lsusb shows a different USB product id now:

drfalken@wopr:~# lsusb
Bus 001 Device 007: ID 12d1:14db Huawei Technologies Co., Ltd.

At the same time, dmesg should output something like this:

drfalken@wopr:~# dmesg -T
[Fri May 29 20:55:41 2015] usb 1-1: New USB device found, idVendor=12d1, idProduct=14db
[Fri May 29 20:55:41 2015] usb 1-1: New USB device strings: Mfr=2, Product=1, SerialNumber=0
[Fri May 29 20:55:41 2015] usb 1-1: Product: HUAWEI Mobile
[Fri May 29 20:55:41 2015] usb 1-1: Manufacturer: HUAWEI Technology
[Fri May 29 20:55:41 2015] cdc_ether 1-1:1.0 eth1: register 'cdc_ether' at usb-0000:00:14.0-1, CDC Ethernet Device, 57:2d:70:33:22:10

Since the modem registered itself on eth1 (the name depends on the number of network devices, it doesn’t HAVE to be on eth1), we now simply fetch an IP address from the modem using:

drfalken@wopr:~# dhclient -v eth1
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth1/57:2d:70:33:22:10
Sending on   LPF/eth1/57:2d:70:33:22:10
Sending on   Socket/fallback
DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 3 (xid=0x3b73326b)
DHCPREQUEST of 192.168.1.100 on eth1 to 255.255.255.255 port 67 (xid=0x3b73326b)
DHCPOFFER of 192.168.1.100 from 192.168.1.1
DHCPACK of 192.168.1.100 from 192.168.1.1
bound to 192.168.1.100 -- renewal in 36557 seconds.

Yay, the modem has made itself available on 192.168.1.100 (it even has a web interface on port 80) with a /24 prefix and a gateway at 192.168.1.1.
By the way, make sure none of your local networks use 192.168.1.0/24 or it will collide with the Huawei’s local network.

Depending on a few factors dhclient may or may not have changed the default gateway. If the default gateway points to the modem, it will be at 192.168.1.1 on eth1:

drfalken@wopr:~# ip route show | grep default
default via 192.168.1.1 dev eth1

If this is not the case, you may have to remove the existing default gateway and replace it using:

drfalken@wopr:~# ip route del default ; ip route add default via 192.168.1.1

And… connected!

drfalken@wopr:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=22.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=34.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=57 time=39.7 ms

Make sure /etc/resolv.conf contains a valid nameserver if you can’t resolve domain names.

To switch the Huawei E3276 into a modem at boot time, create /etc/udev/rules.d/70-usb-modeswitch.rules and insert this line:

ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1f01", RUN+="/usr/sbin/usb_modeswitch -v 12d1 -p 1f01 -M '55534243123456780000000000000011062000000101000100000000000000'"

To automatically add a valid nameserver in /etc/resolv.conf when eth1 comes up, add these lines to /etc/dhcp/dhclient.conf:

interface "eth1" {
  prepend domain-name-servers 8.8.8.8;
  request subnet-mask, broadcast-address, time-offset, routers,
          domain-name, domain-name-servers, domain-search, host-name,
          dhcp6.name-servers, dhcp6.domain-search,
          netbios-name-servers, netbios-scope, interface-mtu,
          rfc3442-classless-static-routes, ntp-servers,
          dhcp6.fqdn, dhcp6.sntp-servers;
  require routers, domain-name-servers;
}

If you don’t want to run dhclient manually, you can either add an eth1 dhcp section in /etc/network/interfaces or add the dhclient eth1 command to /etc/rc.local.

Just FYI: I’ve been using Vivid Vervet’s (Ubuntu 15.04) 3.19 kernel in Ubuntu Server 14.04 LTS. Vivid’s newer kernel can be installed using apt-get install linux-image-generic-lts-vivid. Not sure if it makes a difference compared to 14.04’s default kernel though.

How to encrypt an ODROID-C1’s Ubuntu root filesystem with DM-crypt LUKS

The starting point is a running ORDOID-C1 with the Ubuntu minimal image. Make sure the Ubuntu installation has been dist-upgraded and to use the latest linux-image-c1 kernel image available (3.10.72-78 as of this writing). The ODROID-C1 has to be running the lastest kernel available, check with uname -r. Some of the early Ubuntu minimal images may suffer from a dist-upgrade problem, see here how to fix it. I’m using an 8 GB eMMC card but if you change the /dev ids throughout the installation process, it may work with an SD-card too.

root@c1:/# apt-get -y install lvm2 cryptsetup parted nano rsync

Since we can’t convert a running root partition, we’re going to add a 3rd partition which will be our encrypted root filesystem. Check with parted -l where the 2nd partition ends because we’re going to add the new partition right there.

root@c1:~# parted -l
Model: MMC 008G92 (sd/mmc)
Disk /dev/mmcblk0: 7818MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 135MB 134MB primary fat16 lba
2 135MB 4295MB 4160MB primary ext4

In this case, the 2nd partition ends at 4295MB, which will be the starting value for the new partition.

root@c1:~# parted
(parted) mkpart primary ext4
Start? 4295MB
END? 100%
q

Now we’re going to tag the new partition for LVM:
root@c1:~# fdisk /dev/mmcblk0
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e

Command (m for help): w

The next step is to setup LVM and to encrypt our new root partition. I’m going to name the physical volume (PV) lvm, the volume group vg and the logical root volume root.

root@c1:~# cryptsetup -c aes-xts-plain -y -s 512 luksFormat /dev/mmcblk0p3
root@c1:~# cryptsetup luksOpen /dev/mmcblk0p3 lvm
root@c1:~# pvcreate /dev/mapper/lvm
root@c1:~# vgcreate vg /dev/mapper/lvm
root@c1:~# lvcreate -l 100%FREE -n root vg
root@c1:~# mkfs.ext4 /dev/mapper/vg-root

Let’s try to mount the new, encrypted root volume:

root@c1:~# mount /dev/mapper/vg-root /mnt

Copy the existing root volume to the new, enrypted root volume:

root@c1:~# rsync -av --exclude=/media --exclude=/mnt --exclude=/proc --exclude=/dev --exclude=/sys / /mnt

These are the commands to chroot our new root volume from the old root partition. You can always use them to gain access to the encrypted root volume from the old root partition.

mkdir -p /mnt/dev
mkdir -p /mnt/mnt
mkdir -p /mnt/proc
mkdir -p /mnt/sys
mkdir -p /mnt/media
cryptsetup luksOpen /dev/mmcblk0p3 lvm
mount /dev/mapper/vg-root /mnt
mount -o rbind /dev /mnt/dev
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -t vfat /dev/mmcblk0p1 /mnt/media
chroot /mnt

The following actions take place in the chrooted environment on the new, encrypted root LV. Make sure you don’t modify the existing root partition!

Register the encrypted volume in /etc/crypttab:

echo lvm UUID=$(cryptsetup luksUUID /dev/mmcblk0p3) none luks|tee /etc/crypttab

Add a line in /etc/fstab for the root volume:

/dev/mapper/vg-root / ext4 errors=remount-ro 0 1

Being chrooted to the encrypted volume, we can now regenerate the initrd.img using

root@c1:~# update-initramfs -u -k $(uname -r)

Since support for dm-crypt has already been built into the kernel of the minimal image, we don’t have to add any crypto-modules to /etc/initramfs-tools/modules/. initramfs also does a pretty good job (thanks to the MODULES=most setting) determining what to add for lvm and dm-crypt support. Just to make sure, let’s have a look what has been placed into the new initrd.img:

root@c1:~# lsinitramfs /boot/initrd.img-$(uname -r) | grep crypt
sbin/cryptsetup
conf/conf.d/cryptroot
scripts/local-top/cryptopensc
scripts/local-top/cryptroot
scripts/local-bottom/cryptopensc
lib/libcryptsetup.so.4
lib/modules/3.10.72-78/kernel/arch/arm/crypto
lib/modules/3.10.72-78/kernel/arch/arm/crypto/aes-arm.ko
lib/modules/3.10.72-78/kernel/crypto
lib/modules/3.10.72-78/kernel/crypto/rmd160.ko
lib/modules/3.10.72-78/kernel/crypto/seed.ko
lib/modules/3.10.72-78/kernel/crypto/michael_mic.ko
lib/modules/3.10.72-78/kernel/crypto/rmd320.ko
lib/modules/3.10.72-78/kernel/crypto/cast6_generic.ko
lib/modules/3.10.72-78/kernel/crypto/rmd128.ko
lib/modules/3.10.72-78/kernel/crypto/rmd256.ko
lib/modules/3.10.72-78/kernel/crypto/lzo.ko
lib/modules/3.10.72-78/kernel/crypto/blowfish_generic.ko
lib/modules/3.10.72-78/kernel/crypto/khazad.ko
lib/modules/3.10.72-78/kernel/crypto/serpent_generic.ko
lib/modules/3.10.72-78/kernel/crypto/fcrypt.ko
lib/modules/3.10.72-78/kernel/crypto/camellia_generic.ko
lib/modules/3.10.72-78/kernel/crypto/twofish_generic.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_tx.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_pq.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_raid6_recov.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_xor.ko
lib/modules/3.10.72-78/kernel/crypto/async_tx/async_memcpy.ko
lib/modules/3.10.72-78/kernel/crypto/cast5_generic.ko
lib/modules/3.10.72-78/kernel/crypto/tea.ko
lib/modules/3.10.72-78/kernel/crypto/xts.ko
lib/modules/3.10.72-78/kernel/crypto/xor.ko
lib/modules/3.10.72-78/kernel/crypto/ansi_cprng.ko
lib/modules/3.10.72-78/kernel/crypto/wp512.ko
lib/modules/3.10.72-78/kernel/crypto/cast_common.ko
lib/modules/3.10.72-78/kernel/crypto/tgr192.ko
lib/modules/3.10.72-78/kernel/crypto/zlib.ko
lib/modules/3.10.72-78/kernel/crypto/lrw.ko
lib/modules/3.10.72-78/kernel/crypto/anubis.ko
lib/modules/3.10.72-78/kernel/crypto/blowfish_common.ko
lib/modules/3.10.72-78/kernel/crypto/sha512_generic.ko
lib/modules/3.10.72-78/kernel/crypto/twofish_common.ko
lib/modules/3.10.72-78/kernel/crypto/salsa20_generic.ko
lib/modules/3.10.72-78/kernel/crypto/gf128mul.ko
lib/cryptsetup
lib/cryptsetup/askpass
lib/arm-linux-gnueabihf/libgcrypt.so.11

Looking good, the scripts/local-top/cryptroot script is now part of initrd.img. It will be called during the first phase of the Linux boot and ask for the root volume passphrase once initramfs is executed.

To update the uInitrd file in the FAT32 boot partition we first have to recreate the image using the new initrd.img containing lvm and crypto support:

root@c1:~# mkimage -A arm -O linux -T ramdisk -C none -a 0 -e 0 -n "uInitrd $(uname -r)" -d /boot/initrd.img-$(uname -r) /tmp/uInitrd-$(uname -r)

If everything went well, copy the uInitrd to /boot and /media:

root@c1:~# cp /tmp/uInitrd-$(uname -r) /boot
root@c1:~# cp /tmp/uInitrd-$(uname -r) /media/uInitrd

Edit /media/boot.ini (make a backup copy first) and replace the root UUID and add cryptdevice in the setenv booargs line:

# Boot Arguments
setenv bootargs "root=/dev/mapper/vg-root cryptdevice=/dev/mmcblk0p3:lvm ...and so on

Important: leave the rest of the setenv bootargs line intact!

That’s it.

root@c1:~# shutdown -r now

initramfs should now aks for the passphrase to decrypt the root volume during boot. Make sure to have a keyboard nearby ;-)

Free multi-domain SSL certificates from WoSign and HAProxy OCSP stapling

Since everyone now can get free 2-year multi-domain certificates from WoSign, I grabbed one for one of my web sites. However, WoSign’s OCSP server is located in China which may, depending on your and your server’s location, increase latency once the web browser is verifying the certificate’s revocation status. In my case from Europe:

PING ocsp6.wosign.com (111.206.66.61) 56(84) bytes of data.
64 bytes from 111.206.66.61: icmp_seq=1 ttl=53 time=428 ms
64 bytes from 111.206.66.61: icmp_seq=2 ttl=53 time=347 ms
64 bytes from 111.206.66.61: icmp_seq=3 ttl=53 time=312 ms
64 bytes from 111.206.66.61: icmp_seq=4 ttl=53 time=328 ms
64 bytes from 111.206.66.61: icmp_seq=5 ttl=53 time=313 ms

OCSP stapling comes in handy to reduce the latency for the revocation status check, again, depending on your clients and your server’s location.

Here’s the all-in-one shell script in /etc/cron.daily I’m using…

  1. to create the domain’s OCSP file for HAProxy
  2. to inject the latest OCSP data into a running HAProxy instance using its stats socket
#!/bin/sh
ROOT_CERT_FILE=/etc/ssl/private/wosign-root-bundle.crt
SERVER_CERT_FILE=/etc/haproxy/certs.d/domain.crt
HAPROXY_SOCKET=/var/run/haproxy.socket
OCSP_URL=`/usr/bin/openssl x509 -in $SERVER_CERT_FILE -text | grep -i ocsp | cut -d&quot;:&quot; -f2-2,3`
OCSP_FILE=${SERVER_CERT_FILE}.ocsp

/usr/bin/openssl ocsp -noverify -issuer $ROOT_CERT_FILE -cert $SERVER_CERT_FILE -url &quot;$OCSP_URL&quot; -respout $OCSP_FILE -header Host `echo &quot;$OCSP_URL&quot; | cut -d&quot;/&quot; -f3`
echo &quot;set ssl ocsp-response $(/usr/bin/base64 -w 10000 $OCSP_FILE)&quot; | socat stdio $HAPROXY_SOCKET

To check if OCSP stapling works:

openssl s_client -connect mydomain.xyz:443 -tls1 -tlsextdebug -status

or for SNI-only configurations:

openssl s_client -connect mydomain.xyz:443 -servername mydomain.xyz -tls1 -tlsextdebug -status

If it works, there should be an OCSP section in the response like this:

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = CN, O = WoSign CA Limited, CN = WoSign Free SSL OCSP Responder(G2)
    Produced At: Mar  8 14:01:14 2015 GMT
    .
    .
    .

A few notes:

  1. HAProxy’s stats socket needs to be enabled
  2. wosign-root-bundle.crt was taken from the Apache bundle in the certificate .zip file I received from WoSign
  3. /etc/haproxy/certs.d/domain.crt contains the private key and the certificate bundle from the “for Other Server” directory, however you could remove the last certificate since it’s the root CA cert.
  4. Requires HAProxy >= 1.5
  5. If socat is missing: apt-get install socat in Debian/Ubuntu
  6. Always aim for an A or A+ grade: SSL Server Test

How to set up a virtual KVM/VNC console on your OVH server

Want to install your own image on a OVH Kimsufi or SoYouStart server? Want to install an official image on your server instead of the pre-built OVH OS templates? Want to encrypt the home directory at install-time? Want to use RAID 5 using mdadm on one of those SSD equipped SoYouStart servers? Or do you want to use a more refined, custom partition layout which is not supported by the OVH partitioner? And you want this without having access to or having to pay for a KVM console?
Continue reading

KVM VM waits forever / stuck at kernel selection screen

If a KVM virtual machine has not been properly shutdown, it may wait forever at the boot up kernel selection screen for user input. You won’t see any output when trying to virsh console into the VM from the host. You have to fire up a remote VNC session in order to press enter.

In Ubuntu Server you can override this behaviour by adding the following line to /etc/default/grub:

GRUB_RECORDFAIL_TIMEOUT=2

Don’t forget to run update-grub afterwards.

How to install Kodi on Ubuntu Server 14.04

I have a small Shuttle Barbebone computer which I’m mainly using as a KVM hypervisor on top of Ubuntu Server 14.04 to run a few VMs. Since the Barebone also sports a HDMI port and the CPU comes with an integrated Intel HD GPU I thought it would be a great Kodi (ex XBMC) mediacenter as well. However, I’ve been unable to find a working walk-through on how to install it on Ubuntu Server. Most likely because nobody ever does this on a server OS. Anyway, here’s how to install the latest Kodi release on Ubuntu Server 14.04 including hardware acceleration for the Intel HD GPU. Continue reading