Random delay for cron.daily, cron.weekly, cron.monthly

cron-logoWouldn’t it be nice if cron’s daily, weekly and monthly jobs could be run with a slight offset? At least that’s what I thought when 20+ servers were hitting my backup infrastructure at once. The scripts in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly are triggered directly from crontab at fixed times. Here’s what /etc/crontab looks like in Ubuntu Server 16.04:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

I’ve found several tips which suggested to use a RANDOM_DELAY variable in crontab. Unfortunately, this variable doesn’t seem to be implemented in Debian/Ubuntu’s version of crontab at this time. I even checked the source code, there’s no RANDOM_DELAY variable to be found.

Here’s the solution I came up with. I’m using a combination of sleep and numrandom with a time range between 0 and 30 minutes.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

In order to use the numrandom command, you have to apt-get -y install num-utils it first.

I didn’t delay the cron.hourly execution but the same sleep/numrandom combo could be used for it as well, just maybe replace the m (minutes) with s (seconds).

Installing Ubuntu Server 16.04 on PC Engines APU or APU2

Most people use PC Engines APU series (APU1D4, APU2C4) system boards for pfSense firewalls (pfSense is awesome!). However, the Ubuntu Server x86-64 version runs on these boards very well too which can turn them into a lightweight, portable Plex Media Server for instance. The APU series doesn’t have a video port, that’s why the Ubuntu Server 16.04 image requires some modifications in order to use the serial port for console output instead. Since the Ubuntu image is using a read-only CD-ROM filesystem, I’m using UNetbootin to create a bootable Live USB drive which lets me modify files. While UNetbootin is available on Linux and MacOS too, only the Windows version gave me consistent results after formatting the USB drive to FAT32 file format. YMMW, but if you get weird bootloader errors, try formatting/creating the bootable drive on Windows.

To access the APU’s serial port, a RS-232 DB9 null-modem to USB interface is required and some software to connect to it. I’m using a Prolific PL-2303 based interface and minicom on Linux or Serial when I’m on my Mac.

Once the Live USB drive has been successfully prepared by UNetbootin, the following files have to be modified in order to send the console output over the serial port:

In /isolinux/isolinux.cfg, insert the following lines at the top:

serial 0 115200
console 0

In /isolinux/txt.cfg, the replace the first occurrence of the append keyword (in the “install” section) with:

append file=/cdrom/preseed/ubuntu-server.seed vga=off initrd=/install/initrd.gz -- console=ttyS0,115200n8 -

In /syslinux.cfg, insert the following lines at the top:

serial 0 115200
console 0

Again in /syslinux.cfg, replace the first occurrence of the append keyword (in the “unetbootindefault” section) with:

append initrd=/ubninit vga=off console=ttyS0,115200n8 --

Using the serial cable you should now be able to install Ubuntu Server 16.04 on the APU:

apu-ubuntu-serial-console-1apu-ubuntu-serial-console-2

During the installation:

  • Make sure the APU is connected to a router. While configuring the network, always keep in mind that the rightmost network port ist the first port (eth0 or enp1s0)
  • Make sure to include “OpenSSH server” when choosing software to install

Most likely, there won’t be any visible console output (i.e. a login prompt) after the first reboot because the installer didn’t add the necessary parameters to GRUB_CMDLINE_LINUX. This is where the SSH daemon comes in handy (-:

To fix this, use SSH to login to the server and modify /etc/default/grub to include the following line:

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

Run update-grub , reboot the APU and eventually there should be a login prompt:

apu-ubuntu-serial-console-3

Setting up a Huawei E3276-150 4G/LTE USB modem on Ubuntu Server/Desktop

I just received an unlocked Huawei E3276s-150 4G/LTE USB modem/surfstick I bought on eBay the other day. I went for the E3276s-150 because the 150 seemed to be the most compatible option for European 4G mobile networks. There are even cheaper Huawei E3276 models like the E3276-920 which you can buy for less than 20 bucks. However, the 920 seems to be frequency-optimized for Asian mobile networks and may not perform as well as a E3276s-150 in Western Europe.
huawei_e3276_lte_cat4_usb_dongle
To my great surprise, setting up the Huawei E3276 on Ubuntu 15.04 Desktop was literally plug & play. After a few seconds after plugging it in, I was greeted with a “Connection Established” message. Nicely done, Canonical!

On Ubuntu Server, like most Huawei modems, the stick is recognised as a memory card reader. It has to be switched to a USB modem device first using the usb_modeswitch command in order to establish a mobile network connection. If it’s not already installed, usb_modeswitch can be installed using apt-get -y install usb-modeswitch.

Memory card reader mode:

drfalken@wopr:~# lsusb
Bus 001 Device 007: ID 12d1:1f01 Huawei Technologies Co., Ltd.

To turn the E3276 into a modem:

drfalken@wopr:~# usb_modeswitch -v 12d1 -p 1f01 -M '55534243123456780000000000000011062000000101000100000000000000'

If the change was successful, lsusb shows a different USB product id now:

drfalken@wopr:~# lsusb
Bus 001 Device 007: ID 12d1:14db Huawei Technologies Co., Ltd.

At the same time, dmesg should output something like this:

drfalken@wopr:~# dmesg -T
[Fri May 29 20:55:41 2015] usb 1-1: New USB device found, idVendor=12d1, idProduct=14db
[Fri May 29 20:55:41 2015] usb 1-1: New USB device strings: Mfr=2, Product=1, SerialNumber=0
[Fri May 29 20:55:41 2015] usb 1-1: Product: HUAWEI Mobile
[Fri May 29 20:55:41 2015] usb 1-1: Manufacturer: HUAWEI Technology
[Fri May 29 20:55:41 2015] cdc_ether 1-1:1.0 eth1: register 'cdc_ether' at usb-0000:00:14.0-1, CDC Ethernet Device, 57:2d:70:33:22:10

Since the modem registered itself on eth1 (the name depends on the number of network devices, it doesn’t HAVE to be on eth1), we now simply fetch an IP address from the modem using:

drfalken@wopr:~# dhclient -v eth1
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth1/57:2d:70:33:22:10
Sending on   LPF/eth1/57:2d:70:33:22:10
Sending on   Socket/fallback
DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 3 (xid=0x3b73326b)
DHCPREQUEST of 192.168.1.100 on eth1 to 255.255.255.255 port 67 (xid=0x3b73326b)
DHCPOFFER of 192.168.1.100 from 192.168.1.1
DHCPACK of 192.168.1.100 from 192.168.1.1
bound to 192.168.1.100 -- renewal in 36557 seconds.

Yay, the modem has made itself available on 192.168.1.100 (it even has a web interface on port 80) with a /24 prefix and a gateway at 192.168.1.1.
By the way, make sure none of your local networks use 192.168.1.0/24 or it will collide with the Huawei’s local network.

Depending on a few factors dhclient may or may not have changed the default gateway. If the default gateway points to the modem, it will be at 192.168.1.1 on eth1:

drfalken@wopr:~# ip route show | grep default
default via 192.168.1.1 dev eth1

If this is not the case, you may have to remove the existing default gateway and replace it using:

drfalken@wopr:~# ip route del default ; ip route add default via 192.168.1.1

And… connected!

drfalken@wopr:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=22.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=34.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=57 time=39.7 ms

Make sure /etc/resolv.conf contains a valid nameserver if you can’t resolve domain names.

To switch the Huawei E3276 into a modem at boot time, create /etc/udev/rules.d/70-usb-modeswitch.rules and insert this line:

ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1f01", RUN+="/usr/sbin/usb_modeswitch -v 12d1 -p 1f01 -M '55534243123456780000000000000011062000000101000100000000000000'"

To automatically add a valid nameserver in /etc/resolv.conf when eth1 comes up, add these lines to /etc/dhcp/dhclient.conf:

interface "eth1" {
  prepend domain-name-servers 8.8.8.8;
  request subnet-mask, broadcast-address, time-offset, routers,
          domain-name, domain-name-servers, domain-search, host-name,
          dhcp6.name-servers, dhcp6.domain-search,
          netbios-name-servers, netbios-scope, interface-mtu,
          rfc3442-classless-static-routes, ntp-servers,
          dhcp6.fqdn, dhcp6.sntp-servers;
  require routers, domain-name-servers;
}

If you don’t want to run dhclient manually, you can either add an eth1 dhcp section in /etc/network/interfaces or add the dhclient eth1 command to /etc/rc.local.

Just FYI: I’ve been using Vivid Vervet’s (Ubuntu 15.04) 3.19 kernel in Ubuntu Server 14.04 LTS. Vivid’s newer kernel can be installed using apt-get install linux-image-generic-lts-vivid. Not sure if it makes a difference compared to 14.04’s default kernel though.

How to install Kodi on an ODROID-C1 as a standalone mediacenter

The ODROID-C1 ist just too cool not to have. This feature-packed ARM7 quad core Linux mini computer comes with an incredible price tag of $35. However, with all the accessories (RTC-battery, power supply, case, mini-HDMI cable, eMMC card, remote control…) and shipping from Korea, the final price is around $100. It’s going to replace my Raspberry Pi which I initially intended to use as a media center but it always felt a little too slow for the task, even with the highly tuned Raspbmc.

Since I wanted the ODROID-C1 to run Kodi without a desktop manager (but with an Ubuntu repository), I started off with the Ubuntu 14.04 minimal image provided by Hardkernel. Don’t forget to resize the root partition to its true size once the ODROID-C1 is up and running (and reboot again!). I’m recommending the Odroid-Utility for doing this. And while you’re at it, make sure to “Update udev rules for ODROID subdevices” in the “Update your Kernel/Firmware” menu. If you forget this step, Kodi might abort with ERROR: failed to initialize egl display.

odroid-c1-kodi-summary

I’m assuming here that the ODROID-C1 has network connectivity and you’re logged in as root.

With a few adaptations, this information was taken from my existing post on how to install Kodi on an Ubuntu 14.04 server. Continue reading

How to set up a virtual KVM/VNC console on your OVH server

Want to install your own image on a OVH Kimsufi or SoYouStart server? Want to install an official image on your server instead of the pre-built OVH OS templates? Want to encrypt the home directory at install-time? Want to use RAID 5 using mdadm on one of those SSD equipped SoYouStart servers? Or do you want to use a more refined, custom partition layout which is not supported by the OVH partitioner? And you want this without having access to or having to pay for a KVM console?
Continue reading

How to install Kodi on Ubuntu Server 14.04

I have a small Shuttle Barbebone computer which I’m mainly using as a KVM hypervisor on top of Ubuntu Server 14.04 to run a few VMs. Since the Barebone also sports a HDMI port and the CPU comes with an integrated Intel HD GPU I thought it would be a great Kodi (ex XBMC) mediacenter as well. However, I’ve been unable to find a working walk-through on how to install it on Ubuntu Server. Most likely because nobody ever does this on a server OS. Anyway, here’s how to install the latest Kodi release on Ubuntu Server 14.04 including hardware acceleration for the Intel HD GPU. Continue reading

Ubuntu release upgrade says ‘no new release found’ on IPv6-only server

I’m running some sort of an experimental KVM guest with IPv6 connectivity only. Since it still had Ubuntu Server 13.10 installed I tried to run a do-release-upgrade on it to upgrade it to the latest Ubuntu Server release – which at the time of this writing is 14.10. However, the do-release-upgrade command kept saying that no new release could be found:

root@ipv6lab:~# do-release-upgrade
Checking for a new Ubuntu release
No new release found

I verified the /etc/update-manager/release-upgrades configuration file but it already contained the Prompt=normal line. After doing some digging I found out that the do-release-upgrade tries to connect to http://changelogs.ubuntu.com but there is no AAAA DNS record for this host. Essentially, this means that an Ubuntu server can’t be upgraded to a newer release over IPv6 because it can’t connect to the update info site over IPv6.

root@ipv6lab:~# dig +short changelogs.ubuntu.com A
91.189.95.36
root@ipv6lab:~# dig +short changelogs.ubuntu.com AAAA
root@ipv6lab:~#

Interestingly, the Ubuntu APT repository update site is accessible over IPv6, which is why something like apt-get update runs fine on IPv6-only Ubuntu servers.

I solved the problem by creating an IPv6 to IPv4 HTTP proxy using HAProxy on a IPv4/IPv6 dual stack server. The proxy listens on an IPv6 address and “tunnels” all requests to changelogs.ubuntu.com using the IPv4 address of the changelogs server. I was able to upgrade to a newer Ubuntu release this way on an IPv6-only Ubuntu server. Continue reading

strongSwan 5 based IPSec VPN, Ubuntu 14.04 LTS and PSK/XAUTH

I prefer strongSwan over Openswan because it’s still in active development, easier to setup and doesn’t require a L2TP daemon. I prefer a simple IKEv1 setup using PSK and XAUTH over certificates. If you plan to share your VPN server with your friends it’s also a lot easier to setup for them without certificates. I haven’t tried the VPN configuration below with non-Apple clients but it works well with iOS and OS X clients. Make sure to use the Cisco IPSec VPN profile, not the L2TP over IPSec profile you need for Openswan. While strongSwan works well with KVM and Xen containers, it probably won’t work with non-virtualised containers like OpenVZ or LXC. Continue reading

LXC 1.0 Web Panel for Ubuntu 14.04

LXC is awesome! You can create and start your own virtual container with just 3 commands in Ubuntu 14.04.

apt-get install lxc debootstrap lxc-templates
lxc-create -t ubuntu -n demo
lxc-start -n demo -d

It doesn’t get any easier than this. There’s even a Boostrap-based fronted available: LXC Web Panel.

lxc-web-panel

Unfortunately, LXC Web Panel doesn’t work with LXC 1.0 which is part of Ubuntu 14.04. Fortunately though, there’s a fork available on GitHub which adds support for LXC 1.0:

https://github.com/claudyus/LXC-Web-Panel

I re-forked claudyus’ LXC Web Panel fork and added support for Ubuntu 14.04 and a few other things. My forked fork is available here: https://github.com/trick77/LXC-Web-Panel Claudyus has already updated his repository with my changes.

By the way, the original author of LXC Web Panel said he’s currently working on a Bootstrap 3 based version for LXC 1.0 which will include a RESTful API and other new features. Make sure to follow this guy on GitHub: https://github.com/googley