ethOS 1.2.9 brings a few changes which break my auto-restart script for ethOS 1.2.7. Since 1.2.9 contains improved GPU crash detection, I rewrote the restart script to use the built-in detection mechanisms. For the required cron job please see my initial post which is available here.
As long as the DRY_RUN variable is set to false, the script won’t take any action, it just logs what it would do.
I have been running a crypto-currency mining rig on the Linux based ethOS distro for quite some time now. While I realize that ethOS is problematic license-wise, it’s still a great distro to get a mining rig up and running in almost no time. The Nvidia GPUs in my rig are well tuned to operate at their optimum cost/hashrate ratio. However, due to bugs in the miner and/or the GPU drivers, every few days one of the GPUs stops mining. Sometimes ethOS is able to recover the GPU and gets it back to mining but sometimes it doesn’t seem to detect the crashed/hanging mining process at all. This is why I added a cron job that runs every 15 minutes and checks if all GPUs are still mining. If not, the miners will be re started using the minestop and minestart commands provided by ethOS.
The cron job starts the Bash script below. If it detects a problem, it writes to the console and additionally to /tmp/rigcheck.log. It’s been running smoothly on my ethOS v1.2.7 mining rig. I recommend putting it in /home/ethos/rigcheck.sh and don’t forget to add execute permissions using chmod +x /home/ethos/rigcheck.sh
The cron job can be created like this:
cat << EOF | sudo tee /etc/cron.d/rigcheck PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root */15 * * * * root /home/ethos/rigcheck.sh EOF
Thanks to this script, crashed or hanging miners will be restarted fairly quickly and my rig’s pool-reported hashrate stopped dropping in such situations.
BitBar has been around for a while but I didn’t notice it until I wanted to display crypto currency rates (who isn’t into crypto these days :)) on my macOS menu bar. BitBar is as simple as it can get: it takes the output from a shell script and displays the result on the menu bar. What’s really cool about BitBar is the support for Base64 encoded images which get displayed as icons on the menu bar.
This is how my “crypto menu bar” looks like:
I just put all my BitBar shell scripts into ~/bitbar and that’s it.
Here are BitBar shell scripts for Monero, Ether and Bitcoin (against the €). Since the icons are just 16×16 they probably look crappy on a retina display and have to be replaced with larger icons.
A commenter recently asked if I had any USB 3.0 related issues with my Gigabyte Z97X-UD5H equipped Hackintosh. Since every USB port was working out of the box I thought everything was fine. However, having a closer look at the USB section in macOS Sierra’s System Information revealed that none of the USB 3.0 ports were operating at USB 3.0 speeds, they were all capped at 480 Mb/sec.
Here’s how I was able to get USB 3.0 speed back:
FakePCIID.kext and FakePCIID_XHCIMux.kext to Clovers kext directory.All front and back panel USB 3.0 ports are now reporting 3.0 speeds when connecting a 3.0 compatible device:
JSM578: Product ID: 0x0578 Vendor ID: 0x152d (JMicron Technology Corp.) Version: 2.03 Serial Number: DB123456789B Speed: Up to 5 Gb/sec Manufacturer: JMicron Location ID: 0x14f00000 / 20 Current Available (mA): 900 Current Required (mA): 896 Extra Operating Current (mA): 396
Major kudos to RehabMan for providing these easy-to-use injector kexts!
Since injector kexts are not drivers and thus do not have to be signed, my Hackintosh still runs with maximum system integrity protection (SIP):
Jans-Mac:~ jan$ csrutil status System Integrity Protection status: enabled.
Wouldn’t it be nice if cron’s daily, weekly and monthly jobs could be run with a slight offset? At least that’s what I thought when 20+ servers were hitting my backup infrastructure at once. The scripts in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly are triggered directly from crontab at fixed times. Here’s what /etc/crontab looks like in Ubuntu Server 16.04:
# /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do. SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) #
I’ve found several tips which suggested to use a RANDOM_DELAY variable in crontab. Unfortunately, this variable doesn’t seem to be implemented in Debian/Ubuntu’s version of crontab at this time. I even checked the source code, there’s no RANDOM_DELAY variable to be found.
Here’s the solution I came up with. I’m using a combination of sleep and numrandom with a time range between 0 and 30 minutes.
# /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do. SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) #
In order to use the numrandom command, you have to apt-get -y install num-utils it first.
I didn’t delay the cron.hourly execution but the same sleep/numrandom combo could be used for it as well, just maybe replace the m (minutes) with s (seconds).
PowerDNS has been powering authoritative DNS lookups to this web site for quite a while now. It’s such a remarkable piece of software. Here’s how to create a daily statistics report for PowerDNS.
/etc/cron.daily/powerdns-report:
#!/bin/sh
/usr/bin/curl -s localhost:8081 | mail -s "$(echo "PowerDNS Daily Report\nMIME-Version: 1.0\nContent-Type: text/html")" root
chmod +x /etc/cron.daily/powerdns-reportwebserver=yesThis is how the PowerDNS statistics report will look like:
PowerDNS 4.1.0 Uptime: 3.94 days Queries/second, 1, 5, 10 minute averages: 0.402, 0.212, 0.186. Max queries/second: 1.17 Cache hitrate, 1, 5, 10 minute averages: 45.0%, 33.4%, 30.9% Backend query cache hitrate, 1, 5, 10 minute averages: 55.1%, 41.7%, 38.3% Backend query load, 1, 5, 10 minute averages: 0.343, 0.317, 0.32. Max queries/second: 2.37 Total queries: 52665. Question/answer latency: 1.25ms Log Messages gmysql Connection successful. Connected to database 'pdns' on 'localhost'. 422 27.8% AXFR of domain 'nogo.com.' initiated by 231.91.120.148 70 4.6% AXFR of domain 'nogo.com.' failed: 231.91.120.148 cannot request AXFR 70 4.6% AXFR of domain 'nogo.com' denied: client IP 231.91.120.148 has no permission 70 4.6% Rest: 527 34.7% Total: 1519 100% Queries for existing records, but for type we don't have wopr.com./AAAA 2193 47.1% wopr.me./AAAA 503 10.8% drfalken.me./AAAA 482 10.4% mail.wopr.net./AAAA 281 6.0% wopr.net./AAAA 253 5.4% Rest: 685 14.7% Total: 4653 100% Queries for non-existent records within existent domains dn42-fr.wopr.me./A 359 24.9% _sip._udp.sbc.sipmly.com.wopr.net./SRV 335 23.2% _sip._udp.sip.ovh.fr.wopr.net./SRV 214 14.8% dn42-us.wopr.me./A 75 5.2% _adsp._domainkey.woprs.net./TXT 14 1.0% Rest: 290 20.1% Total: 1443 100% UDP Queries Received wopr.com./A 2307 23.1% wopr.net./A 2227 22.3% trick77.com./AAAA 702 7.0% wheniskeynote.com./AAAA 578 5.8% www.trick77.com./A 293 2.9% Rest: 3135 31.4% Total: 10000 100% Queries that could not be answered due to backend errors Total: 0 100% Queries for domains that we are not authoritative for nogo.com./SOA 190 84.4% cpsc.gov./ANY 16 7.1% dnsscan.shadowserver.org./A 4 1.8% ./ANY 3 1.3% cpsc.gov./A 2 0.9% defcon.org./ANY 2 0.9% ./NS 1 0.4% 4caa2d31.openresolvertest.net./A 1 0.4% c.afekv.com./A 1 0.4% doc.gov./ANY 1 0.4% Rest: 4 1.8% Total: 225 100% Remote server IP addresses 137.24.55.110 289 2.9% 164.8.230.15 108 1.1% 8.91.147.135 106 1.1% 8.91.150.198 103 1.0% 28.1.240.64 100 1.0% 163.19.151.12 87 0.9% 15.16.90.42 84 0.8% 141.28.51.41 73 0.7% 169.16.124.226 70 0.7% 164.891.213.199 63 0.6% Rest: 8917 89.2% Total: 10000 100% Remote hosts sending corrupt packets 219.116.116.12 1 50.0% 116.125.42.131 1 50.0% Total: 2 100% Remote hosts querying domains for which we are not auth 8.194.50.198 70 31.1% 8.194.47.135 68 30.2% 163.19.151.12 52 23.1% 185.59.233.131 15 6.7% 88.65.195.128 2 0.9% 64.32.96.66 2 0.9% 41.12.122.91 1 0.4% 45.20.24.195 1 0.4% 15.52.206.174 1 0.4% 89.146.222.158 1 0.4% Rest: 12 5.3% Total: 225 100% Variables corrupt-packets 2 Number of corrupt packets received deferred-cache-inserts 1042 Amount of cache inserts that were deferred because of maintenance deferred-cache-lookup 250 Amount of cache lookups that were deferred because of maintenance dnsupdate-answers 0 DNS update packets successfully answered. dnsupdate-changes 0 DNS update changes to records in total. dnsupdate-queries 1 DNS update packets received. dnsupdate-refused 1 DNS update packets that are refused. incoming-notifications 0 NOTIFY packets received. packetcache-hit 16206 packetcache-miss 36486 packetcache-size 1252 query-cache-hit 52387 Number of hits on the query cache query-cache-miss 97378 Number of misses on the query cache rd-queries 212 Number of recursion desired questions recursing-answers 0 Number of recursive answers sent out recursing-questions 0 Number of questions sent to recursor recursion-unanswered 0 Number of packets unanswered by configured recursor security-status 0 Security status based on regular polling servfail-packets 0 Number of times a server-failed packet was sent out signatures 377 Number of DNSSEC signatures made tcp-answers 497 Number of answers sent out over TCP tcp-answers-bytes 434810 Total size of answers sent out over TCP tcp-queries 299 Number of TCP queries received tcp4-answers 485 Number of IPv4 answers sent out over TCP tcp4-answers-bytes 432435 Total size of answers sent out over TCPv4 tcp4-queries 287 Number of IPv4 TCP queries received tcp6-answers 12 Number of IPv6 answers sent out over TCP tcp6-answers-bytes 2375 Total size of answers sent out over TCPv6 tcp6-queries 12 Number of IPv6 TCP queries received timedout-packets 0 Number of packets which weren't answered within timeout set udp-answers 52665 Number of answers sent out over UDP udp-answers-bytes 4955023 Total size of answers sent out over UDP udp-do-queries 42490 Number of UDP queries received with DO bit udp-queries 52665 Number of UDP queries received udp4-answers 40096 Number of IPv4 answers sent out over UDP udp4-answers-bytes 3508260 Total size of answers sent out over UDPv4 udp4-queries 40096 Number of IPv4 UDP queries received udp6-answers 12569 Number of IPv6 answers sent out over UDP udp6-answers-bytes 1446763 Total size of answers sent out over UDPv6 udp6-queries 12569 Number of IPv6 UDP queries received key-cache-size 12 Number of entries in the key cache latency 1251 Average number of microseconds needed to answer a question meta-cache-size 61 Number of entries in the metadata cache qsize-q 0 Number of questions waiting for database attention real-memory-usage 56033280 Actual unique use of memory in bytes (approx) signature-cache-size 179 Number of entries in the signature cache sys-msec 37296 Number of msec spent in system time udp-in-errors 0 UDP 'in' errors udp-noport-errors 45 UDP 'noport' errors udp-recvbuf-errors 0 UDP 'recvbuf' errors udp-sndbuf-errors 0 UDP 'sndbuf' errors uptime 340341 Uptime of process in seconds user-msec 57664 Number of msec spent in user time © 2013 - 2016 PowerDNS.COM BV.
In the past, I have been using the immensely useful virt-backup.pl script to migrate a LVM-based (raw volume) Linux KVM guest from one host to another. However, there is an even easier way to cold-migrate a KVM guest. This approach is particularly helpful if there’s not enough disk space on the host to create a gzipped backup of the logical volume using the virt-backup.pl script.
Here’s how it works:
lvcreate to create the new logical volume on the destination host with the same size as the source logical volume. Use the lvdisplay command to find out the required size.virsh shutdown the source KVM guestscreen dd if=/dev/vg_ssd/lv_vm_wopr | pv | ssh -C root@desthost dd of=/dev/vg0/lv_vm_woprI’m using the screen command so it will continue running in the background once I close the ssh session on the source host. Use CTRL-A-D to background a screen session and screen -dr to bring it back up.
Using ssh makes sure the entire transfer is encrypted. The -C parameter makes sure the content will be compressed which may speed up the transfer considerably (or not, on a slow CPU). Obviously, the new KVM guest has to be virsh define‘d on the destination based on the virsh dumpxml configuration data from the source host.
The Intel Gigabit CT Desktop ethernet PCI adapter is still one of the fastest and most robust NICs for the Hackintosh. This did not change with macOS Sierra 10.12. I’m still using the IONetworkingFamilyInjector.kext in Clover’s kext folder to override the compatibility list in Apple’s own Intel82574L.kext. However, while the installation of macOS Sierra went smoothly, I lost all network connectivity after installing Sierra. A quick look at the network kernel extensions revealed that Apple changed the driver identifier of the Intel82574L.kext, rendering the injector useless. After changing the identifier in the injector and a reboot, network connectivity was back again.
The patched injector kext is available for download here: IONetworkingFamilyInjector.kext_.macos-sierra.zip. The kext injector has to be placed into the EFI/CLOVER/kexts/10.12 folder.
The Hackintosh is still running in full protected mode (if enabled in Clover):
$ csrutil status
System Integrity Protection status: enabled.
While writing this post, I stumbled upon an alternative solution, which seems to be permanent. However, it requires flashing the Intel NIC and changing it’s device ID property. Check out this post on InsanelyMac. I’m going to try this approach in the near future since it would reduce the number of kexts in my Hackintosh rig to just one (only FakeSMC).
Most people use PC Engines APU series (APU1D4, APU2C4) system boards for pfSense firewalls (pfSense is awesome!). However, the Ubuntu Server x86-64 version runs on these boards very well too which can turn them into a lightweight, portable Plex Media Server for instance. The APU series doesn’t have a video port, that’s why the Ubuntu Server 16.04 image requires some modifications in order to use the serial port for console output instead. Since the Ubuntu image is using a read-only CD-ROM filesystem, I’m using UNetbootin to create a bootable Live USB drive which lets me modify files. While UNetbootin is available on Linux and MacOS too, only the Windows version gave me consistent results after formatting the USB drive to FAT32 file format. YMMW, but if you get weird bootloader errors, try formatting/creating the bootable drive on Windows.
To access the APU’s serial port, a RS-232 DB9 null-modem to USB interface is required and some software to connect to it. I’m using a Prolific PL-2303 based interface and minicom on Linux or Serial when I’m on my Mac.
Once the Live USB drive has been successfully prepared by UNetbootin, the following files have to be modified in order to send the console output over the serial port:
In /isolinux/isolinux.cfg, insert the following lines at the top:
serial 0 115200 console 0
In /isolinux/txt.cfg, the replace the first occurrence of the append keyword (in the “install” section) with:
append file=/cdrom/preseed/ubuntu-server.seed vga=off initrd=/install/initrd.gz -- console=ttyS0,115200n8 -
In /syslinux.cfg, insert the following lines at the top:
serial 0 115200 console 0
Again in /syslinux.cfg, replace the first occurrence of the append keyword (in the “unetbootindefault” section) with:
append initrd=/ubninit vga=off console=ttyS0,115200n8 --
Using the serial cable you should now be able to install Ubuntu Server 16.04 on the APU:
During the installation:
Most likely, there won’t be any visible console output (i.e. a login prompt) after the first reboot because the installer didn’t add the necessary parameters to GRUB_CMDLINE_LINUX. This is where the SSH daemon comes in handy (-:
To fix this, use SSH to login to the server and modify /etc/default/grub to include the following line:
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
Run update-grub , reboot the APU and eventually there should be a login prompt:
Since the Raspberry Pi 3 doesn’t come with a battery-powered real time clock, it will only show the correct time once it has Internet connectivity (thanks to the NTP daemon). If the Raspberry Pi 3 is not connected to the Internet, you might want to add a hardware clock to set the current date. Here’s how to add a DS3231 real time clock GPIO module to the Raspberry Pi 3 in Raspbian Jessy Lite:
dtoverlay=i2c-rtc,ds3231
apt-get purge fake-hwclock
hwclock -w
dpkg-reconfigure tzdata
/sbin/hwclock -s
HWCLOCKACCESS=no
apt-get purge ntp apt-get install ntpdate
ntpdate-debian which you might add to /etc/rc.local as well (after the hwclock command though) – just in case there is an Internet connection available during boot. And/or add it to /etc/cron.daily for example.Raspbian Jessy Lite will detect the DS3231 real time clock module automatically (as a DS1307 module but nevermind), there’s no need to whitelist or blacklist any I2C modules. There’s no need to run the i2cdetect command from the i2c-tools package. Once the clock module is detected, this line should be visible using dmesg:
# dmesg | grep rtc [ 6.640799] rtc-ds1307 1-0068: rtc core: registered ds3231 as rtc0
Check /proc/driver/rtc for more data on the RTC:
# cat /proc/driver/rtc rtc_time : 19:26:18 rtc_date : 2016-03-25 alrm_time : 00:00:00 alrm_date : 1970-01-01 alarm_IRQ : no alrm_pending : no update IRQ enabled : no periodic IRQ enabled : no periodic IRQ frequency : 1 max user IRQ frequency : 64 24hr : yes
While Huawei provides status information for its HiLink modems via a web page, this is hardly useful when using the modem on a headless Linux server. I just published a small Python-based command-line tool on Github which displays some useful information about the modem’s status.
root@wopr~#: python ./hstatus.py Huawei E3372 LTE Modem (IMEI: 121032526613216) Hardware version: CL1D3271AM Ver.A Software version: 22.286.53.01.161 Web UI version: 16.100.05.00.03-Mod1.4 Serial: L8FDW11512114431 MAC address (modem): 00:0D:87:12:1C:1D Connection status: Connected Network type: UMTS (3G) Signal level: ▁▃▄▆█ Roaming: Enabled Modem WAN IP address: 10.197.75.231 Public IP address: 185.13.106.181 DNS IP addresses: 212.113.0.5, 66.28.0.62 Network operator: Swisscom Connected for: 03:15:15 (hh:mm:ss) Downloaded: 615.17 KB Uploaded: 258.69 KB Total downloaded: 14.69 MB Total uploaded: 1.34 MB Unread SMS: 1
The tool has been tested on a Huawei E3276 and a E3372 modem. For the newer E3372 modem I had to add some code to supply a RequestVerificationToken in the HTTP header.
Feel free to send a pull request on Github with your own tweaks!
The repository is available here: github.com/trick77/huawei-hilink-status
I’ve been trying to get my Huawei E3276-150 4G/LTE USB modem to work with my PC Engines APU2 router board for quite a while. Once connected, the Huawei modem creates its own network thanks to its HiLink mode and is detected as an Ethernet over USB device in Linux. All I have to do is to ask the modem’s DHCP server for an IP address, use the modem as the default gateway and I should be all set. Right? Continue reading
Recently, I’ve replaced my five year old 13″ MacBook Air with a new budget Linux laptop. Since I’ve been using Linux desktops more and more there was just no point in buying another Apple product. The time was ripe to go fully Linux on my to-go laptop.
When I started evaluating laptops I quickly realized that even in 2016 it’s still nearly impossible to buy a new laptop with full Linux driver support. Linux and mobile hardware support is somewhat like a good red wine, it gets better over time. One notable exception is the Dell XPS 13 Developer Edition which comes with a preloaded Ubuntu Desktop operating system. I’m not sure if Dell still sells an updated version of the Developer Edition model but it was not available in my country and I didn’t want to shell out more than $800 for my new laptop anyway.
When it comes to portable computers, I’m probably kinda old-school. I don’t need a touch screen (fingerprints everywhere, eeek!) or some funky 2-in-1 tablet/laptop convertible like the Dell Yoga. Just a slim, lightweight 13″ laptop with a nice display (preferably matte), a battery-friendly processor, flash-based storage and a couple of full-size USB 3.0 ports for a decent price. After looking at a half dozen or so potential candidates I’ve set my eyes on the ASUS Zenbook UX305CA with the 6th gen. Intel Skylake mobile processor.
Since the default Quagga package in Ubuntu doesn’t have SNMP support enabled, the Quagga package has to be compiled locally. The following instructions may work for Debian as well but I only tested it in Ubuntu Server 14.04 LTS.
Now, edit /etc/quagga/daemons and enable at least zebra and bgpd and let’s create some empty config files for Quagga:
touch /etc/quagga/bgpd.conf ; touch /etc/quagga/zebra.conf
To enable SNMP support in Quagga, the line agentx has to be inserted into bgpd.conf and zebra.conf:
drfalken@wopr:/etc/quagga# head bgpd.conf
hostname AS65535
log file /var/log/quagga/bgpd.log
agentx
debug bgp events
debug bgp filters
debug bgp updates
router bgp 65535
bgp router-id 1.2.3.4
…
I wont dwelve into how to setup the SNMP daemon but don’t forget to add the following lines to the snmpd.conf configuration file and restart the SNMP daemon afterwards:
master agentx
agentxsocket /var/agentx/master
agentxperms 777 777
Make sure to set proper permissions for the agentx directory with a
chmod 755 /var/agentx/
or you will get error messages like snmp[warning]: Warning: Failed to connect to the agentx master agent ([NIL])
Once Quagga is able to connect to the local SNMP daemon, a message like this will show up in Quagga’s log file:
snmp[info]: NET-SNMP version 5.7.2 AgentX subagent connected
Monitoring BGP sessions works fantastically using LibreNMS. You can chose to receive push notifications and/or emails if a BGP session goes down/up or is flapping. However, there’s some tinkering involved to display 32-bit ASNs properly in LibreNMS (let me know in the comments if you’re interested) because the MIB only handles 16-bit integers. Unfortunately, there’s no IPv6 support in Quagga’s current SNMP implementation as well.
