Auto-restart crashed mining processes in ethOS 1.2.7

I have been running a crypto-currency mining rig on the Linux based ethOS distro for quite some time now. While I realize that ethOS is problematic license-wise, it’s still a great distro to get a mining rig up and running in almost no time. The Nvidia GPUs in my rig are well tuned to operate at their optimum cost/hashrate ratio. However, due to bugs in the miner and/or the GPU drivers, every few days one of the GPUs stops mining. Sometimes ethOS is able to recover the GPU and gets it back to mining but sometimes it doesn’t seem to detect the crashed/hanging mining process at all. This is why I added a cron job that runs every 15 minutes and checks if all GPUs are still mining. If not, the miners will be re started using the minestop and minestart commands provided by ethOS.

The cron job starts the Bash script below. If it detects a problem, it writes to the console and additionally to /tmp/rigcheck.log. It’s been running smoothly on my ethOS v1.2.7 mining rig. I recommend putting it in /home/ethos/rigcheck.sh and don’t forget to add execute permissions using chmod +x /home/ethos/rigcheck.sh

The cron job can be created like this:

cat << EOF | sudo tee /etc/cron.d/rigcheck
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
*/15 * * * *   root    /home/ethos/rigcheck.sh
EOF

Thanks to this script, crashed or hanging miners will be restarted fairly quickly and my rig’s pool-reported hashrate stopped dropping in such situations.

How to display crypto currency rates on macOS menu bar

BitBar has been around for a while but I didn’t notice it until I wanted to display crypto currency rates (who isn’t into crypto these days :)) on my macOS menu bar. BitBar is as simple as it can get: it takes the output from a shell script and displays the result on the menu bar. What’s really cool about BitBar is the support for Base64 encoded images which get displayed as icons on the menu bar.

This is how my “crypto menu bar” looks like:

I just put all my BitBar shell scripts into ~/bitbar and that’s it.

Here are BitBar shell scripts for Monero, Ether and Bitcoin (against the €). Since the icons are just 16×16 they probably look crappy on a retina display and have to be replaced with larger icons.

Gigabyte Z97X-UD5H and USB 3.0 in macOS Sierra 11.12

2000px-usb_icon-svgA commenter recently asked if I had any USB 3.0 related issues with my Gigabyte Z97X-UD5H equipped Hackintosh. Since every USB port was working out of the box I thought everything was fine. However, having a closer look at the USB section in macOS Sierra’s System Information revealed that none of the USB 3.0 ports were operating at USB 3.0 speeds, they were all capped at 480 Mb/sec.

Here’s how I was able to get USB 3.0 speed back:

  1. Inspired by https://github.com/RehabMan/OS-X-USB-Inject-All/blob/master/config_patches.plist I added the entire “Patches” section to my clover.plist. From the “KextsToPatch” section I only added the “Change 15 port limit to 20 in XHCI kext (9-series)”  related to macOS 11.12 and removed the disabled line.
  2. I downloaded https://bitbucket.org/RehabMan/os-x-fake-pci-id/downloads and copied FakePCIID.kext and FakePCIID_XHCIMux.kext to Clovers kext directory.
  3. I made sure XHCI mode was set to “Smart Auto” and both, XHCI and EHCI hand-off were enabled in the BIOS.

All front and back panel USB 3.0 ports are now reporting 3.0 speeds when connecting a 3.0 compatible device:

JSM578:
  Product ID:	0x0578
  Vendor ID:	0x152d  (JMicron Technology Corp.)
  Version:	2.03
  Serial Number:	DB123456789B
  Speed:	Up to 5 Gb/sec
  Manufacturer:	JMicron
  Location ID:	0x14f00000 / 20
  Current Available (mA):	900
  Current Required (mA):	896
  Extra Operating Current (mA):	396

Major kudos to RehabMan for providing these easy-to-use injector kexts!
Since injector kexts are not drivers and thus do not have to be signed, my Hackintosh still runs with maximum system integrity protection (SIP):

Jans-Mac:~ jan$ csrutil status
System Integrity Protection status: enabled.

Random delay for cron.daily, cron.weekly, cron.monthly

cron-logoWouldn’t it be nice if cron’s daily, weekly and monthly jobs could be run with a slight offset? At least that’s what I thought when 20+ servers were hitting my backup infrastructure at once. The scripts in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly are triggered directly from crontab at fixed times. Here’s what /etc/crontab looks like in Ubuntu Server 16.04:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

I’ve found several tips which suggested to use a RANDOM_DELAY variable in crontab. Unfortunately, this variable doesn’t seem to be implemented in Debian/Ubuntu’s version of crontab at this time. I even checked the source code, there’s no RANDOM_DELAY variable to be found.

Here’s the solution I came up with. I’m using a combination of sleep and numrandom with a time range between 0 and 30 minutes.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	sleep `numrandom /0..30/`m ; test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

In order to use the numrandom command, you have to apt-get -y install num-utils it first.

I didn’t delay the cron.hourly execution but the same sleep/numrandom combo could be used for it as well, just maybe replace the m (minutes) with s (seconds).

How to generate daily PowerDNS statistics reports

daae8fe19d3ab224f3a104f987acd8bfPowerDNS has been powering authoritative DNS lookups to this web site for quite a while now. It’s such a remarkable piece of software. Here’s how to create a daily statistics report for PowerDNS.

  1. Put the lines below in /etc/cron.daily/powerdns-report:
    #!/bin/sh
    /usr/bin/curl -s localhost:8081 | mail -s "$(echo "PowerDNS Daily Report\nMIME-Version: 1.0\nContent-Type: text/html")" root
  2. Make sure the file is executable: chmod +x /etc/cron.daily/powerdns-report
  3. Enable the internal web server (defaults to 127.0.0.1:8081) in the PowerDNS configuration file:
    webserver=yes
  4. Restart PowerDNS
  5. Make sure all Mails to root are forwarded to your e-mail account (or replace the recipient in the powerdns-report script)

This is how the PowerDNS statistics report will look like:

PowerDNS 4.1.0
Uptime: 3.94 days
Queries/second, 1, 5, 10 minute averages: 0.402, 0.212, 0.186. Max queries/second: 1.17
Cache hitrate, 1, 5, 10 minute averages: 45.0%, 33.4%, 30.9%
Backend query cache hitrate, 1, 5, 10 minute averages: 55.1%, 41.7%, 38.3%
Backend query load, 1, 5, 10 minute averages: 0.343, 0.317, 0.32. Max queries/second: 2.37
Total queries: 52665. Question/answer latency: 1.25ms

Log Messages
gmysql Connection successful. Connected to database 'pdns' on 'localhost'.	422	27.8%
AXFR of domain 'nogo.com.' initiated by 231.91.120.148	70	4.6%
AXFR of domain 'nogo.com.' failed: 231.91.120.148 cannot request AXFR	70	4.6%
AXFR of domain 'nogo.com' denied: client IP 231.91.120.148 has no permission	70	4.6%
Rest:	527	34.7%
Total:	1519	100%

Queries for existing records, but for type we don't have
wopr.com./AAAA	2193	47.1%
wopr.me./AAAA	503	10.8%
drfalken.me./AAAA	482	10.4%
mail.wopr.net./AAAA	281	6.0%
wopr.net./AAAA	253	5.4%
Rest:	685	14.7%
Total:	4653	100%

Queries for non-existent records within existent domains
dn42-fr.wopr.me./A	359	24.9%
_sip._udp.sbc.sipmly.com.wopr.net./SRV	335	23.2%
_sip._udp.sip.ovh.fr.wopr.net./SRV	214	14.8%
dn42-us.wopr.me./A	75	5.2%
_adsp._domainkey.woprs.net./TXT	14	1.0%
Rest:	290	20.1%
Total:	1443	100%

UDP Queries Received
wopr.com./A	2307	23.1%
wopr.net./A	2227	22.3%
trick77.com./AAAA	702	7.0%
wheniskeynote.com./AAAA	578	5.8%
www.trick77.com./A	293	2.9%
Rest:	3135	31.4%
Total:	10000	100%

Queries that could not be answered due to backend errors
Total:	0	100%

Queries for domains that we are not authoritative for
nogo.com./SOA	190	84.4%
cpsc.gov./ANY	16	7.1%
dnsscan.shadowserver.org./A	4	1.8%
./ANY	3	1.3%
cpsc.gov./A	2	0.9%
defcon.org./ANY	2	0.9%
./NS	1	0.4%
4caa2d31.openresolvertest.net./A	1	0.4%
c.afekv.com./A	1	0.4%
doc.gov./ANY	1	0.4%
Rest:	4	1.8%
Total:	225	100%

Remote server IP addresses
137.24.55.110	289	2.9%
164.8.230.15	108	1.1%
8.91.147.135	106	1.1%
8.91.150.198	103	1.0%
28.1.240.64	100	1.0%
163.19.151.12	87	0.9%
15.16.90.42	84	0.8%
141.28.51.41	73	0.7%
169.16.124.226	70	0.7%
164.891.213.199	63	0.6%
Rest:	8917	89.2%
Total:	10000	100%

Remote hosts sending corrupt packets
219.116.116.12	1	50.0%
116.125.42.131	1	50.0%
Total:	2	100%

Remote hosts querying domains for which we are not auth
8.194.50.198	70	31.1%
8.194.47.135	68	30.2%
163.19.151.12	52	23.1%
185.59.233.131	15	6.7%
88.65.195.128	2	0.9%
64.32.96.66	2	0.9%
41.12.122.91	1	0.4%
45.20.24.195	1	0.4%
15.52.206.174	1	0.4%
89.146.222.158	1	0.4%
Rest:	12	5.3%
Total:	225	100%

Variables
corrupt-packets	2	Number of corrupt packets received
deferred-cache-inserts	1042	Amount of cache inserts that were deferred because of maintenance
deferred-cache-lookup	250	Amount of cache lookups that were deferred because of maintenance
dnsupdate-answers	0	DNS update packets successfully answered.
dnsupdate-changes	0	DNS update changes to records in total.
dnsupdate-queries	1	DNS update packets received.
dnsupdate-refused	1	DNS update packets that are refused.
incoming-notifications	0	NOTIFY packets received.
packetcache-hit	16206	
packetcache-miss	36486	
packetcache-size	1252	
query-cache-hit	52387	Number of hits on the query cache
query-cache-miss	97378	Number of misses on the query cache
rd-queries	212	Number of recursion desired questions
recursing-answers	0	Number of recursive answers sent out
recursing-questions	0	Number of questions sent to recursor
recursion-unanswered	0	Number of packets unanswered by configured recursor
security-status	0	Security status based on regular polling
servfail-packets	0	Number of times a server-failed packet was sent out
signatures	377	Number of DNSSEC signatures made
tcp-answers	497	Number of answers sent out over TCP
tcp-answers-bytes	434810	Total size of answers sent out over TCP
tcp-queries	299	Number of TCP queries received
tcp4-answers	485	Number of IPv4 answers sent out over TCP
tcp4-answers-bytes	432435	Total size of answers sent out over TCPv4
tcp4-queries	287	Number of IPv4 TCP queries received
tcp6-answers	12	Number of IPv6 answers sent out over TCP
tcp6-answers-bytes	2375	Total size of answers sent out over TCPv6
tcp6-queries	12	Number of IPv6 TCP queries received
timedout-packets	0	Number of packets which weren't answered within timeout set
udp-answers	52665	Number of answers sent out over UDP
udp-answers-bytes	4955023	Total size of answers sent out over UDP
udp-do-queries	42490	Number of UDP queries received with DO bit
udp-queries	52665	Number of UDP queries received
udp4-answers	40096	Number of IPv4 answers sent out over UDP
udp4-answers-bytes	3508260	Total size of answers sent out over UDPv4
udp4-queries	40096	Number of IPv4 UDP queries received
udp6-answers	12569	Number of IPv6 answers sent out over UDP
udp6-answers-bytes	1446763	Total size of answers sent out over UDPv6
udp6-queries	12569	Number of IPv6 UDP queries received
key-cache-size	12	Number of entries in the key cache
latency	1251	Average number of microseconds needed to answer a question
meta-cache-size	61	Number of entries in the metadata cache
qsize-q	0	Number of questions waiting for database attention
real-memory-usage	56033280	Actual unique use of memory in bytes (approx)
signature-cache-size	179	Number of entries in the signature cache
sys-msec	37296	Number of msec spent in system time
udp-in-errors	0	UDP 'in' errors
udp-noport-errors	45	UDP 'noport' errors
udp-recvbuf-errors	0	UDP 'recvbuf' errors
udp-sndbuf-errors	0	UDP 'sndbuf' errors
uptime	340341	Uptime of process in seconds
user-msec	57664	Number of msec spent in user time
© 2013 - 2016 PowerDNS.COM BV.