As I mentioned in my previous post, the open source DNS forwarder Dnsmasq is ideal for the DNS part of DNS unblocking. I’m running Dnsmasq on a $30 Raspberry Pi credit card sized mini computer which is up 24/7 anyway since it also handles all VOIP phone calls at home. I point my Mac, Apple TV and iPad to the RPi as the primary DNS server.
On the server side, I’ve setup a HAProxy instance using just a single IP address as a proof of concept. This poor-man’s approach works beautifully with SNI-capable devices like my Mac and iOS devices. I think newer Android devices are SNI-compatible as well but I haven’t tested it. Windows 7 and up should be OK too. Older devices like the Playstation 3 or Xbox 360 are most likely not SNI-compatible and won’t work with my highly cost-efficient single IP address approach. Unfortunately, even some of the newest multimedia players don’t support SNI.
The HAProxy server is running on a lowend virtual private server in the U.S. As a starting point, feel free to use my proof of concept server as shown in the Dnsmasq configuration below. In the web browser, you should be able to watch Netflix, Hulu/HuluPlus, free episodes/TV shows on MTV, Disney XD, Syfy, NBC, ABC, Vevo, Crackle, PBS and CWTV. Netflix works on iPad and Apple TV too. HuluPlus could work on iOS as well.
Please remember, even though I’m planning to keep the HAProxy server up for some time to come, this is just a proof of concept and not a fully fledged DNS unblocking service.
On Debian-based Linux distributions, add the content below to a file named dnsmasq-catchall.conf in /etc/dnsmasq.d and it will get included by Dnsmasq. If Dnsmasq is running, i.e. on 192.168.178.99, you can test it using:
dig @192.168.178.99 trick77.com
This should bring up a few NS and A records for this site.
dig @192.168.178.99 abc.go.com
The result should be an A record for abc.go.com pointing to 199.204.184.146
/etc/dnsmasq.d/dnsmasq-catchall.conf:
address=/abc.go.com/199.204.184.146 address=/api.watchdisneyxd.go.com/199.204.184.146 address=/api.watchabc.go.com/199.204.184.146 address=/release.theplatform.com/199.204.184.146 address=/www.crackle.com/199.204.184.146 address=/api.crackle.com/199.204.184.146 address=/ios-api.crackle.com/199.204.184.146 address=/appletv.crackle.com/199.204.184.146 address=/ios-api-us.crackle.com/199.204.184.146 address=/ios-api.crackle.com/199.204.184.146 address=/android-api-us.crackle.com/199.204.184.146 address=/xboxone-api-us.crackle.com/199.204.184.146 address=/ps3-api-us.crackle.com/199.204.184.146 address=/roku-api.crackle.com/199.204.184.146 address=/content.uplynk.com/199.204.184.146 address=/content-us-east-1.uplynk.com/199.204.184.146 address=/www.crunchyroll.com/199.204.184.146 address=/api.crunchyroll.com/199.204.184.146 address=/static.discoverymedia.com/199.204.184.146 address=/www.dramafever.com/199.204.184.146 address=/token.dramafever.com/199.204.184.146 address=/link.theplatform.com/199.204.184.146 address=/s.hulu.com/199.204.184.146 address=/play.hulu.com/199.204.184.146 address=/www.iheart.com/199.204.184.146 address=/www.last.fm/199.204.184.146 address=/ws.audioscrobbler.com/199.204.184.146 address=/ext.last.fm/199.204.184.146 address=/www.logotv.com/199.204.184.146 address=/activity.flux.com/199.204.184.146 address=/j.maxmind.com/199.204.184.146 address=/mog.com/199.204.184.146 address=/www.mtv.com/199.204.184.146 address=/c.brightcove.com/199.204.184.146 address=/video.nbcuni.com/199.204.184.146 address=/video.nbc.com/199.204.184.146 address=/video.syfy.com/199.204.184.146 address=/signup.netflix.com/199.204.184.146 address=/www.netflix.com/199.204.184.146 address=/appboot.netflix.com/199.204.184.146 address=/cbp-us.nccp.netflix.com/199.204.184.146 address=/a248.e.akamai.net/199.204.184.146 address=/api-global.netflix.com/199.204.184.146 address=/movies.netflix.com/199.204.184.146 address=/movies1.netflix.com/199.204.184.146 address=/secure.netflix.com/199.204.184.146 address=/moviecontrol.netflix.com/199.204.184.146 address=/api.netflix.com/199.204.184.146 address=/api-us.netflix.com/199.204.184.146 address=/uiboot.netflix.com/199.204.184.146 address=/cbp.nccp.netflix.com/199.204.184.146 address=/ios.nccp.netflix.com/199.204.184.146 address=/xbox.nccp.netflix.com/199.204.184.146 address=/nccp-nrdp-31.cloud.netflix.net/199.204.184.146 address=/nintendo.nccp.netflix.com/199.204.184.146 address=/playstation.nccp.netflix.com/199.204.184.146 address=/nrdp.nccp.netflix.com/199.204.184.146 address=/android.nccp.netflix.com/199.204.184.146 address=/www.pandora.com/199.204.184.146 address=/mediaserver-sv5-rt-1.pandora.com/199.204.184.146 address=/tuner.pandora.com/199.204.184.146 address=/urs.pbs.org/199.204.184.146 address=/video.dl.playstation.net/199.204.184.146 address=/api.wipmania.com/199.204.184.146 address=/www.rdio.com/199.204.184.146 address=/www.smithsonianchannel.com/199.204.184.146 address=/once.unicornmedia.com/199.204.184.146 address=/media.mtvnservices.com/199.204.184.146 address=/www.spike.com/199.204.184.146 address=/udat.mtvnservices.com/199.204.184.146 address=/www.thewb.com/199.204.184.146 address=/www.cwtv.com/199.204.184.146 address=/media.cwtv.com/199.204.184.146 address=/pdl.warnerbros.com/199.204.184.146 address=/cdn.wwtv.warnerbros.com/199.204.184.146 address=/www.vevo.com/199.204.184.146 address=/sb.vevo.com/199.204.184.146 address=/videoplayer.vevo.com/199.204.184.146 address=/www.vh1.com/199.204.184.146 address=/screen.yahoo.com/199.204.184.146 address=/geo.yahoo.com/199.204.184.146 address=/mvid.yql.yahoo.com/199.204.184.146 address=/hls.video.query.yahoo.com/199.204.184.146 server=/beta.abc.go.com/8.8.8.8 server=/cdn.media.abc.go.com/8.8.8.8 server=/site.abc.go.com/8.8.8.8 server=/cdn.abc.go.com/8.8.8.8 server=/cdn.media.abc.go.com/8.8.8.8 server=/a.verdict.abc.go.com/8.8.8.8 server=/theview.abc.go.com/8.8.8.8 server=/adsatt.abc.go.com/8.8.8.8 server=/ll.static.abc.go.com/8.8.8.8 server=/static.east.abc.go.com/8.8.8.8 server=/share.mog.com/8.8.8.8 server=/logger.mog.com/8.8.8.8 server=/search.mog.com/8.8.8.8 server=/support.mog.com/8.8.8.8 server=/www.mog.com/8.8.8.8 server=/api.mog.com/8.8.8.8 server=/api.au.mog.com/8.8.8.8 server=/images0.mog.com/8.8.8.8 server=/images1.mog.com/8.8.8.8 server=/images2.mog.com/8.8.8.8 server=/images3.mog.com/8.8.8.8 server=/cdn2.mog.com/8.8.8.8
Once HAProxy is up and running, it will show a lot of statistics if you’ve enabled the built-in web interface:
Please let me know in the comments below once you have successfully set up your own DNS unblocking solution!
Thanks for this useful post
Can someone point me to an article that allows me to do this using DNSMasq and SNI Proxy instead of HAProxy?
I think you just need to point DNSMasq for the correct domains to the server running SNI proxy and you’re done
I’ve written an article in German on how to do this with DNSmasq, HAproxy, Ubuntu and also with some OpenWrt flavor: SmartDNS with HAproxy and DNSmasq (in German).
Greetings, Jan
If “Ed” is reading this: please resend me your email address using the contact form, the address got lost somehow in the contact form sent to me.
Cheers
Jan
My setup is dnsmasq and haproxy running on VPS, and just setting my DNS of the media PC to the VPS since I get 20ms ping times and I don’t use my HomeServer for general browsing.
Has anyone scripted anything to make it quick and easy to add more domains to the haproxy and dnsmasq files? I like plink (from putty) to automate stuff with very hacky and basic windows .bat files. Stuff like appending stuff to a file is cake, but adding stuff in the middle of files (haproxy conf) is out of my league. Something would need to sed or keep track of the last rule for each section?
Thanks very much for putting together these informative posts!
FYI, I’m going to skip automating it with .bat and plink, I’ll just pull up the following, search and replace the last one I added, and then paste into my ssh window:
sed -i.bkp ‘/backend b_catchall_http$/i \
use_backend b_catchall_http if { hdr(host) -i http://www.starzplay.com }\
‘ /etc/haproxy/haproxy.cfg
sed -i.bkp ‘/frontend f_catchall_https$/i \
use-server http://www.starzplay.com if { hdr(host) -i http://www.starzplay.com }\
server http://www.starzplay.com http://www.starzplay.com:80 check inter 10s fastinter 2s downinter 2s fall 1800\
‘ /etc/haproxy/haproxy.cfg
sed -i.bkp ‘/backend b_catchall_https$/i \
use_backend b_catchall_https if { req_ssl_sni -i http://www.starzplay.com }\
‘ /etc/haproxy/haproxy.cfg
sed -i.bkp ‘/^backend b_deadend_http$/i \
use-server http://www.starzplay.com if { req_ssl_sni -i http://www.starzplay.com }\
server http://www.starzplay.com http://www.starzplay.com:443 check inter 10s fastinter 2s downinter 2s fall 1800\
‘ /etc/haproxy/haproxy.cfg
/etc/init.d/haproxy restart
Note: I couldn’t figure out how to prevent some extra blank lines, but I’ve decided I could live with it and eventually manually clean it up.
Oh geez, I feel stupid.
So for anyone reading this, the correct approach would have been to add it to the config.json and regenerate the files, copy them to the right places and restart services.
D’oh. It’s done for me anyways.
got everything working right, have the non-sni based setup
all the sites work as they should, including xfinityTV, however the liveTV section of xfinitytv isnt working, im guessing they will need to be added to config.json
if anyone can give me the the few lines to add i would appreciate the help
Does any body has the configuration for hbogo and showtime anywhere?
This is great. I now have a cheaper solution for Netflix (which is 99% of my use).
I also like to be able to occasionally rent some new release movies when they are not yet available in Australia. Vudu doesn’t seem to work with this setup. I did wireshark the transaction using unblock-us to see what goes through and what doesn’t. I seemed to be able to get to the watching point and got a black screen. Is there another service that I can use that allows this (preferrably available on AppleTV or PS3) that I can use this setup for?
Jan- thank you for writing this up — but it falls short of my goal, which is your anti-goal. I want to route *all* netflix traffic through this and not just the geofenced features.
Why? Verizon/Comcast are intentionally letting Netflix underperform and routing traffic through a VPN/remote server overcomes the problem. So instead of logging into a VPN, I’d love to route my traffic through a remote VPS.
Could you help me identify the correct configuration to route all netflix.com traffic to the VPS?
Hello,
Thanks for this invention :)
I have the same question as fred, how to do the root level unblocking, I believe haproxy wont be able to do that, right?
BR, Sherif.
Is it true that sni pure works with ios devices? I am trying to run pandora and it works in the browser, but in my iphone it says “Cannot connect to Pandora”; I tried my vps server configuration and yours (199.204.184.146) with same results. Thanks!
wonderful, thank you.
Steve
Here’s how I do it:
curl https://raw2.github.com/exceliance/haproxy/master/scripts/haproxy.init.debian > /etc/init.d/haproxy
chmod +x /etc/init.d/haproxy
update-rc.d haproxy defaults
You may have to edit the start script and change the paths.
Cheers,
Jan
Thank you very much, managed to get my service installed and running between to raspberry pi’s – one in the US and one in Canada. Now, I’m looking for an easy way to restart haproxy when I make changes. Any advice is appreciated.
Steve
Jan,
thanks a lot for this guide – I got everything setup and working without a problem. However, as of yesterday Netflix thinks I’m based in UK. Has anything changed? How exactly does Netflix determine my location?
Cheers,
Martin
Rodrigo, I used Fiddler and tcpdump.
Cheers,
Jan
Jan,
Congrats, nice work.
How do u find out what domains and subdomains you should add???
For the moment I have found this tool https://github.com/guelfoweb/knock , but when using it, the list is pretty large…
Thanks
Rodrigo
Alessandro, the server on that IP address is running the poor-mans configuration using a single IP address. And you’re right about your 2nd question.
Cheers,
Jan
Hi, thanks for the nice post, what’s the 199.204.184.146 ip is for? The haproxy server running https://github.com/trick77/tunlr-style-dns-unblocking/blob/master/haproxy.conf ?
But that’s the single ip configuration right? For multiple ip I would need to change each domain ip to the corresponding haproxy frontend configuration right?
I’m in the same boat as others. I have a tomato router but, only want certain traffic at certain times, from specified PCs to go to US sites as needed. I have set up BIND9 on my VPS and now thanks to Jan :) haproxy. The only thing I would use my VPS for at the moment would be the DNS/Proxying browser wise.
I know a lot (ok ALL) of us noobs would greatly appreciate a how-to for dummies from ye olde linux gawds :)
@Fred: Address ties a domain name directly to an IP address, while server does a name server lookup.
@djvdorp: Another idea is to add the IP addresses and domain names to the /etc/hosts file (Linux/Mac), there is an equivalent for Windows as well. However, this would only work on your computer.
Cheers,
Jan
Get yourself a router and install Tomato. Dnsmasq is included. Eg Asus RT-N66U or RT-AC66U and there are even cheaper ones.
@Jan:
Thanks for the info and pointing out that my proposed solution wouldnt be ideal (my VPS is across the ocean indeed).
Do you happen to know other solutions when not running an in-home Pi for DNS tunneling? Can it be done on router / modem level for example? (dont have a tomato one unfortunately)
Right, I understand. But what is the difference between “address” and “server” in dnsmasq on the (local) router?
Some websites require DNS unblocking on the root domain level. This is my attempt to not route every subdomain to my proxy since it’s not necessary for unblocking and to conserve bandwidth.
Cheers,
Jan
What is the difference between “address” and “server” in dnsmasq on the (local) router?
Some services create random subdomains. One minute its ljgidk.subdomain.com, the other one it’s ljasdlq.subdomain.com. It is impossible to know these beforehand. That’s why I ask this ;-)
HAProxy doesn’t know http://myip.nl, only http://www.myip.nl as per your HAProxy configuration. While you’re sending the entire domain myip.nl (including all sub-domains btw.) to your HAProxy server. This works in sniproxy because sniproxy wildcard-tunnels the entire domain. Tunneling an entire domain is convenient but you’re essentially creating a VPN for the domain, tunnelling all traffic from/to it through sniproxy. Not necessarily a good idea because it kills the benefits of both, Anycast and Geocast. CDN-friendly DNS unblocking will only send the essential (sub-)domains required to pass the geo-fence through your proxy server.
Cheers,
Jan
Well I have done some testing. If I type server=/.myip.nl/MYVPS IP in dnsmasq on my router, going to http://www.myip.nl does NOT work. If I type address=/.myip.nl/MYVPS IP, it does work. (of course, I have previously added http://www.myip.nl to the haproxy.conf file). This is odd, because when I use SNIProxy, “server=” works just fine. Any thoughts, Jan?
What’s the difference between address and server in dnsmasq?
You can use my Dnsmasq setup on a remote server as well. However, I don’t think that’s a good idea since it increases latency for every DNS query a lot, particularly if there’s an ocean between you and the DNS server. Same applies to BIND btw.
Using a remote DNS server also messes with services which rely on Geocast (http://en.wikipedia.org/wiki/Geocast). As a much cheaper alternative to Anycast, the destination IP address of a geocasted service is determined by the geo-location of the DNS server. This can wreak havoc on your download rate! I’m sorry if this is too much geek speak but I’m all about bandwidth when it comes to DNS unblocking ;-)
Cheers,
Jan
Thanks for this useful post as well, was kinda waiting for it! Any chance you can also help people out who are willing to use BIND instead of DNSmasq (eg I dont run my DNS server in my house, but would like to use the HAProxy box as DNS server as well) as they do here:
http://corporate-gadfly.github.io/Tunlr-Clone/#your-own-dns-server