Since Tunlr closed down unexpectedly this week, I decided to publish my ideas and findings on the subject of DNS unblocking. I used Tunlr for some time when I decided to develop my own, private DNS unblocking solution last year.
Why VPNs are no good for streaming
DNS unblocking refers to a technique used to circumvent geo-fenced Internet services without the use of a VPN. When we’re using a VPN to access geo-fenced websites, usually all our Internet traffic gets routed through a remote VPN server. With DNS unblocking, only selected traffic gets routed through a remote proxy server, ideally just the minimum traffic required to trick geo-fenced services like Pandora, Netflix or Hulu into “thinking” our current geolocation is within the United States (or any other country required to pass the geo-fence). One advantage is that DNS unblocking works for all devices that allow custom DNS settings while a VPN only works on a computer or in the router. But the big advantage over a VPN is that DNS unblocking allows the full and intended use of Content Delivery Networks (CDN).
Without going too far into the subject, CDNs usually rely on BGP Anycast or Geocast to find the closest destination server. Here’s a real life example for Anycast: if we ping Google’s DNS server (8.8.8.8), we will usually get a response within, let’s say 30ms or less, no matter where we are in the civilised world. This is because Google operates many DNS servers responding to the same 8.8.8.8 IP address, distributed all over the globe. The announced routes for 8.8.8.8 and the path vector protocol BGP make sure we’re getting the one closest (with the shortest AS path) to us. While using a VPN, we will get the Google DNS server closest to the remote VPN server, which can be far away on a different continent. Far away is bad for bandwidth, and bandwidth is important for high quality video streams!
If a CDN is using Geocast, you will get the destination server closest to the DNS server, not the one closest to your real location. If you’re resolving DNS queries through a (far away) DNS server, you’re essential killing the benefits of Geocast. This will wreak havoc on your download rates and increases latency for every DNS request. The latter applies to every scenario which includes a DNS server which is far away, including DNS unblocking. You should always use the DNS server with the lowest latency – which in almost every case is your provider’s DNS server.
On-demand Internet streaming providers like Netflix rely heavily on CDN technology to transport their video streams to the end user.
And then there is the HTTPS tunnelling problem
There are many ways to tunnel a HTTP connection through a proxy. We could use Nginx for instance, or Squid. Even Apache comes with a HTTP proxy module. However, it gets a bit more difficult once we have to tunnel a HTTPS connection without terminating the SSL certificate in the proxy. As of today, none of the previously mentioned software products are able to tunnel a HTTPS connection without SSL termination.
Another problem are IP addresses. In the old days, every SSL endpoint required a dedicated IP address. Thanks to Server Name Indication (SNI), a client is able to present the desired domain name to a server during the initial SSL handshake. Unfortunately though, SNI only works in more recent browser versions and just a few standalone multimedia devices, iOS devices being among them. If we want to tunnel non-SNI-capable devices through a HTTPS proxy, we will have to use a dedicated IP address for every SSL tunnel.
Let’s go back to the HTTPS tunnelling problem. There are a few solutions available in the open source marketplace but absolutely none of them come even close to HAProxy. HAProxy is the mother of all proxies. Among a myriad of other things, HAProxy is able to tunnel HTTPS connections, SNI-based or not, and it does this… wait for it……. without SSL termination! It will just passthrough any connection we throw at it. HAProxy is incredibly fast, unbelievably lightweight and very reliable. It’s so stable I’m even using snapshot versions from the development branch in production environments (YMMW, that’s just me).
Let’s use HAProxy for DNS unblocking!
Here’s a sample HAProxy configuration which includes support for Pandora, Netflix, Hulu, MTV, ABC and quite a few others. You can’t use it without modification. It’s best to start with the proxies you need and to throw away the parts you don’t need. I’m probably not going to maintain it on a regular basis but feel free to fork it on Github.
# Check the HAProxy documentation for information about the configuration keywords.
# Make sure to use (compile) the latest HAProxy version from the current development branch or some features may not work!
# Please see https://trick77.com/2014/03/01/tunlr-style-dns-unblocking-pandora-netflix-hulu-et-al/ for more information.
# *** THIS CONFIGURATION WILL NOT RUN WITHOUT PROPER MODIFICATION ***
global
daemon
maxconn 20000
user haproxy
group haproxy
stats socket /var/run/haproxy.sock mode 0600 level admin
log /dev/log local0 debug
pidfile /var/run/haproxy.pid
spread-checks 5
defaults
maxconn 19500
log global
mode http
option httplog
option abortonclose
option http-server-close
option persist
option accept-invalid-http-response
timeout connect 20s
timeout server 120s
timeout client 120s
timeout check 10s
retries 3
listen stats # Website with useful statistics about our HAProxy frontends and backends
bind *:6969
mode http
stats enable
stats realm HAProxy
stats uri /
stats auth haproxy:secure_password_goes_here
# SNI catchall ------------------------------------------------------------------------
# We're trying to save as many IP addresses as possible that's why we're running as many backends as possible on one IP address.
# Obviously, we're using SNI on the 443 frontend only
frontend f_sni_catchall
mode http
bind ip_address_1_here:80
log global
option httplog
option accept-invalid-http-request
capture request header Host len 50
capture request header User-Agent len 150
#--- abc
use_backend b_sni_catchall if { hdr(host) -i abc.go.com }
use_backend b_sni_catchall if { hdr(host) -i api.watchdisneyxd.go.com }
use_backend b_sni_catchall if { hdr(host) -i api.watchabc.go.com }
#--- mylifetime
use_backend b_sni_catchall if { hdr(host) -i c.brightcove.com }
#--- cbs
use_backend b_sni_catchall if { hdr(host) -i release.theplatform.com }
#--- crackle
use_backend b_sni_catchall if { hdr(host) -i www.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i api.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i ios-api.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i ios-api-us.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i appletv.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i android-api-us.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i xboxone-api-us.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i ps3-api-us.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i roku-api.crackle.com }
use_backend b_sni_catchall if { hdr(host) -i content.uplynk.com }
use_backend b_sni_catchall if { hdr(host) -i content-us-east-1.uplynk.com }
#--- crunchyroll
use_backend b_sni_catchall if { hdr(host) -i www.crunchyroll.com }
use_backend b_sni_catchall if { hdr(host) -i api.crunchyroll.com }
#--- discovery
use_backend b_sni_catchall if { hdr(host) -i static.discoverymedia.com }
#--- dramafever
use_backend b_sni_catchall if { hdr(host) -i www.dramafever.com }
use_backend b_sni_catchall if { hdr(host) -i token.dramafever.com }
#--- fox
use_backend b_sni_catchall if { hdr(host) -i link.theplatform.com }
#--- hulu
use_backend b_sni_catchall if { hdr(host) -i s.hulu.com }
#--- iheart
use_backend b_sni_catchall if { hdr(host) -i www.iheart.com }
#--- last.fm
use_backend b_sni_catchall if { hdr(host) -i www.last.fm }
use_backend b_sni_catchall if { hdr(host) -i ws.audioscrobbler.com }
use_backend b_sni_catchall if { hdr(host) -i ext.last.fm }
#--- logotv
use_backend b_sni_catchall if { hdr(host) -i www.logotv.com }
use_backend b_sni_catchall if { hdr(host) -i activity.flux.com }
#--- netflix
use_backend b_sni_catchall if { hdr(host) -i www.netflix.com }
use_backend b_sni_catchall if { hdr(host) -i appboot.netflix.com }
use_backend b_sni_catchall if { hdr(host) -i cbp-us.nccp.netflix.com }
use_backend b_sni_catchall if { hdr(host) -i a248.e.akamai.net }
...
Please see the full configuration source on Github.
And where’s the DNS part in DNS unblocking?
I’m using Dnsmasq on my Rasperry Pi to “intercept” the domain names from my HAProxy configuration and forward all other DNS queries to my ISP’s DNS server. You could use BIND as a local caching DNS server as well but you would end up writing many DNS zone files. Dnsmasq is a lot easier to setup. I will publish a sample Dnsmasq configuration for DNS unblocking in a future post.
EDIT: Here it is.
So you want to start a DNS unblocking company?
Good luck, you’re a bit late to the party. You will need DNS servers, lots of IP addresses, redundancy for everything, a way to deal with Akamai’s geo-protected transport streams (=lots of bandwidth!), a frontend for clients (WHMCS comes to mind) and many other things. Please do me and the Internet a favour and make sure your open DNS servers are rate-limited. All professionally operated, open (recursive) DNS server do have some sort of rate-limitation to make them less interesting (and harmful) in DNS amplification DDoS attacks. Hackers, or rather, script kiddies, permanentely scan the Internet for open, recursive DNS servers and they will find your DNS server within hours.
You my try out ultra fast vpn which unblock Pandora, Netflix, Hulu et al and much mores
Anyone got Pandora working on Samsung Smart TV?
Awesome work Trick77 with dockerflix! Anyone can help point me in the right direction to using selective IPs to only use dockerflix with and my Tomato firmwared router? I have a bunch of Rokus kicking around but want to only direct some of them to the US IP of my VPS…..
hi and thanx for the great tutorial, befor i try it i want to make sure of something :
i want to set it up not to watch netflix or other streaming services, all i want is to use the sites that gives me the annoying (403) error cos of my ip location and to be able to download form google playstore and apple appstore and so on .
will this work for me ? and thanks again for ur hard work
I think you need VPN not Smart DNS
ya i do have vpn but i dont like to open vpn every time i get the 403 error its annoying that is why am asking if smartDns can do that
Anyone got Hulu, HBO GO and ABC work on Apple TV, Fire TV and iPad?
Whoopz – A reboot of the server took care of the problem.
Strange …..
Is it only me or is Netflix down ?
My haproxy stats says – appboot.netflix.com down for a day
Bingo
Anyone succeeded to get it work on Android or iPhone
It works fine for me on PC & Windows Phone
I mean Hulu ;)
Solved :)
For anyone that have the same problem remove all the servers and just keep this one “s.hulu.com” in both DNS & HAProxy
In HAProxy no https just http only :D
I currently have Netflix working in 10 regions (Canada / Usa / Uk / Nl / Germany / France / Brazil / Sweden / Luxembourg and Australia (March 24) ) and I’m thinking about starting a service to allow others to connect to my haproxy server’s around the world in order for you and whoever else you allow on your dns to watch netflix in the various different regions. I’m also interested in being able to add more regions but of course this costs money / time. I’m wondering if anyone would be interested in this or is there anyone else that is currently doing this? You can contact me for more information at toadyus@gmail.com. If this is against any rules, I apologize in advance.
@roger It’s depending on your provider if he have working IPs or not
If you want send me an email and I’ll give you a name of working provider (not in public)
email:black.dragon90000@gmail.com
Have never tried Hulu
What are you guyzz using for watching HULU ?
Does Hulu cost $$ (require a subscription)
My Netflix are still runing strong
TIA
/Bingo
Anyone got lucky with getting Hulu working
Netflix & Hulu works for me last I checked. What’s happening?
Hulu is blocking VPS providers IPs :(
But I got a working IP now and it’s working
How did you get your vps working … Did you change providers or just get them to give you a new ip? I ask because mine just stopped after over 1 year working great.
@Bingo,
Is your Hulu working with your setup? Mine is not, but Netflix is.
Thanx for this nice setup & “Generator”
I Just got a US based SNI for 10$/yr , and setup a Debian machine using dnsmasq & HAproxy.
I decided to run dnamasq & HAproxy , both on the SNi Server.
So all i need is to change the DNS on my clients.
In order to prevent “mis-use” i locked down the server via a
“petty tight” iptables setup.
I suppose the “advice” of not exposing UDP/53 , is ok to break if you control the clients allowed to access the Server, per host/ip via iptables.
Thank you Jan et all.
Bingo
Absolutely, a private/protected recursive resolver can’t be used in amp attacks.
Cheers,
Jan
Hi,
I am unable to get Hulu working on Apple TV. Was working but completed blocked today..
Also can HBOGO work as well? I have tried the configs but I am stuck.
@Paul I have noticed this too, only once so far (and I’ve been using this for many months), but I have no idea what causes it so far. Maybe you can set a nightly cronjob to restart haproxy? It’s not the prettiest solution but it’ll probably work.
Happy new year!
This is still going strong, thanks for the tips on setting this up. I have one niggling fault that keeps occurring and wondered if anyone else has the issue.
After a period of time, maybe a week or two one of the netflix connections will go red and drop in haproxy stats interface – usually appboot.netflix.com. The connections from end clients stop working until haproxy is restarted and then everything is back to normal until the next time.
Is this a config issue in haproxy – not recovering the connection when netflix presumably goes down for a short time? Anyone else see this?
Thanks, Paul.
I can use my haproxy to get to hulu from pc but not roku
Also my apple tv gives me usa netflix but no icon for hulu.
Something I did wrong?
Both posts above can be deleted — turns out restarting haproxy fixed everything, it seems some ports stopped responding, but not others.
Excuse me, it’s not stopped worked entirely, but it seems to have stopped working on the Wii (1) and the Chromecast for me. Wii says it’s unable to connect, the chromecast seems to crash when I start netflix on it now.
If I remove the hack, it works (but it’s not the US netflix, obviously).
It still seems to work on my Android phone at least.
This hack and the non-SNI hack seems to have stopped working entirely for me, is it just me, or anyone else?
Gary, yes, this will work as well if you use the proper addresses to send traffic to a U.K. proxy. At this time, the config generator just supports 1 remote proxy. I’ve been planning to extend the generator to support multiple server locations but I haven’t had the time to do so.
Cheers,
Jan
This has really inspired me. Thank you so much.
Will this work the other way around, USA to the UK for websites like BBC IPlayer and ITVPlayer?
I am a bit late to the smart DNS party but my business partners and I own some of the top websites for watching British TV abroad.
Jan, what in the config would cause a 503 Service Unavailable?
Can’t get to http://hulu.com or http://www.hulu.com
thanks for the comment jan but I used iprules on my router which worked with unblockus but not with my local rpi+ remote vps
solution
Rooting Chromecast may help setting a non-Google DNS server:
http://hackaday.com/2014/08/25/chromecast-is-root/
Thanks for the article Jan .Why using a VPN when you can use a DNS service like UnoTelly? The DNS option is much faster and easier to use.
i think there is one service missing in the list for the chromecast connection to netflix as it tries to load and stops at 13-15%
i have exactly the same problem @lucac81
chromecast and hulu not working
Nice guide, it works flawlessly in pure-sni mode
I’m still having trouble with chromecast devices and netflix, but I cannot say at the moment if it’s a problem with the hardcoded DNS servers (tried to re-route them to my local dnsmasq server but it still get stuck to switching controllers screen) or it’s because it doesn’t supports SNI.
Also hulu doesn’t seems to work (it detects that I’m coming from some sort of anonymizer)
A quick question, just in case, how did you extracted the whole host list that haproxy uses? for example if we need to add some new service, how could we discover which url to use?
Ok, I have this all working great, thank you very much for the info. However I have a bit of a noob question. I don’t know much about linux security, and when using the genconf.php script, I get the following message when it has completed :-
If you are using an inbound firewall on xxx.xxx.xxx.xxx:
/sbin/iptables -A INPUT -p tcp -m state –state NEW -d xxx.xxx.xxx.xxx –dport 27199 -j ACCEPT
/sbin/iptables -A INPUT -p tcp -m state –state NEW -m multiport -d xxx.xxx.xxx.xxx –dports 80:443 -j ACCEPT
File generated: haproxy.conf
File generated: dnsmasq-haproxy.conf
***********************************************************************************************
Caution: it’s not recommended but it’s possible to run a (recursive) DNS forwarder on your
remote server xxx.xxx.xxx.xxx. If you leave the DNS port wide open to everyone,
your server will get terminated sooner or later because of abuse (DDoS amplification attacks).
***********************************************************************************************
Can someone advise how to setup a debian 6 system so that it is secure from a DNS side of things as suggested in this warning information? Do I just run the commands to add the entries to the inbound firewall (how do i setup an inbound firewall?) and how do I make sure that I don’t enable my system to be a DDoS tunnel?
Cheers,
Dom
Thank Jan for this.
I have managed to get this working using pure-sni. I appreciate all your hard work.
I’m guessing that the PS3 has been updated or perhaps the individual apps, as I am able to watch netflix on the PS3. I can’t get Vudu or Crunchyroll going on the PS3. I have used wireshark (i’m not an expert) to locate secure.vvond.net which looks to need to be added but both these services still fail to work.
I can use netflix and crunchyroll on the AppleTV but there is no Vudu there (and I like the feature to download to watch later).
Anyone know if there are more addresses I need to add to get crunchyroll or Vudu working? Maybe I should give the non-sni configuration a go?
Thanks.
Thanks a ton for this guide. I got this up and working. Sort of…
SSLLabs also shows me my servers ip when I check for netflix. Dig and nslookup show me my server ip when I check, so DNSMasq is all cool.
HAProxy also shows requests coming in the fronted, but all the numbers are zero in the backend.
Help please?
So much info, but also so complex :/
I’m only interested as I would like to continue viewing Netflix now that Tunlr has gone down.. Pretty much looking for an alternative to giving UnoTelly or Unlocator all my cash, on top of the subscription I’m already paying for.
I see a lot of trial and error going on as well… Any change one of you tech savvy guys has a working configuration for this apps ‘n all that could just be zipped up… Pushing my luck here I know ;)
Thanks for sharing this!
I had to add tuner.pandora.com to frontend and backend of catchall to get Pandora working on my Android phone.
Thanks for sharing your efforts.
In the ‘poor-mans-haproxy.conf’ I needed to add ‘play.hulu.com’ in the catchall_sni front and backend to get that service working.
Fixed. Thanks for the heads up, Mike.
@Konstantinos: Check out this project on Github, it may be more end user friendly: https://github.com/corporate-gadfly/Tunlr-Clone
@Bobby: I think you may need one of the 1.5 dev version because some features are not available in earlier versions. You can contact me at my_firstname (at) this_sitename.
Cheers,
Jan
Hey guys is there, by any luck, anyone with a step-by-step tutorial on how to implement this DNS Unblock service?
I have my own VPS with a friend but we couldn’t get anywhere with just the config files (I know. We suck).
So is there anyone at least to help me out at the very least?
I would appreciate it so much!
Thanks in advance!
Hi Jan,
Thanks for the very detailed post. Is there a way to make this work with the stable version of HAProxy?
Also, is there a way to contact you directly or could you send me an email? I wanted to discuss something privately with you.
Regards,
Bobby
How demanding is this ‘VPS’ side of things? Would I be able to run an install on a Raspberry Pi, say, and leave it running at a mate’s house instead of renting a VPS?
I wouldn’t think that the residential ADSL2 line speeds would be much of an issue as my understanding is that only a small amount of data is tunnelled, not the whole video stream – is that correct?
As I only want BBC iPlayer access I would just forward *.bbc.co.uk to the remote instance using my router’s dnsmasq config as I did previously when I used Tunlr so my general lookups shouldn’t be impacted by this setup providing the little bit of proxying isn’t too demanding.
Just to address a couple of the questions here, tunneling and tunnelling are both correct. They are part of the English language rules per region. USA tend to change English word spellings, and the UK stand by the original spelling. Canada(where I am) take the British way of spelling (as well as how to pronounce the last letter of the alphabet. The americans take out the “u” in many words like labour, honour, colour, and so on. They have gone so far as to alter the liquid measurements. In Canada, we have 20 ounces in our pints, the states have 16. We have 160 ounces in a gallon, the states have 128. Used to be funny listening to American teenagers in the 70’s go on about how Canadian gas gave them great mileage.
As for why Netflix does not seem interested in stopping people using vpn or dns to view US content from Canada.(I will use me as the example). Reason is likely simply because when they license the rights to broadcast something, it is only licensed for the one country. But, if they show their suppliers(not sure what the people that Netflix licenses from are called) that they are taking reasonable action to prevent people from outside the license area from watching movies licensed only for viewers in the states, they are likely fulfilling their contractual agreement. Netflix wants people to join, pay and watch, if Canadians can get the better movies by using a service, and they are still paying Netflix, why would Netflix mind? Now once these movie people hire someone that actually understands networking, that may change.
The people that will complain the most though, are the ones that have the rights to distribution in Canada. They lose money on it.
Hi, I like your concept and it is pretty thorough, I kinda of tweaked it and built my own configuration for haproxy as well as dnsmasq by binding multiple ips on the vpn connection and routing the http/https/dns queries that are necessary over this vpn connection. My problem now though, is that I have my sony blu-ray player showing the american only content, yay, except when I try to stream any of it, such as hulu plus it gives me an error. To get it this far I added some of the sony.tv and sony.net urls that the blu-ray requests to the haproxy dnsmasq config which got it this far. Any advice would be appreciated, willing to share my additional configs with you if you want for the help I am asking.
Ok will do that now Jan (many thanks btw!) On another note I rebooted my server and Apache2 came back…killed it off manually. I’ll work on that after I get haproxy going.
You need to set up a DNS forwarder as well :-) Have a look at my more recent post.
Cheers,
Jan
LOL literally just did the purge on Apache :)
Thanks for the auto start add. Firefox still just shows connection reset :(
Is there anything we need to add to get the proxy to handle the requests and send them out via the US IP? From what I know buyvm VPS have an external (199) address and an internal US (173) address.
You may want to uninstall Apache and set HAProxy to auto-start on reboot.
apt-get purge apache2
update-rc.d haproxy defaults
So I “kill”ed off Apache and restarted haproxy and now get:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8593/haproxy
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1675/sshd
tcp 0 0 0.0.0.0:6969 0.0.0.0:* LISTEN 8593/haproxy
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1942/sendmail: MTA:
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8593/haproxy
tcp 0 0 127.0.0.1:587 0.0.0.0:* LISTEN 1942/sendmail: MTA:
tcp6 0 0 :::22
Your netstat (now added to my notes) was better than my netstat (netstat -an | egrep ‘Proto|LISTEN’) :P. Apparently I have Apache2 on port 80 running at PID 2042….
It won’t work with 8080. You need to free up 80 or find out why you can’t bind to 80.
netstat -tulpn is your friend.
Cheers,
Jan
OK I changed port 80 to 8080 instead. No errors when I started the server….the stats page is working even. But, the proxy itself isn’t. Using VPSIP:8080 in FireFox gives connection reset errors. I have a feeling some work with iptables is needed right?
You actually don’t really need to do this because it’s highly unlikely someone finds out you’re running this kind of proxy on that particular IP address. If someone’s asking for a website not known in your HAProxy configuration it will just return a HTTP 503 error.
ok I’ll change the IPs back to *. Nothing should be using port 80 as this is a brand new Debian 7 install with just nano and haproxy installed/added.
How would we limit haproxy to only allow whitelisted IPs to use it??
Thanks in advance to all for helping my noobness :)
Something else on your VPS is already using port 80. Btw. you don’t need to replace *
Cheers,
Jan
@Jan ok – I replaced the config file. I changed the “*” on the binds to my VPS assigned IP (199.X…). Started haproxy and got the following:
[….] Starting haproxy: haproxy[ALERT] 065/132429 (8256) : Starting frontend f_catchall: cannot bind socket [199.XX.XX.XX:80]
failed!
@Dave, have a look at the poor man’s configuration, I think it’s better suited for your case. Unless you have 15-20 spare IP addresses of course.
Also I changed your config file (thanks for providing it btw) as follows: (XX has my valid VPS IP numbers)
frontend f_sni_catchall
mode http
bind 199.XX.XX.XX:80
If I may add a VPN vs DNS comparison based on my own findings : http://thevpn.guru/vpn-vs-smart-dns-proxy-unblock-location/
Jan! Thanks! That gave me an idea and I checked the /etc/init.d/haproxy file that I changed and fixed the usr path. Now when I start haproxy it actually gives:
[….] Starting haproxy: haproxy[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:415] : ‘bind’ : invalid address: ‘ip_address_1_here’ in ‘ip_address_1_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:612] : ‘bind’ : invalid address: ‘ip_address_2_here’ in ‘ip_address_2_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:632] : ‘bind’ : invalid address: ‘ip_address_2_here’ in ‘ip_address_2_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:654] : ‘bind’ : invalid address: ‘ip_address_3_here’ in ‘ip_address_3_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:671] : ‘bind’ : invalid address: ‘ip_address_3_here’ in ‘ip_address_3_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:689] : ‘bind’ : invalid address: ‘ip_address_4_here’ in ‘ip_address_4_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:704] : ‘bind’ : invalid address: ‘ip_address_4_here’ in ‘ip_address_4_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:722] : ‘bind’ : invalid address: ‘ip_address_5_here’ in ‘ip_address_5_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:737] : ‘bind’ : invalid address: ‘ip_address_5_here’ in ‘ip_address_5_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:755] : ‘bind’ : invalid address: ‘ip_address_6_here’ in ‘ip_address_6_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:770] : ‘bind’ : invalid address: ‘ip_address_6_here’ in ‘ip_address_6_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:788] : ‘bind’ : invalid address: ‘ip_address_7_here’ in ‘ip_address_7_here:80’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:803] : ‘bind’ : invalid address: ‘ip_address_7_here’ in ‘ip_address_7_here:443’
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:821] : ‘bind’ : invalid address: ‘ip_address_8_here’ in ‘ip_address_8_here:80’
————————
[ALERT] 065/125452 (8139) : parsing [/etc/haproxy/haproxy.cfg:1194] : ‘bind’ : invalid address: ‘ip_address_19_here’ in ‘ip_address_19_here:443’
[ALERT] 065/125452 (8139) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 065/125506 (8139) : Proxy ‘f_sni_catchall’: unable to find required default_backend: ‘b_sni_deadend’.
[ALERT] 065/125506 (8139) : Fatal errors found in configuration.
failed!
Do I put in the ip address of my (ISP assigned) modem? in the “ip_address_#_here” and do I need so many entries or just the one?
Did you chmod +x /etc/init.d/haproxy
And did you specify the path to haproxy in /etc/init.d/haproxy
/usr/sbin/haproxy
Here’s a beginner (that understands mind you what is going on but hasn’t touched linux/unix in a long while) that wants to learn, hoping someone will release a “dummies” type guide to this. :) I have a buym VPS (New York) running Debian 7.0 with haproxy (1.5) installed (but not running :`(). Doing a “Service haproxy status” produces nothing (yes i did a sudo /etc/init.d/haproxy start). I have changed the /etc/default/haproxy config to Enabled=1 and also the same in haproxy.cfg
I’d appreciate ANY help provided, on how to get this up and running, so I can point a copy of FireFox via proxy config to Netflix (US). Trying to get my kids Phineas and Ferb for march break :)
What’s the output of
which haproxy
Cheers,
Jan
Seems the developers of HAProxy don’t think it should be used as a “forwarding proxy” as this article explains. It works, but they prefer squid (lol). See http://marc.info/?l=haproxy&m=139409231103646&w=2
> Like I said, HAproxy is not a real forwarding proxy. Some people still use
> it a such, but its very limited.
I agree. We often see this confusion and each time we try to look closer, it’s obvious that other products like squid for example are much better suited for the task and are as easy to deploy. That’s why I’m not really tempted to make haproxy bigger and more complex just to handle such situations at the moment.
Not without some script magic.
Cheers,
Jan
This seems to work.
iptables -A INPUT -s localip/32 -p tcp -m tcp –dport 80 -j ACCEPT
iptables -A INPUT -s localip/32 -p tcp -m tcp –dport 443 -j ACCEPT
I suppose you cannot enter your dyndns-hostname instead of the localip there?
It’s odd none of them are doing that…
Is there a way not to have port 80 and 443 open on the server? Eg by using iptables and entering the local IP address there? Or by somehow re-routing the ports on the local router (from 80 to 8080 and from 443 to 4443 for example). Is this doable?
Sure, they could easily start some sort of cat n’ mouse game. Kicking the big DNS unblocking providers out would be a matter of minutes since they tunnel a large amount of connections/users over just a handful of IP addresses. That’s an easily recognisable pattern.
Cheers,
Jan
I am amazed that Netflix and the others somehow allow this. I mean, I can watch Netflix in Super HD at 5800 kbps in an unsupported country, and only about 1 MB traffic goes over my VPS. All the rest comes apparently directly from CDNs, without going over HAProxy. They must have a way to block this, if they really wanted, no?
There is a theoretical way to do it. You can replace the servername by * or 0.0.0.0 according to the documentation (so eg: server http://www.speedtest.net * but not followed by check). However this results in a crash of oom_killer due to HAProxy, and this is due, according to some webpages, to version 1.5 beta of HAProxy which is broken.
Address “0.0.0.0” or “*” has a special meaning.
It indicates that the connection will be forwarded to the same IP
address as the one from the client connection. This is useful in
transparent proxy architectures where the client’s connection is
intercepted and haproxy must forward to the original destination
address.
I could try using version 1.4 of HAProxy, but don’t feel like going back to an older version.
Well, you could always ask the developers if they intend to make this feature available in 1.5. again. Or some sort of a placeholder to insert the hostname as the destination server for both, HTTPS-SNI and normal HTTP connections.
Cheers,
Jan
Sorry, no idea, I’ve never tried/needed this.
Cheers,
Jan
Jan, thanks I am using this. eg:
#— speedtest
use-server http://www.speedtest.net if { hdr_end(host) -i http://www.speedtest.net }
server http://www.speedtuest.net http://www.speedtest.net:80 check inter 10s fastinter 2s downinter 2s fall 1800
But what to write after server? I want HAProxy to visit the requested domain, so NOT going to http://www.speedtest.net:80 all the time. I tried adding { host } after server http://www.speedtuest.net but HAProxy won’t take it.
It’s many months later, but is the problem the typo (u)?
Doesn’t resolve for me today, in November :)
http://www.speedtuest.net
hdr([[,]]) : exact string match[,]]) : prefix match[,]]) : subdir match[,]]) : domain match[,]]) : suffix match[,]]) : length match[,]]) : regex match[,]]) : substring match
hdr_beg([
hdr_dir([
hdr_dom([
hdr_end([
hdr_len([
hdr_reg([
hdr_sub([
Cheers,
Jan
I am RTFM. I can’t find it. Even Google won’t find it. Any idea?
But you can of course do this as well in HAProxy. But hey, RTFM ;-)
Cheers,
Jan
As I said in my other comment, this is not a good idea.
I am having some major issue with HAProxy. I want a complete domain, including all subdomains, to go through the VPS. There seems to be no way to just do that: you have to specify each and every subdomain separately. Is this really the case? I tried working with hdr_end instead of hdr, but still it won’t work. In BIND you can just use zone “domain.com.” and it works. Am I missing something?
It’s best to start without iptables rules and to add them later once everything works. You will need at least TCP 80, TCP 443 and TCP 6969 in the input chain.
I typically prefer Prometeus, BuyVM, SecureDragon or RamNode but that’s just me. If you need them, you can get additional IP addresses from almost any VPS provider on request and with justification – which can land you in hot water if you’re telling them you want to run a proxy server :-) Be prepared to shell out $1-2/month for each additional address.
Cheers,
Jan
The new poor man’s one has the error corrected and loads fine indeed.
Now, where do you get a VPS with so many IP addresses? Or is it just theoretical? What VPS do you use?
OK I found the error. Line 1183, there is b_deadend. It should be b_sni_deadend.
Also, what ports should be open in iptables for this, except 6969 (web interface)?
All HTTP requests without a hostname in the HTTP header and all HTTPS requests without a SNI-handshake will be sent to the deadened because we don’t know what to do with it.
I’m going to upload my poor man’s single IP address configuration to Github in a minute. It’s the config I’m using on my PoC server (see my other post on the subject of DNS unblocking).
Cheers,
Jan
Hmm, ok but it works for you? What does b_sni_deadend mean and why is there no code below it?
Fred, as it says in the configuration file, this configuration won’t run without modification. There could be bugs as well. It’s just a starting point with the most comprehensive set of domain names that has ever been published. You can’t use it without tinkering. You will have to invest some time to understand what’s going on. Feel free to improve it and fork it on Github or send me a pull request.
Cheers,
Jan
So I have tried this. I installed HAProxy and removed everything for non-SNI capable devices as my VPS only has 1 IP address. I’m getting this error: [ALERT] 062/091938 (2250) : Proxy ‘f_sni_catchall’: unable to find required default_backend: ‘b_sni_deadend’.
But “default_backend b_sni_deadend” is in the file, just copied from yours. What is wrong?
Of course. At least if you’re only using SNI-capable devices.
Cheers,
Jan
Right. But in the solution of belodedenko, if you combine this with a local dnsmasq, this is the same, isn’t it? Not that I doubt that your solution is better, it surely is, I love the HAProxy web interface! ;-)
No. I’m using a local DNS forwarder which either sends traffic for certain domain names directly through my remote proxy server or resolves everything else using my ISP’s DNS (roundtrip less than 1ms).
Netflix’s CDN for video streams is not affected with the traffic I’m tunneling/tunnelling through my remote proxy.
Cheers,
Jan
Yes, but in your solution, when connecting to Netflix, this also messes with Geocast, no?
(copied from the other post)
Using a remote DNS server also messes with services which rely on Geocast. As a much cheaper alternative to Anycast, the destination IP address of a geocasted service is determined by the location of the DNS server. This can wreak havoc on your download rates if your DNS server is far away.
Cheers,
Jan
Why is it bad for websites relying on Geocast? The principles are similar, aren’t they?
And it’s bad for websites which rely on Geocast. See my explanation in the comments of my other unblocking post. Not trying to bash the belodedenko guy though, it’s a good tutorial if you don’t care about download rates too much.
Cheers,
Jan
Yeah, must use dnsmasq in any case. That tutorial adds 2 seconds for every DNS resolution, so dnsmasq is a must.
So what is the advantage of using HAProxy instead of SNIProxy such as described here? http://blog.belodedenko.me/2014/02/diy-clone-of-netflix-tunlr-vps.html
Hey Fred, feel free to use whatever suits your needs. But to answer your question: HAProxy is from the pros for the pros, it’s very fast, non-blocking, supports splicing and has a ton of options like L4-tunnelling which is important for non-SNI-HTTPS connections, has a statistics interface, a web interface, a support mailing list, you can use it for load-balancing web servers as well… By the way I don’t think that’s a good tutorial because it proposes to use a remote DNS server which increases latency for _every_ DNS query.
Cheers,
Jan
P.S. Darn, is it tunnelling or tunneling? Any native English speakers here?
Definitely waiting for the update on setting this up with DNSMASQ. Tried for a few mins to play around but couldn’t get it to work properly.
Thanks for the great post! I found it via lowendtalk and I love it, very feature complete. I cant wait for the Dnsmasq (and hopefully BIND) post explaining how to get this up and running as well!