Friday, 7 February 2014

DIY clone of Netflix Tunlr/Unblock-Us/UnoTelly on cheap US based VPS



TL;DR head to netflix-proxy on GitHub or subscribe to Unzoner VPN service to un-block:





(for posterity) If you are in the UK and would like to access Netflix US catalogue, which is about three times bigger, read on.

There are commercial services out there, such as Unblock-Us and Smart DNS Proxy to name a few, so you could pay for the privilege to use their services or you can knock together a similar solution quickly yourself for a cost of a cheap Virtual Private Server (VPS). How cheap, depends on where you decide to host your VPS.
In essence, the following solution works by selectively proxying only certain requests (*.netflix.com) via a server located in the US, thus bypassing Netflix application level geo-restrictions. This solution does not proxy the actual video delivery, which uses different domain(s).

Please note, this basic solution doesn't work on devices which don't support the use of SNI during SSL handshake (e.g. Netflix on Chromecast). To support such devices, you will need to extend this solution to selectively proxy individual Netflix hosts using something akin to HAProxy, go with a commercial provider or switch to black.box unzoner.

Also, if you are looking how to watch BBC iPlayer abroad, please see the flowing guide: http://blog.belodedenko.me/2014/12/watch-bbc-iplayer-in-australia-for-free.html

Update 04/2015: You can also watch HBO NOW on your Apple TV outside the US using this solution, here's how: http://blog.belodedenko.me/2015/04/watch-hbo-now-in-uk-for-free-apple.html

Update 05/2015: You can also watch Hulu Plus on your Apple TV outside the US using this solution, here's how: http://anton.belodedenko.me/watch-hulu-plus-outside-of-the-us/

However, if you still want to build the system manually, perhaps to understand how it all works, please read on...

Prerequisites


You'll need the following:
  • A working Internet connection at home
  • A Netflix subscription
  • A US based VPS with root access and clean CentOS x86 6.x image
  • Basic knowledge in navigating a Linux console
For my hosting I choose Digital Ocean with the VPS hosted in a New York data centre for $5 USD per month. If you sign up with the above link, you will receive $10 USD credit, which is enough for two months hosting. Note, Netflix has blocked Digital Ocean IP address ranges, so don't expect it to work. Other services may still work though.

Configuration


The configuration depends on the following two components:
  • DNS server
  • HTTP/SSL proxy
For the purpose of this demo, we assume your VPS's public Internet IP is 162.243.x.y and your host is called myhost. We also assume your ISP has assigned 86.144.x.y to your home router.

You can check what your IP is by going to http://www.ifconfig.io/. Note, if you ISP assigns you a dynamic IP, you will need to update your configuration every time this changes. Better ask them for a static IP address. When creating your configuration files, please change the place holder IPs to your actual IPs.

Base VPS Deployment


This step depends on the VPS provider you choose, but broadly consists of the following steps:
  • Create account with VPS provider
  • Deploy a CentOS x64 6.x image to a US based data centre, ideally on the east coast.
Once your VPS server is created, you will get a public IP address and SSH credentials to use to login.

First, establish an SSH session to your server and install additional repositories:
[root@myhost ~]# yum -y install epel-release
[root@myhost ~]# yum -y update

Install certain pre-requisites to make your life a little easier:
[root@myhost ~]# yum install openssl vim git wget curl bind-utils telnet -y

Install development tools to enable you to build packages later:
[root@myhost ~]# yum groupinstall "Development Tools" -y
[root@myhost ~]# yum install rpmbuild autoconf automake curl libev libev-devel pcre pcre-devel perl udns-devel

IP Tables (firewall)


Since our VPS is on a public Internet, we are going to firewall it off from all the unpleasantness out there. Add rules to allow DNS, HTTP and HTTPS inbound from your home's public IP:
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p udp -m udp --dport 53 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 80 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 443 -j ACCEPT
[root@myhost ~]# service iptables save
[root@myhost ~]# service iptables restart

BIND (DNS server)


BIND is going to be our DNS server. It will respond with our VPS IP to DNS requests for netflix.com domain. Note, we are configuring the DSN server to run in a chroot jail and as a non-root user, to provide added security and minimise the potential effects of a security compromise.

Install BIND:
[root@myhost ~]# yum install bind -y

Edit your /etc/named.conf and make it look as follows:
options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { any; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

        allow-query { trusted; };
        allow-recursion { trusted; };
        allow-transfer  { none; };
        recursion yes;
        additional-from-auth yes;
        additional-from-cache yes;

        auth-nxdomain no;    # conform to RFC1035
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };
};

acl "trusted" {
        any;
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/etc/named/zones.override";

Edit your /etc/named/db.override and make it look as follows:
$TTL  86400

@   IN  SOA ns1 root (
            YYYYMMDD01  ; serial
            604800      ; refresh 1w
            86400       ; retry 1d
            2419200     ; expiry 4w
            86400       ; minimum TTL 1d
            )

    IN  NS  ns1
ns1 IN  A   162.243.x.y
@   IN  A   162.243.x.y
*   IN  A   162.243.x.y

* Note, change the above IP address to your the public IP of your VPS and update the datestamp. 

Edit your /etc/named/zones.override and make it look as follows:
zone "netflix.com." {
    type master;
    file "/etc/named/db.override";
};

Note, you may need to change the ownership of the new files you created from root to named:
[root@myhost ~]# chown named:named /etc/named/zones.override
[root@myhost ~]# chown named:named /etc/named/db.override

Install BIND (chroot package):
[root@myhost ~]# yum install bind-chroot -y

Finally, start BIND:
[root@myhost ~]# rndc-confgen -a -r /dev/urandom -t /var/named/chroot
[root@myhost ~]# chkconfig named on
[root@myhost ~]# service named start

From one of your home machines, run the following quick test:
$ nslookup www.netflix.com 162.243.x.y
Server:         162.243.x.y
Address:        162.243.x.y#53

Name:   www.netflix.com
Address: 162.243.x.y

You should get the IP address of your VPS in the DNS response.

SNI Proxy (HTTP/SSL proxy)


SNI Proxy is going to be our HTTP/SSL proxy, which supports for TLS SNI extensions.

Build and install SNI Proxy:
[root@myhost ~]# cd /opt
[root@myhost ~]# git clone https://github.com/dlundquist/sniproxy.git
[root@myhost ~]# cd ./sniproxy
[root@myhost ~]# ./autogen.sh && ./configure && make dist
[root@myhost ~]# rpmbuild --define "_sourcedir `pwd`" -ba --nodeps redhat/sniproxy.spec
[root@myhost ~]# yum install $(ls /root/rpmbuild/RPMS/x86_64/sniproxy-[0-9]*.rpm) -y

Note: if the RPM fails to install, check the exact name of the file in the /root/rpmbuild/RPMS/x86_64/directory.

Also, please review the ./configure command output to make sure LIBUDNS or libudns is detected, if it hasn't, you will not be able to get SNI Proxy to work with the below configuration file. On a Red Hat/CentOS server, the library should be available from EPEL repository (yum install udns-devel), however for other distributions, you will need to build and install it manually.

Edit your sniproxy.conf and make it look as follows:
[root@myhost ~]# vim /etc/sniproxy.conf

user daemon
pidfile /var/tmp/sniproxy.pid

error_log {
        filename /var/log/sniproxy_error.log
        priority notice
}

listener 162.243.x.y 80 {
        proto http
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

listener 162.243.x.y 443 {
        proto tls
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

table {
        netflix\.com *
}

Finally, install the start-up script and start SNI Proxy:
[root@myhost ~]# cp ./redhat/sniproxy.init /etc/init.d/sniproxy
[root@myhost ~]# chmod +x /etc/init.d/sniproxy
[root@myhost ~]# chkconfig sniproxy on
[root@myhost ~]# service sniproxy start

Note, if the proxy complains about ports being in use, check if you got another web server already running and if so, disable it or move it to a different port.

From one of your home machines, run the following quick test:
$ telnet 162.243.x.y 80
Trying 162.243.x.y...
Connected to 162.243.x.y.
Escape character is '^]'.
^C
Connection closed by foreign host.

$ openssl s_client -servername www.netflix.com -connect 162.243.x.y:443
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=CALIFORNIA/L=Los Gatos/O=Netflix, Inc./OU=Operations/CN=www.netflix.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MII...

You should get a socket connection on port 80 and get an SSL handshake with Netflix.

Putting It All Together


The last step is to change the DNS server on the device you are planning to watch US Netflix on. Set it to your VPS public IP (i.e. 162.243.x.y) Fire up Netflix and you should see the US catalogue and if you do - enjoy!

Note, this won't work on devices which don't support the use of SNI during SSL handshake, you'll have no choice but to use a system described in this guide or use black.box unzoner.

References


I've used the following reference material to prepare the solution described in this article. Many thanks to the respective authors.


169 comments:

MrMadafakas said...

Hey, thanks for the great method - it works very well. However, how about doing it without the need of SSL3 handshake? Many things do not work with this (like LG TV's or Xbox or Playstations...).

--- ab1 said...

Thanks. To be honest, I haven't got access to a device which doesn't support SNI during the SSL handshake. But in theory, if you somehow inject the host header during the SSL handshake, it could work. Not sure how to do this in practice though.

-- ab1

Ari V said...

Hi I was wondering how much bandwidth wood i need to watch netflix 2 units average 10hrs a day

trying to figure out what package to get from digital oceans

--- ab1 said...

Very little, because you are only directing Netflix web and API traffic, not the actual streaming video. I am using their $5 per month Linux server

Ari V said...

cheHi,
That was a great tutorial.
I followed all your steps and finally tried to start the BIND.
But it FAILED to start and was showing an error that the Netflix.com file was not found!

This is what being shown after I try to start it(I have even tried to remove the dot after the Netflix.com):


[root@flixfix etc]# service named start
Starting named:
Error in named configuration:
zone localhost.localdomain/IN: loaded serial 0
zone localhost/IN: loaded serial 0
zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
zone 0.in-addr.arpa/IN: loaded serial 0
zone netflix.com/IN: loading from master file /var/named/chroot/etc/named/db.override failed: file not found
zone netflix.com/IN: not loaded due to errors.
_default/netflix.com/IN: file not found
[FAILED]

--- ab1 said...

Try changing the ownership of the file and all other files you create in this tutorial to the user:group that BIND runs under (e.g. chown named:named ./db.override)

Ari V said...

I'm still little confused

eg: chown a:b myfilename what is A and what is B ( what is users and groups mean?

--- ab1 said...

In this case it will be named:named, but you can do "ls -la" in the chroot directory to verify.

Ari V said...

thanks for the help so far still no joy I must be missing something in between?

I even started fresh wiped my droplet with ocean digital and started again I did get some response

Log below

root@myhost etc]# service named start
Starting named:
Error in named configuration:
/var/named/chroot/etc/zones.override:1: unknown option 'zones'
/var/named/chroot/etc/zones.override:7: unexpected end of input
[FAILED]

I now back to same error as my previous post. I tried everything has to be something I missing :-(

--- ab1 said...

Just make sure db.override and zones.override are in the correct paths according to your config file and both are owned by named user:group.

Ari V said...

Ok did all that now I get the error without any msg

Finally, start BIND:

[root@myhost ~]# rndc-confgen -a -r /dev/urandom -t /var/named/chroot
[root@myhost ~]# chkconfig named on
[root@myhost ~]# service named start

Note, you may need to change the ownership of the new files you created from root to named (e.g. chown named:named ./db.override).


I now get

[root@myhost etc]# service named start
Starting named: [FAILED]

--- ab1 said...

Check various logs on the system to figure out why it isn't starting. It should be something fairly obvious. Check /var/log/messages and logs under /var/named.

Ari V said...
This comment has been removed by a blog administrator.
--- ab1 said...

You need to substitute your actual IPs everywhere, where I redacted mine with x.y

Felipe Stanham said...

I just can't get SNI Proxy service started.

I've tried with "sudo sniproxy service start"
but it doesn't start

If I type "sniproxy" it ouputs
*** buffer overflow detected ***: sniproxy terminated

I'm using a Ubuntu server from Chicago VPS

Any advice?

Thanks

--- ab1 said...

Try the Ubuntu build instructions:
https://github.com/dlundquist/sniproxy/blob/master/README.md

Ari V said...

Seriously there is something you have left out in the tutorial.

I mean we have the same host so the configurations would be pretty smilar

this is the error log now

adjusted limit on open files from 4096 to 1048576
Feb 20 06:07:28 hostit named[2179]: found 1 CPU, using 1 worker thread
Feb 20 06:07:28 hostit named[2179]: using up to 4096 sockets
Feb 20 06:07:28 hostit named[2179]: loading configuration from '/etc/named.conf'
Feb 20 06:07:28 hostit named[2179]: /etc/named.conf:51: open: /etc/named.rfc1912.zones: permission denied
Feb 20 06:07:28 hostit named[2179]: loading configuration: permission denied
Feb 20 06:07:28 hostit named[2179]: exiting (due to fatal error)


--- ab1 said...

I'll try building another host in a few weeks and update the guide.

Ric McCunning said...

Really cool! Managed to get most of this setup using just copy and paste. I did notice one thing where the you say to edit db.override, but didn't mention to put it into /var.named/chroot/etc/named/. Threw me a bit, but spotted it eventually. Also I had to copy the named.conf into the /etc folder as it wasn't being picked up where it was and I also needed to rename the sniproxy.init file in the /etc/init.d folder to take off the .init extention and 'chmod 777 sniproxy' the file so that it didn't throw an access denied error.

The bit that I'm stuck with now is that it does seem that there's something else on port 443. You mention to check for other web servers, but I've only been using unix for a grand total of 2 hours so not sure where to begin to look. (I'm building this in an azure vm btw if that's any help)

Thanks for putting this together btw!

--- ab1 said...

I'll update this guide in the next few weeks to include all the suggestions and then test it end to end.

The bit about other services listening on 433/80 is easy. Just run a command 'netstat -anp | grep 443' and see what process is already using it. It will probably be Apache. Stop it and try again.

Ric McCunning said...

Turned out not to be that. SNIProxy was just failing to bind to the IP address. I changed the IP address in the sniproxy.conf to be the internal IP address of the server and now it works fine.

Ta!

Felipe Stanham said...

@-- ab1

I installed sniproxy again and I've got no luck.

This is the error output I get when trying to start sniproxy not as a service


root@netflix:/etc/init.d# sniproxy
*** buffer overflow detected ***: sniproxy terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f0db4a174f7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf9410)[0x7f0db4a16410]
/lib/x86_64-linux-gnu/libc.so.6(+0xf86f6)[0x7f0db4a156f6]
sniproxy[0x402924]
sniproxy[0x402a88]
sniproxy[0x4033bf]
sniproxy[0x40346a]
sniproxy[0x403c9b]
sniproxy[0x401dfd]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f0db493e30d]
sniproxy[0x401f9d]
======= Memory map: ========
00400000-00409000 r-xp 00000000 08:03 6181098 /usr/sbin/sniproxy
00609000-0060a000 r--p 00009000 08:03 6181098 /usr/sbin/sniproxy
0060a000-0060b000 rw-p 0000a000 08:03 6181098 /usr/sbin/sniproxy
02392000-023b3000 rw-p 00000000 00:00 0 [heap]
7f0db4483000-7f0db4498000 r-xp 00000000 08:03 6162979 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0db4498000-7f0db4697000 ---p 00015000 08:03 6162979 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0db4697000-7f0db4698000 r--p 00014000 08:03 6162979 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0db4698000-7f0db4699000 rw-p 00015000 08:03 6162979 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0db4699000-7f0db471c000 r-xp 00000000 08:03 6163023 /lib/x86_64-linux-gnu/libm-2.13.so
7f0db471c000-7f0db491b000 ---p 00083000 08:03 6163023 /lib/x86_64-linux-gnu/libm-2.13.so
7f0db491b000-7f0db491c000 r--p 00082000 08:03 6163023 /lib/x86_64-linux-gnu/libm-2.13.so
7f0db491c000-7f0db491d000 rw-p 00083000 08:03 6163023 /lib/x86_64-linux-gnu/libm-2.13.so
7f0db491d000-7f0db4ab4000 r-xp 00000000 08:03 6162990 /lib/x86_64-linux-gnu/libc-2.13.so
7f0db4ab4000-7f0db4cb3000 ---p 00197000 08:03 6162990 /lib/x86_64-linux-gnu/libc-2.13.so
7f0db4cb3000-7f0db4cb7000 r--p 00196000 08:03 6162990 /lib/x86_64-linux-gnu/libc-2.13.so
7f0db4cb7000-7f0db4cb8000 rw-p 0019a000 08:03 6162990 /lib/x86_64-linux-gnu/libc-2.13.so
7f0db4cb8000-7f0db4cbe000 rw-p 00000000 00:00 0
7f0db4cbe000-7f0db4ccb000 r-xp 00000000 08:03 6584965 /usr/lib/libev.so.4.0.0
7f0db4ccb000-7f0db4eca000 ---p 0000d000 08:03 6584965 /usr/lib/libev.so.4.0.0
7f0db4eca000-7f0db4ecb000 r--p 0000c000 08:03 6584965 /usr/lib/libev.so.4.0.0
7f0db4ecb000-7f0db4ecc000 rw-p 0000d000 08:03 6584965 /usr/lib/libev.so.4.0.0
7f0db4ecc000-7f0db4f07000 r-xp 00000000 08:03 6163020 /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f0db4f07000-7f0db5106000 ---p 0003b000 08:03 6163020 /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f0db5106000-7f0db5107000 r--p 0003a000 08:03 6163020 /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f0db5107000-7f0db5108000 rw-p 0003b000 08:03 6163020 /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f0db5108000-7f0db5129000 r-xp 00000000 08:03 6162989 /lib/x86_64-linux-gnu/ld-2.13.so
7f0db531d000-7f0db5321000 rw-p 00000000 00:00 0
7f0db5325000-7f0db5328000 rw-p 00000000 00:00 0
7f0db5328000-7f0db5329000 r--p 00020000 08:03 6162989 /lib/x86_64-linux-gnu/ld-2.13.so
7f0db5329000-7f0db532b000 rw-p 00021000 08:03 6162989 /lib/x86_64-linux-gnu/ld-2.13.so
7fffe269e000-7fffe26b3000 rw-p 00000000 00:00 0 [stack]
7fffe26fb000-7fffe26fd000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted


any ideas?

--- ab1 said...

Odd.. Are you trying this on your Ubuntu VPS or Red Hat?

Anthony Paul said...

I have this partially working - I can browse the US catalog (The Office (U.S.), but I get an error when I try and play US content. I can play other content fine. Any ideas? Thanks.

--- ab1 said...

What happens when you do a DNS resolution as well as OpenSSL test? You should resolve to the IP if your host and get the SSL cert. for Netflix.

Jeff Detmold said...
This comment has been removed by the author.
Frederick said...

Would doing this on Debian also work similarly? anything to look out for?

Frederick said...

I can confirm this works. I tried this on exactly the same configuration. However what gives errors is bind-chroot. It works just find if you use yum install bind and don't use the chrooted bind. (see 3rd post on https://www.centos.org/forums/viewtopic.php?t=5314 ).
Then make sure not to use the chrooted dirs for bind. Also, you need to manually change sniproxy.init to sniproxy and chmod it. Lastly, when I telnet from my home PC to the server, I get "503 Service Temporarily Unavailable" and "Backend not available", but it still works with Netflix. Throughput is amazing, I get SuperHD within seconds.

Frederick said...

BTW: if you want to know how fast it REALLY is, add speedtest.net to the zones.override and sniproxy.conf files, and give it a whirl.
ab1: what is the difference between sniproxy and squid (which I am more used to)? Is it SSL support?

Frederick said...

ab1: does this also mean that now ALL DNS requests pass through the USA VPS? This probaly slows surfing down a bit I guess. Ideally, one would have a VPS close by (in same country), despatching only the necessary DNS requests to the US VPS. But it's probably very complicated to set this up.

Frederick said...

One more comment: after testing, it is clear that DNS resolving of other domains takes much longer (.5-2 seconds instead of instantaneous), probably because all DNS requests have to go through USA. So it's not really usable for daily use, but only on machines or devices which need an almost constant US connection (eg Roku box)...
That's an advantage of eg Unblock Us and similar services: they have servers all over the world, and redirect only to the necessary server if a matched domain is found.

Frederick said...

I ran GRC's DNS benchmark tool. And indeed: uncached resolving takes between 0.5 and 2.5 seconds, while most other servers (such as Google's) only need 0.02-0.04 s. Ouch.

Tom Balon said...

Great post ... I ended up setting this up on an Ubuntu System. Seems to work ok on my laptop but not on my older roku set top box. Perhaps it uses server name indication (sni) ...

Anyway, I was concerned about data usage on the VPS. I fired up a video and looked at eth0 at start and end. After about a minute, I stopped the video and looked at eth0 again. The VPS and transmitted 1,640,981 bytes during that time. The VPS TX averages 725KB / day .. so I'm a bit concerned about what is being proxied through the system. 1.6MB in 1 minute...

Perhaps it was just the site (netflix.com) loading and the startup ... I'm sure hope the content (CDN) isn't sending anything thru the proxy.

Have you noticed any large data usage on your VPS since you started ?

Again, great job with this...

Abisai Cruz Valdez said...

I have a dedicated server, but is behind a firewall may open certain ports to work?

--- ab1 said...

Thanks everyone for having to troubleshoot my mess :) glad that it works at least for some of you!

Regarding excess traffic on the VPS - the setup only proxies *.netflix.com but not the actual video delivery, which is done using a different domain. The traffic you will be seeing is web/API requests as well as telemetry data from the players.

Regarding sending all DNS requests via the VPS - I agree this is undesirable. If you have a router capable of DNSmask, you can configure it to only send netflix.com DNS requests via your VPS while resolving everything else locally.

I use a fortigate firewall for this with a local DNS server.

-- ab1

--- ab1 said...

Also, I don't think roku devices will work using this approach, because they don't perform SNI during SSL handshake.

Frederick said...

Rokus don't allow for any configuration: you cannot change the DNS server at all. My Roku works, because I have a router behind it wiht my VPS IP as DNS server configured. Only way to do it.

Ari V said...

I'm glad people are posting and not me being the only one with issue's.

I cannot wait for the updated tutorial as i have wasted 3 weeks of server space :-)

Roberto said...

Thanks for this great tutorial. Works well in my pc but not works in my smart tv Viera Panasonic.

--- ab1 said...

I've updated the tutorial to make it a bit clearer:

- added full path names under DNS section
- added file ownership changes for new files
- fixed sniproxy start-up script instructions

I haven't tested this using copy/paste end-to-end as yet, but if anyone wants to have a go in the meantime and let me know if they encounter any more issues, hat would be great..

-- ab1

Ari V said...

an I missing some parameters in the named.rfc1912.conf?

adjusted limit on open files from 4096 to 1048576
Mar 15 22:27:11 myhost named[11367]: found 1 CPU, using 1 worker thread
Mar 15 22:27:11 myhost named[11367]: using up to 4096 sockets
Mar 15 22:27:11 myhost named[11367]: loading configuration from '/etc/named.conf'
Mar 15 22:27:11 myhost named[11367]: /etc/named.conf:52: open: /etc/named.rfc1912.zones: permission denied
Mar 15 22:27:11 myhost named[11367]: loading configuration: permission denied
Mar 15 22:27:11 myhost named[11367]: exiting (due to fatal error)

--- ab1 said...

Perhaps try installing BIND without chroot:
https://www.digitalocean.com/community/articles/how-to-install-the-bind-dns-server-on-centos-6

daniel cook said...

stupid question, do I have to have HTTPD killed in order for SNIPROXY to run ? Can i attach the SNIPROXY to my internal IP address instead?

--- ab1 said...

Yes, HTTPD will need to be shutdown in order for SNI Proxy to listen on port 80, since we need to transparently proxy HTTP.

You could probably do some iptables redirects internally on your host to make both of them work, but that is outside the scope of this article.

Isti said...

Hello, does it work with Apple TV?

--- ab1 said...

Yes, works on my ATV.

Beenje said...

Hi. Nice article.
One comment about iptables. You allow specific traffic, but you haven't blocked anything...
You need to change the default policy to DROP (iptables -P INPUT DROP) or add a rule to block all traffic (iptables -A INPUT -j DROP).
You should not block traffic on the loopback interface. See Iptables How To

It could be:
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -s 86.144.x.y/32 -p udp -m udp --dport 53 -j ACCEPT
iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A INPUT -j DROP

You might want to allow ssh as well depending on how you connect to the machine.

--- ab1 said...

Good point, however the CentOS distro I used came by default with the following at the end of the iptables policy:

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

There is a discussion about DROP vs. REJECT here:
http://www.chiark.greenend.org.uk/~peterb/network/drop-vs-reject

--ab1

Beenje said...

It would probably be good to mention that in the article. I'm sure some people don't know (even if I don't think you should follow that procedure if you don't understand what you are doing).

Thanks for the link about DROP vs REJECT.

envoys said...

Nice article. Working flawless on android, iPad, wii and ps3! Can we get your complete iptables rules? Every time I add the above, soon as I add the drop clause, netflix stops working but ssh etc continues fine...

--- ab1 said...

Good stuff, here is what is in the /etc/sysconfig/iptables on my VPS:

[root@ ~]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Wed Feb 5 17:24:23 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [20:2632]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s myhomeip.belodedenko.me -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -s myhomeip.belodedenko.me -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -s myhomeip.belodedenko.me -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Wed Feb 5 17:24:23 2014

This rule set allows SSH/ICMP in from anywhere as well as DNS/HTTP/HTTPS from my home IP. The rest is rejected.

-- ab1

Paylaştıkça said...

Hi, I try to install on centos 6.4 and 6.5 x64. All fails with permission denied error on /etc/named.conf. I try to remove and re-install bind. After reinstall bind, the service could be started but when I try to install bind-chroot, It fails with same error :(

--- ab1 said...

Some others had problems with bind-chroot as well. I would just install BIND without chroot and be done with it.

Unless you are planning to store sensitive information on this VPS, I wouldn't bother with the extra security of putting BIND into chroot jail.

-- ab1

Ari V said...

If i wanted to add a second ip address so i can share the rent of the server with my sister how would i add a second ip addess?

Ari V said...
This comment has been removed by the author.
Ari V said...

ok worked most of it out got to the sniproxy now

chkconfig sniproxy on

but i get error now
error reading information on service sniproxy: No such file or directory


any idea's

--- ab1 said...

Wow, congratulations - you are almost there! :)

Did you copy the sniproxy.init into /etc/init.d? (e.g.):

[root@myhost ~]# cp ./redhat/sniproxy.init /etc/init.d
[root@myhost ~]# chmod +x /etc/init.d/sniproxy.init
[root@myhost ~]# chkconfig sniproxy on
[root@myhost ~]# service sniproxy start

Also, you don't need to add a second IP to share your VPS. All you need to do is allow DNS/HTTP/HTTPS access from your second IP to your VPS (e.g.):

[root@myhost ~]# iptables -A INPUT -s /32 -p udp -m udp --dport 53 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s /32 -p tcp -m tcp --dport 80 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s /32 -p tcp -m tcp --dport 443 -j ACCEPT
[root@myhost ~]# service iptables save
[root@myhost ~]# service iptables restart

-- ab1

Ari V said...

Yes it been battle, and I'm on 3rd base & the carrot is dangling in front of me.

Yes the file is in that folder as mentioned I hope it does not take me another month to figure it out :)

Ari V said...

any other obvious hints?

--- ab1 said...

Try starting it from the command line as follows:
[root@myhost ~]# /usr/sbin/sniproxy -c /etc/sniproxy.conf
2014-04-02 07:51:03 Parsed netflix.com *

If it loads without errors, at least you can test it all out and worry about the service config later on.

--ab1

Ari V said...

yeah got same result as you posted, where do i go from here could it be a permision issue?

--- ab1 said...

Leave your VPs logged in and run the tests first:

$ nslookup www.netflix.com
Server: 162.243.x.y
Address: 162.243.x.y#53

Name: www.netflix.com
Address: 162.243.x.y


$ telnet 80
Trying 162.243.x.y...
Connected to 162.243.x.y.
Escape character is '^]'.
^C
Connection closed by foreign host.


$ openssl s_client -servername www.netflix.com -connect :443
...


If it all works, then point DNS on your Apple TV/whatever at your VPS and see what happens.

-- ab1

Ari V said...

I have tried everything, but still the service is not starting. Giving me error saying that "error reading information on service sniproxy: No such file or directory"
So, does that error means the sniproxy service is not correctly registered?

We are trying to start the service via the chkconfig command. Isn't it? So, will it because the service's conf file or the script file is not in proper location?

I tried looking around and found this: http://svanstrom.nu/2013/10/roll-your-own-unblock-us/

There, I saw something mentioning about changing the ports Do we have to do that? Was that what we are missing?

still no luck

--- ab1 said...

OK, I think I found the problem. The sniproxy init script should be /etc/init.d/sniproxy rather than /etc/init.d/sniproxy.init.

If you followed my previous instructions, just rename the file and then install the service:
[root@myhost ~]# mv /etc/init.d/sniproxy.init /etc/init.d/sniproxy
[root@myhost ~]# chkconfig sniproxy on
[root@myhost ~]# service sniproxy restart

-- ab1

Ari V said...

Thanks heaps that got it going. working of pc browser.

not my western digital live tv i have to test with ipod or ipad

--- ab1 said...

Good to hear you got it going finally :)

This process won't work on some devices, because they don't support the necessary SSL extensions, but will work on iOS devices.

-- ab1

Ari V said...

No Problem also just to note my telnet goes blank black not trying to connect although the service is working I think i may of saw 503.

here is another link to help us with non sni gear

Netflix DNS-unblocking without SNI for your Xbox 360, PS3, WDTV, Samsung TV

http://trick77.com/2014/04/02/netflix-dns-unblocking-without-sni-xbox-360-ps3-samsung-tv/

Cheers

Grant Harris said...

Thanks for the guide! However, I'm still having some issues. The DNS and proxy change initially seemed to work. When I go to the netflix program list it properly shows the American Netflix shows, yet when Itry to play any of the american only titles I get the following error:
http://imgur.com/xTZyCe3

I'm running this on a Macbook Air with Google Chrome. Has anybody seen/solved this issue?

--- ab1 said...

Make sure you only have your device configured to point only to your special DNS server. Also disable IPv6 if you have it configured. Empty DNS resolver cache on your Mac and check if your upstream router isn't caching the wrong DNS entries.

I've seen this before when the system is doing round robin DNS resolution.

-- ab1

Ari V said...

ab1 what are the chances of doing a not sni tutorial to support wd tv and xbox360 samsung tv tutorial.

this tut you did has been a gem and thanks you very much for the effort and time and suppourt.

--- ab1 said...

I am definitely interested in doing this type of setup using HAProxy/DNAT, however I don't have a suitable device to test with at the moment.

-- ab1

Ari V said...

maybe I can test it for you and others, xbox360, wd digital live tv, roku :-)and i would be more than happy to donate towards it

Ari V said...

can you post update to include bbc i player and pandora.

Cheers

--- ab1 said...

If you want Pandora, BBC iPlayer, etc. you'll just need to add the additional domains into sniproxy.conf and zones.override (e.g.):

[sniproxy.conf]
# grep '^[^#]' /etc/sniproxy.conf

user daemon

pidfile /var/tmp/sniproxy.pid

listener 80 {
proto http
}

listener 443 {
proto tls
}

table {
netflix\.com *
pandora\.com *
bbc\.co\.uk *
}


[zones.override]
zone "netflix.com." {
type master;
file "/var/named/chroot/etc/named/db.override";
};

zone "pandora.com." {
type master;
file "/var/named/chroot/etc/named/db.override";
};

zone "bbc.co.uk." {
type master;
file "/var/named/chroot/etc/named/db.override";
};


You'll also need to use a VPS in the UK, since BBC iPlayer is geo locked to the UK.

-- ab1

Mat Allin said...
This comment has been removed by the author.
--- ab1 said...

HGTV at least (not sure about the other one), implement a content provider lock, so you must have a subscription with the supported ISP/TV provider in the US.

Even if you can get around the geo-locking using the approach in this guide, you won't be able to stream unless you have a user account with Verizon, Time Warner, etc.

-- ab1

envoys said...

I do have an account with a US based ISP that's why I was asking...i found some urls to get around it but I can only seem to stream the AD's...and after the ad it comes up and says video not found...

--- ab1 said...

If you have a valid account with one of the providers they support, then you should try bypassing geo-locking using this approach (possibly for both the streaming provider's domain and the TV provider's authentication domain) and it should hopefully just work.

If the system tells you the video is not found, then it sounds like their system is broken or just badly designed, or maybe they have geo-locking enabled on the media distribution end-points (i.e. Akamai, Level3, etc.), in which case you'll have to bypass geo-locking for these as well.

You basically have to watch what DNS requests are being made and evaluate whether or not to proxy those domains via your US VPS.

Trial and error I am afraid.. Sorry if I can't be helpful.

- ab1

Matthew K said...

I'm ALMOST there but I have an issue starting sniproxy...

[root@myhost sniproxy]# service sniproxy start
Starting sniproxy: Only socket address backends are permitted when compiled with libudns
error parsing /etc/sniproxy.conf at 211 near:
212 netflix\.com *
214 }
214 (null)214 (null)214 (null)Unable to load /etc/sniproxy.conf
Usage: sniproxy [-c ] [-f]
[FAILED]
Any ideas?

Ari V said...

Thanks AB1 how would one add a second ip for the uk would you need to create a php script?

thanks for the support at the moment all i have samsung tv, wd tv , so only way i can test these is via pc

envoys said...

Thanks for all the help!

I've discovered the HGTV/DIY Network seems to be loading from akamai but i couldn't see this in my dns logs, i had to run Fiddler to determine this, it's requesting based on IP addresses and not host names, but resolving the host names comes back to "akamaitechnologies.com" but with subdomains in the beginning...how do I tell sniproxy and named to process anything going to that domain and sub domains? Any idea?

--- ab1 said...

@envoys: you could try adding the domain(s) to sniproxy.conf and zones.override:

[sniproxy.conf]
table {
netflix\.com *
akamaitechnologies\.com *
}

[zones.override]
zone "akamaitechnologies.com." {
type master;
file "/var/named/chroot/etc/named/db.override";
};


@Matthew K, I have no idea why you are getting that error, looks like a format error in the sniproxy.conf file maybe? Check it again very carefully, against my examples and examples on the web and also pay attention to any errors it threw during the build.


@Ari V: what exactly do you want to achieve with the 2nd IP?


-- ab1

mkkyah said...

Thanks for the great tutorial. Do you plan to work on a non-SNI version soon? I know you don't have devices but there are lots of testers around willing to help.

Matthew K said...

@ab1

I figured it out. It was actually missing one of the required dependencies of sniproxy: udns

This was on sniproxy's git so I just followed the instructions:

sudo yum install rpmbuild autoconf automake curl libev-devel pcre-devel perl pkgconfig udns-devel

Now I'm getting the following error! I was so close!

[root@myhost sniproxy]# service sniproxy start
Starting sniproxy: Table "(null)" not defined
Failed to initialize listener
listener 162.xxx.xxx.xxx:443 {
protocol tls
}

[FAILED]

On SNIProxy's github someone reported the same issue and that it was due to editing or the conf file, but I've tried every which way, can't seem to make it work! Sniproxy starts just fine without the listener commands in the .conf file

Matthew K said...

Additional update:

Worked when I put the table before the listener in the .conf file.

Thanks for the guide!

envoys said...

I tried that and still not working...as i mentioned above, it appears it's going to the IP address of the AKAMAI servers which I assume change often so I think it's pretty hard to do...it would almost appear that my request goes to the hgtv site, then they do the geo lookup with akamai as I see nothing in my logs pointing to akamai. Strange. i'll keep plugging away.

Matthew K said...

@ab1, have you seen this using HAProxy?

http://trick77.com/2014/03/02/dns-unblocking-using-dnsmasq-haproxy/

Seems to be a good work around for non-SNI devices. Although I'd prefer to use it with BIND instead, as some family/friends cannot use DNSMasq at home. Feel free to write up an addition to your guide with this method using BIND! :-)

I've actually learnt quite a bit about linux from this guide.

--- ab1 said...

And so did I :)

-- ab1

P.S. Yes, I started looking at the HAProxy write-up, however it does require a Linux server at home, which will be doing TCP port translation towards the proxy. Unfortunately without a local server it can not be implemented. Fortunately however, a Raspberry Pi will do the job just fine.

Cyberphobia said...

Thanks for the great post, followed most of it and got it up and working with no issues.

I do have an issue with my Roku 2 box, all other devices work using the VPS dns sniproxy work fine, pc, iphone, wii.

Roku can login and can see the US content, but when you try and stream, Roku throws error 010 and loses its wifi settings.

I have tested US content with Roku via unblock us and unlocator and it has no issues. I have reverted to packet captures but cannot see where its failing.

Any suggestions?

--- ab1 said...

Hi, do you see a difference in the SSL handshake in your packet capture against Roku vs. a working device?

This whole setup requires the client device to support SNI, if it doesn't, you'll need to use a more complicated setup with HAProxy/DNAT.

-- ab1

Jeff J said...

@ ab1

I got everything working following your setup with a few small changes when doing sniproxy, because it needs to be build with the depends.

Hulu, Netflix and Pandora all work great. So i decided to try and add southparkstudios.com in there so i can see their episode teasers with out being told I'm not in the US. however it just seems to redirect the site to port 80 on my VPS when I add it to the Zones database. Any ideas why?

--- ab1 said...

Try adding mtvnservices.com along side southparkstudios.com

Worked for me..

-- ab1

mkkyah said...

Can someone give necessary proxy inputs for UK based services like Sky-go, 4od, BBC Iplayer to use with an UK based VPS?

Jeff J said...

@ab1, thanks that worked. Now to resolve fox.com with the same issue. I am sure i am missing something i will use the Lightbeam plugin on firefox to see what sites connect to fox.com. Thanks for the hand.

Jason Separovic said...

Thanks for posting this. Saved me a few hours of muckin around.

A couple of notes for centos 6.5 minimal:
1) I had to "chown -R named:named /var/named" (not sure if the whole dir needed to be non-root but it worked)

2) Also a few more dependencies on centos6.5 minimal for sniproxy:
- udns udns-devel libev libev-devel

And check out this one liner for yum installing from http
- yum localinstall http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Not really a "localinstall" but hey it works

Anyway really appreciate your effort, got back on X-Files last night, woot!

--- ab1 said...

Cheers, I've updated the guide.

VonMiela said...

Thank you very much for this.
So far it works fine. I would appreciate if you can post the complete iptable configuration.

--- ab1 said...

Thank you, for complete iptables rules, please see post above on the 29 March 2014 15:02.

-- ab1

JatMan said...

I am able to get it to work on Netflix and Pandora however, I can't get Hulu to work. I get error on hulu stating that I am using anonymous proxy tool and can't access hulu. Btw, I am using AWS EC2 as my virtual machine located in oregon.

Hulu error: http://cdn.ghacks.net/wp-content/uploads/2014/04/hulu-vpn-block.jpg

--- ab1 said...

This post sums it up:
http://www.ghacks.net/2014/04/25/can-hulus-anonymous-proxy-tool-block/

You could try getting a different IP assigned to your VPS, which hopefully won't be blocked, but I suspect Hulu are going through and blocking ranges of IPs they believe are being used to proxy through.

-- ab1

Jatin said...

I like to confirm that Netflix on Roku 2 doesn't work. However, it works on Roku 3.

HWY Community said...

Hi,

Thanks for the awesome post. My only problem is with my iptables.

I've whitelist my ip, but if at the end of the chain I drop all with this command.

iptables -A INPUT -j DROP

I can't access netflix or any site that I route to my VPS. I still can access other sites though.

Can you help with this? Thanks

--- ab1 said...

I posted my iptables policy above. You should just use that using your home ip instead of mine. Just edit the file directly and restart iptables service.

-- ab1

HWY Community said...

it still doesn't work for me. Is it because my ISP implement transparent proxy? all other sites is working (the one that doesn't zone override)

--- ab1 said...

It really depends..

(1) Can you resolve netflix.com to the IP of your VPS?

(2) Can you get to port 80 and 443 on your VPS IP?

(3) Can you get a proper SSL handshake with your VPS using SNI option?

To rule out the firewall, just stop the iptables service on your VPS, while you are testing.

If you can't even get a socket connection to your VPS on the above ports, then it is possible your ISP is interfering.

-- ab1

HWY Community said...

Hi,

Yes, I rule out the iptables and all works perfectly.

The only issue is with iptables at rule:

iptables -A INPUT -j DROP

In Singapore, most telco put transparent proxy, is that the root cause? It still strange though because it's totally working without the iptables

Stephan said...

I've been trying to get this to work for a while now.

At first I got the same error as many others because of chroot, so I reinstalled bind without chroot. I now copied the edited files to /etc in stead of /chroot/etc and edited the files so they'd refer to /etc.

Now, named seems to start fine, i only get this warning:
Warning: 'empty-zones-enable/disable-empty-zone' not set: disabling RFC 1918 empty zones

However, when I try the nslookup it fails:

$ nslookup www.netflix.com 54.201.x.y
;; connection timed out; no servers could be reached

I have IPTables set as instructed and checked with yours in the comments, where only the order of the rules differ, but I can't imagine this is the problem. I also set my AWS security group to allow UDP traffic to 53.

I double checked everything, but I just can't get it to werk...

--- ab1 said...

Have tried doing nslookup locally on your VPS? (i.e. nslookup netflix.com localhost)

-- ab1

Jurjen den Toom said...

Thanks for the great guide! I needed a couple of extra things installed on CentOS 6 x64:
- http://dl.fedoraproject.org/pub/epel/6/x86_64/libev-4.03-3.el6.x86_64.rpm
- http://dl.fedoraproject.org/pub/epel/6/x86_64/libev-devel-4.03-3.el6.x86_64.rpm
- pcre pcre-devel udns-devel (yum install pcre pcre-devel udns-devel)
- chmod 777 /etc/sniproxy.cfg

--- ab1 said...

Thank you, I've updated the guide with the missing dependencies. It is odd you had to adjust permissions on /etc/sniproxy.conf. Mine is owned and readable/writable by root, no execute.

-- ab1

Arya Ramandhanu said...

Hi
i had US VPS and setup like your tutorial, but when i test with openssl s_client -servername www.netflix.com -connect ip:443

i got error like this

140380820043592:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:741:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 287 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE


can you explain why ?
Thx

--- ab1 said...

That probably means SNI proxy isn't doing what it is supposed to do.

Try running the test directly from your VPS and go from there.

Check if the corrrect process is listening on your VPS_IP:443

# netstat -anp | grep 443
tcp 0 0 x.x.x.x:443 0.0.0.0:* LISTEN 123456/sniproxy

It should be pid/sniproxy. If it is, then check your sniproxy.conf and the proxy:

# service sniproxy restart
Stopping sniproxy: [ OK ]
Starting sniproxy:
2014-06-19 04:57:19 Parsed netflix.com *
[ OK ]

-- ab1

Calvinz360 said...

I've deployed your method on Ubuntu and it works great, at least for netflix. However it fails for other sites such as hulu/abc.com. It keeps prompting me the error that I'm outside of US.

--- ab1 said...

See the comment on 9 May 2014 07:50 regarding Hulu (and possibly ABC).

-- ab1

Calvinz360 said...

Thanks for the quick response. However I didn't get the same error as that site shown. It was the typical prompt telling me that I am accessing it from outside of US.

Would that still be the same ?

--- ab1 said...

It depends..

I just tried watching Hulu (not Hulu+ via the web and it worked fine this time..

Make sure you are definitely resolving www.hulu.com to your VPS IP from the machine you are watching on and you are able to make an SSL handshake:

$ openssl s_client -connect YOUR_VPS_IP:443 -servername www.hulu.com

-- ab1

Calvinz360 said...

yes.. hulu.com is resolving to my VPS IP and provides the following openssl output :-

CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=US/ST=MA/L=Cambridge/O=Akamai Technologies, Inc./CN=a248.e.akamai.net
i:/O=Cybertrust Inc/CN=Cybertrust Public SureServer SV CA
1 s:/O=Cybertrust Inc/CN=Cybertrust Public SureServer SV CA
i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
2 s:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
i:/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEVjCCAz6gAwIBAgIOAQAAAAABRXa5+9XyrNkwDQYJKoZIhvcNAQEFBQAwRjEX
MBUGA1UEChMOQ3liZXJ0cnVzdCBJbmMxKzApBgNVBAMTIkN5YmVydHJ1c3QgUHVi
bGljIFN1cmVTZXJ2ZXIgU1YgQ0EwHhcNMTQwNDE4MjEyNDM5WhcNMTUwNDE4MjEy
NDM5WjBuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCTUExEjAQBgNVBAcTCUNhbWJy
aWRnZTEiMCAGA1UEChMZQWthbWFpIFRlY2hub2xvZ2llcywgSW5jLjEaMBgGA1UE
AxMRYTI0OC5lLmFrYW1haS5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDGIVQri2U8amwl/e7/vO44H2Ff/atjvxCY1Bey5gXKLOQ7xs1hPevL+83O
QUEzdG/5SjY+3cS+HWgguMksYidNj6Jo6U7be7TiyoeWpG4yrzh9HByMrjGQ6cPL
p2565gXIPJ5YdzGEW8grnO/p+d8D0C6mgAu7GKrscZWITDiL9W5bCl4YVg6xqYni
LR7ReEPWOc1t50cr8/IaMzXcU+Ls0LA2fiyBsNA8kSAE7iWS5phR1OWF2hNL+QbC
b1rdlfu71EhJlkYIwDeWeeTkXylpGgTnSOLGUmxC0gxbdLdwZYuqYmvfMqTHLimW
wQn7Khvqg9wMuxnjPESxuYHYMhozAgMBAAGjggEYMIIBFDAfBgNVHSMEGDAWgBQE
mGDfgBuWSV1lVi2lLAkkCuzcuTA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3Js
Lm9tbmlyb290LmNvbS9QdWJsaWNTdXJlU2VydmVyU1YuY3JsMB0GA1UdDgQWBBTK
87TzjFpmNBnrmZ0wkIaY4hHUXjAJBgNVHRMEAjAAMA4GA1UdDwEB/wQEAwIFoDAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEQYJYIZIAYb4QgEBBAQDAgbA
MEQGA1UdEQQ9MDuCDiouYWthbWFpaGQubmV0ghYqLmFrYW1haWhkLXN0YWdpbmcu
bmV0ghFhMjQ4LmUuYWthbWFpLm5ldDANBgkqhkiG9w0BAQUFAAOCAQEAU5TqUBbB
uyitm5HHiXnEUXTKcT36qpObBQWbjEtTjDD2Hj2F65F6GXTk3D4hdXC1evPXhpqe
mVJ+FIRNcOztSEtcSkgJWW/r33bRYK+e5YC4oeB33/oBFUbkyR64cTGl8Dk+D1kM
40+/o++0U8D1Qh2Q9Ftd1wPRzh2cZ+lPQG69t/mLuvAFlrurDMbUtjn+3L08zj0I
bp/w3xDEC3MhYcYLwLwi7wqivG793b36qhrnGot9SjYKKtGbRgRFxoq29CoNIwUZ
7rSPjR0CyxklDxJDMG+5zhtvWqjO1du9wthTm7uRGWuF2AzuvRJDNRErUFIu4RVM
iOeJ1m34stzoaw==
-----END CERTIFICATE-----
subject=/C=US/ST=MA/L=Cambridge/O=Akamai Technologies, Inc./CN=a248.e.akamai.net
issuer=/O=Cybertrust Inc/CN=Cybertrust Public SureServer SV CA
---
No client certificate CA names sent
---
SSL handshake has read 3576 bytes and written 680 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-SHA
Session-ID: 0A7E79F6BFE1B89EDF16DFACE37C95AC33B3DF230C58EA79FD703D7ABC45E0F5
Session-ID-ctx:
Master-Key: AFA5934FAB6262F645F8FFB106CD647FEB60E8EFB5F2C68739443C07BBEAC4F7DB0020BB2A52E3F650752C773D505059
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
Start Time: 1403277877
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
---
closed

--- ab1 said...

Hmmm... suspect Hulu foul play then.

:)

-- ab1

Calvinz360 said...

would you be able to share with me the table list of your sniproxy.conf ? I sure hope I did not miss out any of the important hosts.

--- ab1 said...

# cat /etc/sniproxy.conf

# grep '^[^#]' /etc/sniproxy.conf

user daemon

pidfile /var/tmp/sniproxy.pid

listener 1.2.3.4 80 {
proto http
}

listener 1.2.3.4 443 {
proto tls
}

table {
netflix\.com *
crackle\.com *
pandora\.com *
hulu\.com *
}

--- end sniproxy.conf

Where 1.2.3.4 is my public VPS server IP.

-- ab1

Calvinz360 said...

Thanks ab1. :)

Another note, could you direct me on how could I determine what hosts are being resolved on mobile applications ? (ios/android). Network packet capturing would be one way, but that would be too noisy

--- ab1 said...

Probably the easiest way would be to turn on query logging in DNSmasq?

Gil Blumberg said...

Sorry to raise what is an old issue, but I'm quite stuck at the same place "Ari V" was.

I have a basic knowledge of Linux and followed instructions exactly line for line.

When attempting to start the BIND service "service named start", I get the dreaded "FAILED".

The last few lines of /var/log/messages says:
Jun 24 05:40:19 vps3 named[11009]: loading configuration from '/etc/named.conf'
Jun 24 05:40:19 vps3 named[11009]: none:0: open: /etc/named.conf: permission den ied
Jun 24 05:40:19 vps3 named[11009]: loading configuration: permission denied
Jun 24 05:40:19 vps3 named[11009]: exiting (due to fatal error)

I've checked the permissions of the .db files and permissions for both appear correct.
-rw-r--r-- 1 named named 94 Jun 24 05:25 zones.override
-rw-r--r-- 1 named named 333 Jun 24 05:25 db.override

I can't quite relay how keen I am to get this working and would be very grateful if you can help me get over this part

--- ab1 said...

This is what my bind/chroot install looks like:

root@ ~]# ls -la /var/named/chroot/etc/
total 44
drwxr-x--- 4 named named 4096 Jun 24 06:10 .
drwxr-x--- 6 named named 4096 Jan 20 12:40 ..
-rw-r--r-- 1 root root 3519 May 4 2010 localtime
drwxr-x--- 2 named named 4096 Jan 20 12:40 named
-rw-r--r-- 1 named named 1121 Jun 24 06:02 named.conf
-rw-r--r-- 1 named named 2389 Jan 20 12:40 named.iscdlv.key
-rw-r----- 1 named named 931 Jun 21 2007 named.rfc1912.zones
-rw-r--r-- 1 named named 487 Jul 19 2010 named.root.key
drwxr-x--- 3 named named 4096 Jan 20 12:40 pki
-rw-r----- 1 named named 77 Feb 11 12:15 rndc.key
-rw-r--r-- 1 named named 376 Jun 19 04:25 zones.override
[root@ ~]# service named restart
Stopping named: [ OK ]
Starting named: [ OK ]
[root@ ~]#

I've chaned the ownership on the entire /var/named/* to named:named and it still works.

Try "chown -hR root:root /etc/named*", perhaps it is trying to run as root..

-- ab1

Gil Blumberg said...

Listing the permissions nailed it.
For whatever reasons, the other commands just weren't apply named:named for all files. Went through and did each one manually.

Hey presto.

Onto the rest now.

Gil Blumberg said...

Thanks for all the help, all good now.

Just for no reason really I installed the sniproxy first, made sure that was working then started from the top of the instructions.

Possibly helped that I was standing one leg facing east while singing yankee-doodle.

--- ab1 said...

Now for some quality couch time? :)

-- ab1

Gil Blumberg said...

Oh yes, lots of it!

The reality, is that while it's only $5/month to keep this running on a VPS, it's actually about $2/month less using a 3rd party service.

But where's the fun in that, right?

Anonymous said...

This was a great blog on how to get this working, however it failed for me first due to the bind-chroot option.

After uninstall bind-chroot and just using bind "yum install bind -y"

(This page showed me why BIND didn't start - down the bottom of page.

https://www.centos.org/forums/viewtopic.php?t=5314)

I then modified

nano /etc/named.conf

md /var/named/etc/
md /var/named/etc/named
cd /var/named/etc/
cp /etc/named.* .

nano /var/named/etc/named/db.override

nano /var/named/etc/zones.override

chown named:named /var/named/etc/zones.override

chown named:named /var/named/etc/named/db.override

I also added squid to the box with a very basic config just to get it working. This allows me to use my PC and get Netflix running on that.

yum install squid -y
rm /etc/squid/squid.conf

nano /etc/squid/squid.conf

acl all src all
http_port 43250
access_log /var/log/squid/access.log squid
hosts_file /etc/hosts
http_access allow all

/etc/init.d/squid restart

Just to test I allowed all incoming to be accepted by iptables

iptables -I INPUT -j ACCEPT
service iptables save
service iptables restart

Thanks again for this my apple TV works well out side the US

--- ab1 said...

Cheers bob, thanks for the updates!

-- ab1

Calvin Alexander said...

I have no idea what is happening
Error when starting named

[root@chloe ~]# service named start
Starting named:
Error in named configuration:
zone localhost.localdomain/IN: loaded serial 0
zone localhost/IN: loaded serial 0
zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
zone 0.in-addr.arpa/IN: loaded serial 0
dns_rdata_fromtext: /var/named/chroot/etc/named/db.override:3: near '<a': not a valid number
dns_rdata_fromtext: /var/named/chroot/etc/named/db.override:12: near '162.243.x.y': bad dotted quad
dns_rdata_fromtext: /var/named/chroot/etc/named/db.override:13: near '162.243.x.y': bad dotted quad
dns_rdata_fromtext: /var/named/chroot/etc/named/db.override:14: near '162.243.x.y': bad dotted quad
/var/named/chroot/etc/named/db.override:14: file does not end with newline
zone netflix.com/IN: loading from master file /var/named/chroot/etc/named/db.override failed: not a valid number
zone netflix.com/IN: not loaded due to errors.
_default/netflix.com./IN: not a valid number
[FAILED]

--- ab1 said...

You do realise '162.243.x.y' is just an example I gave, you actually have to use your IP address?

Try running "ifconfig" on the command line and see what your address is..

-- ab1

Victor Sosa said...

Hi, did you know any HAproxy configuration that doesn't need the local server? I meant that it sould be very similar of what you did with bind + sniproxy; by the way I did the setting and it works great, thanks. I was hopping to do something similar using bind+haproxy; but that post ttp://trick77.com/2014/03/02/dns-unblocking-using-dnsmasq-haproxy/ is to complicate to be implemented in the same way.

--- ab1 said...

Hello, I don't believe there is a way. You need something locally to port forward :80 and :443 to different remote ports on the proxy. You can get a cheap Raspberry Pi, wich works very well for this purpose and learn about working with Linux at the same time :)

The example given on trick77.com, works very well, but requires some maintenance as providers add change/hosts.

-- ab1

spider said...

Beside using the example IP address, another problem is that your code formatting plugin is interpreting the serial number in db.override as a telephone number, so it adds a link to tel: protocol handler.

Victor Sosa said...

port forward :80 and :443 can be done with IPTABLES, right so that's no a problem.

--- ab1 said...

Thank you for pointing this out.

I'll see if there is a way to stop Syntaxhighlighter being too clever.

-- ab1

Blackshroud said...

In regards to the issues that were being had by the AriV and getting named started, I identified this as being cause by the files in directory /var/named/chroot/etc not being owned by the user name.

A simple 'chown named:named named.conf' (and all other files within that directory) resolve the issues for me.

Named then started without any issue.

Blackshroud said...

Also, for those trying to sort this out, when installing sniproxy make sure that you modify the commands listed for the version of sniproxy that comes down from the GIT repo.

Specifically the

'yum install /root/rpmbuild/RPMS/x86_64/sniproxy-0.1-1.el6.x86_64.rpm -y' as the package name will be different based on what is on the GIT at the time.

You will be able to identify the version based on the logs shown after running 'git clone https://github.com/dlundquist/sniproxy.git'

Blackshroud said...

Also, in regards to the iptables configuration, the default policy on CentOS is to allow all inbound communications, which means that irrelevant of what you have in the iptables, the entire interwebs will be able to use your resolver.

You will need to change the default policy for iptables to drop (or reject) for inbound communications to ensure it is safe.

Makes sure you add the IP you are working from the the iptable first as allowed prior to doing this also ;)

http://wiki.centos.org/HowTos/Network/IPTables

Blackshroud said...

Most importantly relating to the iptables setup is the need to add a rule to allow inbound SSH, because if you don't you'll lock yourself out of your server completely.

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

should do the trick ;)

David said...

Hey there,

I have successfully gotten the named working and am currently stuck on the sniproxy, I've gotten it installed, the conf loaded in the /etc folder, the .init renamed and copied to the /etc/init.d folder.....here's the hitch:

If i do server sniproxy start I get an ok. It seems to start, but then immediately fails with no error message (or log file message either), if I check status, it tells me it's dead with an existing pid and lock file.

I'm trying to work through it, but hoping someone has an idea. CentOs 6.5, using sniproxy 0.3.6

Even a way to find out why it's dying would be useful. Awesome page and really appreciated, just need to struggle through the last step! :P

Daniel T said...

Hi, I got this all working pretty nicely. So first of all: thanks for your effort!
I got another question though: Could I have sniproxy forward everything it gets? So that, if I choose to have my server as DNS, it'll only proxy netflix for me; but when I use it as a vpn, it'll proxy everything. Is that possible?
As far as I can see, I'd only need to change the sniproxy.conf to have any address in its table, since bind already takes care of only replacing netflix.com with the server address (in "DNS mode"). Would that work?

--- ab1 said...

So I think your best bet is to start sniproxy with a deamon switch from the command line directly and see what it complains about. Check: https://github.com/dlundquist/sniproxy/wiki/Troubleshooting

Regarding proxying everything with sniproxy, yes all you need is to set sniproxy to accept any hostnames. Although I am not sure if a wildcard hostname is supported in the config file, but if it is, you should be fine, otherwise you'll have to put it all the host names you are planning to proxy in there.

-- ab1

amar Sanghera said...

Hello There,

i got the bind working but unable to install sniproxy getting following error when i run ./autogen.sh && ./configure && make dist. I looked through all the comments and didn't find any one else having this issue.
can you please what might be causing this. thanks

[root@vultr sniproxy]# ./autogen.sh && ./configure && make dist
./setver.sh: line 35: debchange: command not found <<<<<<<<<<<

--- ab1 said...

Have you tried Google?

http://manpages.ubuntu.com/manpages/precise/en/man1/dch.1.html

http://stackoverflow.com/questions/18957811/debian-packages-on-xubuntu-install-debchange-dch

You probably want to:
# sudo apt-get install devscripts

-- ab1

s said...

Having the same issue with debchange on CentOS 6.5 . Was working with a previous VPS now having issues with install

Michael Crook said...

I'm having issues... I get to the point where I type telnet serverip 80 and Im getting could not open connection to the host, on port 80: connection failed. The main diference in my setup was that I used the following rule for my iptables:

[root@server]# iptables -I INPUT 1 -j ACCEPT
[root@server]# iptables -I OUTPUT 1 -j ACCEPT

this is because the machine I was on wasn't my home device and this was for testing purposes (I was going to restrict later)

Oh also, my sniproxy I had to use their install guidelines as yours wern't working.

It's being hosted on azure and I've added endpoints for the 3 ports as described by your iptables

--- ab1 said...

So what specific issue(s) are you having when building/installing sniproxy on your Azure VM?

-- ab1

neeshu said...

Hi Great tutorial, I have been using it for 2 months without any error and i want to thank you for that.
Recently my VPS was held hostage to target an attack as it is an Open DNS Resolver(I am only using it for SNI Proxy).
What steps have you taken to harden your VPS, can you please suggest some if you have got time.
I have added all the firewall rules as you suggested in your post and also am running Denyhosts.

--- ab1 said...

Hello, thank you for your support.

I think all you need is the following restrictions in your iptables configuration:

# iptables -A INPUT -s 86.144.x.y/32 -p udp -m udp --dport 53 -j ACCEPT
# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 80 -j ACCEPT
# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 443 -j ACCEPT

Where 86.144.x.y is your home IP address, from which you will be accessing the DNS/HTTP proxy on your VPS.

Provided those rules are in place, no one else on the internet will be able to query your DNS or proxy through your VPS.

Make sure you test this is actually the case once you put the rules in place, by trying to do a DNS lookup from an unauthorised IP.

-- ab1

raja selvam said...

Hi Ab,

Thank You for your help. I'm planning to implement like commercial for unblocker service like unotelly.com.

For that What i have to do?. Give me valuable suggestions?.

Thank You.

Cloud

--- ab1 said...

Dear Cloud, the most valuable advice I can give you is to do your own research using all of the free tools available to you, such as Google.

And also remember, when people give you money for a service, they expect quality, which for a technical service usually means a lot of hard work in designing it correctly from the start.

Good luck!

-- ab1

David said...

Hey there guys,

In the past I've been given dns numbers by friends for the western digital tv live plus box, and they worked just fine...for a while.

Any suggestions on what those dns's might have been doing to make it work for western digital?

--- ab1 said...

Hi David, do you mean that the 3rd party DNS server you were using before with your WD box worked and this configuration does not?

-- ab1

David said...

See that's the catch, I'm not sure what configuration they were running on those numbers.

The new setup you VERY helpfully provided works great...but of course my WDTVlive doesn't seem to use SNI....so it doesn't work.

I may either replace it with a different device or try something with the router, but those previous dns numbers DID work without any changes on my part...I'm just trying to see if I can identify what they did to have that happen...perhaps I can duplicate it :)

Any suggestions as usual are hugely apppreciated :)

--- ab1 said...

So you were probably using a more advanced commercial service, where they have a different proxy listener for each distinct hostname (we are proxying everything using one listener).

If you wanted to set something like that up yourself, you could follow this guide:
http://trick77.com/2014/03/02/dns-unblocking-using-dnsmasq-haproxy/

This one works pretty well, but you'll need an additional piece of kit at home - a small Linux router with DNSmasq and IPtables.

-- ab1

ChevyLin said...

That’s a good article. I personally prefer UnoTelly. It’s similar to unblock-us (or other DNS service) but they offer a DNS server close to physical location and I can achieve better performance.

Mikael said...

If you want to use apache/nginx/whatever as a httpd server you can easily do that:
1. Change Apache default ports, in my case I changed 80 to 81 (didn't try https)
2. Add a rule in the table part of sniproxy.conf: yourdomain.com 127.0.0.1:81
3. Restart apache and sniproxy (in that order)

--- ab1 said...

Good point, I am also hosting a small public site using httpd on another VPS with sniproxy where I've configured httpd to bind to localhost:80 and in my sniproxy.conf, I've go the following:

listener my.vps.public.ipv4 80 {
proto http
}

table {
mydomain.\info 127.0.0.1:80
}

So snipropxy is binding the the public IP and sends any traffic for the local domain to my httpd instance. Seems to work fine so far.

-- ab1

Humphrey Davy said...

I am just worried about the bad guys using my vps for DNS DDoS Amplification Attack. Would Iptables be enough to secure DNS. Since, I am sharing my vps IP with many friends and colleagues. I feel concerned. Any advice??

--- ab1 said...

You can add a directive into named.conf to limit the DNS response rate as follows:
https://deepthought.isc.org/article/AA-00994/0/Using-the-Response-Rate-Limiting-Feature-in-BIND-9.10.html

Seems to work pretty well on my box. If you are also all using static IPs, you could lock it right down in up tables, but I realise that isn't always practical.

-- ab1

Juan Jose Rodriguez Ponce said...

Nice tutorial. I got it all working but Im still getting my local netflix (scratching head).

Im sure Ive changed the dns servers (did so by editing /etc/resolv.conf).

Also to be even more sure I shut down sniproxy and tested..netflix.com times out. Turn sniproxy back up and..local netflix. Cant figure it out. Any advices?

btw there seems to be a mistake in your bind configurations:

include "/etc/named/etc/zones.override";

should be:

include "/etc/named/zones.override";

--- ab1 said...

Thank you for spotting the typo, I'll update the guide.

Make sure you are changing DNS servers on the machine you are watching Netflix on, not on your VPS.

There are some diagnostic tests at the end of the tutorial, which hopefully you've run locally on yor VPS. The DNS test should resolve to your local VPS IP.

Then also try resolving it directly against your local BIND instance:
dig netflix.com @127.0.0.1

Have a look at the response to make sure it is giving you a US Netflix cluster..

-- ab1

Juan Jose Rodriguez Ponce said...

..yep all that those tests resut as expected and I did changed my local DNS.

Help me to understand something so maybe I will figure this out. They way your hack works is:

1. My local device (the one running netflix) resolves any netflix doain to my remote VPS

2. VPS receives the call on port 80 and 443 from my local and forwards it to Netflix actual servers

3. Netflix servers receive the call with the src ip of my VPS so they respond as they would to an american client.

.. but if this is correct then all the streaming would be going through the VPS..so I most be wrong somwhere

Mikael said...

Are you 100% sure where your server is located? Try tunnel through it and visit www.ip2location.com. Or even better, add that domain to bind and SNI proxy. What does that tell you?

Oh, here's a link everyone should read:
http://gizmodo.com/it-looks-like-netflix-is-cracking-down-on-vpn-pirating-1677277648

--- ab1 said...

Very interesting.. I also found this:
http://www.engadget.com/2015/01/03/netflix-clamps-down-on-vpns/

Response from Netflix apparently. Not sure how true, but may not be as bad as the original article suggests.

In any case, your own DIY solution should be much more bullet proof with respect to these counter measures, as long as not too many Netflix users appear to come from a single IP at the same time.

-- ab1

Phil said...

Really appreciate this guide - most thorough I have found on the net so far for "smart dns" setup. I've not put it into practice yet but all makes sense (which is a good thing!). Once I have this working i'm going to make a YouTube guide too, with credits to you ad the other sources I found. Excellent!

Anton Belodedenko said...

Thank you, hopefully it all works for you.

If you encounter any issues, do let me know and I'll try to help..

-- ab1