Wednesday 22 April 2015

Watch HBO Now in the UK for free*



TL;DR head to netflix-proxy on GitHub or subscribe to Unzoner VPN service to un-block:



Apple slashed the price of their Apple TVs to £59 in the UK to coincide? with the release of HBO Now on the same. However, unfortunately the app is only available on the US Apple TVs.

So what about all the people in the Old World, eager to purchase a subscription and watch quality HBO programming?

Such conversations go something like this:
customer: "HBO, please take my UK pounds!"
HBO: "Sorry, we can't, but we would *really* like to?"


So to help HBO take your UK £'s, here is how:

  1. Create your own US mail drop address (e.g. www.shipito.com)[n1]
  2. Create iTunes US Store account with your US postal address
  3. Purchase iTunes US gift card (e.g. www.offgamers.com)
  4. Load your gift card onto the iTunes account
  5. Change your ATV to use your US account and switch stores (in settings)
  6. Configure geo-bypass for HBO Now (e.g. github.com/ab77/netflix-proxy)[n2]
  7. Enjoy!


Footnotes
[1] You now have an official (and legal) US postal address to use for correspondence.

[2] This is a Netflix-proxy-in-a-box solution I put together using open source software and Docker hosted on Digital Ocean. It will work just the same, as long as you add hbonow.com to the relevant config files along side netflix.com:

edit /opt/netflix-proxy/data/sniproxy.conf and add:


table {
    .*\.hbonow\.com *
    hbonow\.com *
}

edit /opt/netflix-proxy/data/zones.override and add:

zone "hbonow.com." {
    type master;
    file "/data/db.override";
};

restart Docker containers:

# docker restart bind sniproxy


Monday 23 March 2015

Netflix smart DNS proxy in six steps with Docker and Digital Ocean..



TL;DR head to netflix-proxy on GitHub or subscribe to Unzoner VPN service to un-block:




Last year I wrote an article about creating your own smart DNS proxy to watch Netflix out of region.

I have now compressed this solution into something which can be deployed in six easy steps, using Digital Ocean as your hosting provider.

Head over to my GitHub page and follow the instructions detailed there.




Prerequisites


You'll need the following:
  • A working Internet connection at home
  • A Netflix subscription
  • Digital Ocean account


Instructions


These instructions are also detailed on my GitHub page:
  1. Head over to Digital Ocean[1] and create an account and get $10 USD starting credit enough for 2 months hosting
  2. Create a Droplet using Docker 1.5.0 on Ubuntu 14.04 (find in under Applications images)
  3. Make sure you create the Droplet in the right location, for example if you want to watch US content, create in the US.
  4. SSH to your Droplet and run the following command:
  5. cd /opt && git clone https://github.com/ab77/netflix-proxy.git && cd netflix-proxy && ./build.sh
  6. Point your DNS at the Droplet IP and watch Netflix out of region.
  7. Enjoy!

Note, this won't work on devices which don't support the use of SNI during SSL handshake (e..g Chromecast, PS3, most TVs), you'll have to use an alternative system described in here.


Footnotes
[1] I am part of the Digital Ocean's referral program.


Monday 12 January 2015

Designing the perfect motorhome or how I stopped worrying and learned to love DIY?



Part 1: How to Build a Motorhome from a VW Crafter



The plan is to convert a standard MWB (that's a technical acronym in the industry for "Medium Wheel Base") Volkswagen Crafter delivery van into a motorhome for general camping and rock climbing all over Europe and possibly beyond.

This is the first entry, in a series of many, which are going to document the transformation of an ordinary delivery van into an open plan studio apartment on four wheels. Given the average size of a London bedroom, I feel the preceding statement is fully justified.


But enough with the introductions, I've taken liberty of drawing out a boxy design of what the finished internals may look like, should everything go according to plan..


Update 13/05/1015: the next instalment is ready, read it here.


The Thumbnail (2D)


So this is basically the day layout of the van, looking from the front towards the back.




Permit your imagination to strain a bit, in order to imagine what the night time layout of this modern automotive convenience may look like (hint: that couch looking thing with boxes underneath, folds out into a bed).


However to get the maximum understanding of what this is actually going to look like and what all the bits are, you'll need to scroll down..



3D View


Given 3D is all the rage these days, i've taken the liberty of including a 3D web viewer. The model has been created with free software called SketchUp Make, which I recommend for projecting/prototyping work. You can even send your designs straight to a 3D printer, should you happen to have one. We won't be 3D printing this one however.




As you pan, zoom and fly around the model, you'll see that the web viewer is fairly basic in the functionality it offers. To get the most you'll need to download the SketchUp files and open them locally on your computer.



Shopping List


As with most of my projects, the budget has been slowly creeping up. I believe this is due to a phenomenon known as "scope creep". In any case, for budgeting purposes, anything below £20K would be acceptable.




I think the above is the minimum amount of kit any self-respecting motorhome should carry. Some things like the generator, awning and roof racks are probably optional of shorter journeys, but will definitely come in useful on longer trips.

There will no doubt be sundries I haven't accounted for, but these should not amount to more than £1K, which should still place this project loosely on budget.


Coming Up Next...


In the next instalment, if all goes well, we should be fitting windows, insulating and lining the van and installing roof lights, in the comfort of dark, cold Northern European winter evenings and nights (this author also has a day job).




Friday 9 January 2015

X10 remote boiler control



Remote boiler control with Ferroli ROMEO W/RF OpenTherm thermostat and Marmitek UM7206 X10 module


Overview


My wife and I have been on a couple of back to back trips in December, when the outside temperature in London dropped below zero degrees centigrade.

We turn our heating off when we are away, so when we came back, the flat was at a balmy 7 degrees centigrade. Needless to say, it took a number of days to heat it back up to reasonable temperature.

Since we already have a Ferroli boiler connected to a ROMEO W/RF (P/N: 013101XA) wireless OpenTherm thermostat, I've decided to try set it up to enable us to switch the heating on remotely.

I always wondered what those "GSM" terminals on the RF bridge were for. In the thermostat manual, I noticed the following at the bottom of the last page :
If the telephone contact (voltage-free contact) has to be connected, use the "GSM" terminals
I asked Ferroli UK support to clarify the operation of these contacts, but they didn't know and referred me to the factory in Italy. At the time of writing, the factory has not come back to me.

I did however stumble upon a hint on one of the forums on how to make this work,  so if you have one of these thermostats and would like to set it up for remote control away from home, read on..


Prerequisites


In order to set this up, you will need the following:

This guide assumes you have a working X10 stack already.


Basic Operation


By default, the thermostat is configured to switch the boiler OFF when the "GSM"  contacts on the ROMEO W/RF wireless bridge PCB are connected together and resume normal (daily/weekly programmed) operation when the contacts are disconnected.

This guide assumes you are looking to switch your boiler ON instead, when the GSM  contacts are connected together.



Thermostat Setup


You'll need to change the following registers, by pressing and holding down the "P" button (button #7 in the diagram from the manual) and then keep pressing it until you get to P8 register. The following table from the manual lists the defaults for P8 and P9 registers:


Set P8 to 1 (1=Switch ON heating when GSM contacts are connected together) and also set P9 to desired temperature in this mode (e.g. 21.5C). This is the temperature the thermostat will try to maintain, while it is in the remote operation mode.


Marmitek UM7206 Setup


Set the unit and house codes on the device using the two dials to match your X10 configuration. Set the slider on the left to "continuous" and the slider on the right to "relay only" (unless you want a bit of noise pollution).


Wiring


Connect a two-core wire between the GSM contacts on the ROMEO W/RF wireless bridge to the output relay contacts on the UM7206 (polarity doesn't matter) and plug the module into the mains. I used the same cable I used to wire up the external temperature compensator probe (1KWMA62U).


Testing


Switch your boiler OFF using the ROMEO W/RF control panel, as you would normally do when you go away for long periods of time. The icon labelled #14 on the diagram of the RF remote control in the manual, should light up. Your boiler should now be in holiday mode (i.e. it won't switch ON according to your program, but will keep the temperature of the water in the pipes above 4-5C Celsius to prevent damage.

Now, push the manual ON switch on your UM7206 unit. Observe the RF remote control panel, it should show the boiler entering "manual heating mode" (icons #15, 21 and 19) as well as an undocumented flashing icon "()", which should appear below #23:



Note, it may take up to 120 seconds for the boiler/thermostat to respond as the "GSM" contacts are changed, so please be patient.

The boiler will work in this mode and maintain a constant temperature, set using register P9. Also, in this mode, the ROMEO remote control panel will not allow the temperature to be changed, until such time, that the GSM contacts are disconnected (i.e. you push the OFF button on the UM7206).

That is about it. The UM7206  relay can be controlled using X10 commands from your X10 controller and if you have allowed access to it from the Internet, you can switch your boiler on remotely a day or so before you are due back from your holiday. That way you arrive at a warm house, instead of having to sit in your woollen hat and gloves nursing a cup of hot tea, while waiting for your dwelling to return to the habitable zone.

When you do arrive back, don't forget to switch the boiler back into programmable mode by either pushing the OFF button on the Marmitek UM7206 or by issuing the appropriate X10 command via your interface.



Other Applications



I choose to use X10 interface because I already have it up and running. If you don't want to use the X10 interface, there are a number of other ways to implement this remote switching capability.

All you really need is a relay you can control with a computer, which is going to close/open the GSM contacts on the ROMEO wireless RF bridge. You can use a single channel USB relay module, plugged into a computer of your choice (e.g. Raspberry Pi) and as long as you can connect to your computer remotely, you can issue a command to control the connected relay from anywhere.

However, regardless of the control implementation you choose, I would recommend to secure your control system using either a VPN and/or SSL with strong password.


Notes

[n1] For some reason ROMEO W/RF is not available in the UK, but you can order it direct from eBay.it or from vendors in Italy/Poland/Russia.
[n2] I use CM12U connected to Raspberry Pi running HEYU with domus.Link front-end.
[n3] Ferroli ROMEO W/RF manual (EN)



Friday 12 December 2014

Run your own proxy server to watch BBC iPlayer outside the UK



TL;DR head to netflix-proxy on GitHub or subscribe to Unzoner VPN service to un-block:



If you are located outside of the UK and you want to watch a subset of content the British Broadcast Corporation (BBC) makes available online via BBC iPlayer, you are out of luck. This content is only available to users based in the UK (or you could pay BBC for it).

You could also pay for commercial services, such as HolaUnblock-Us and 
Smart DNS Proxy to name a few, which allow you to bypass the geo restriction.

Alternatively, or you can knock together a similar solution quickly yourself for a cost of a cheap Virtual Private Server (VPS). How cheap, depends on where you decide to host your VPS. For my hosting I choose Digital Ocean with the VPS hosted in a London data center for $5 USD per month. If you sign up with the link above, you will receive $10 USD credit, which is enough for two months of hosting. Also, the more friends you have sharing a solution like this, the more cost effective it becomes, since a single VPS can support many individuals watching BBC iPlayer at once.

Update 03/2015: I've now packaged the entire system using Docker images. This reduced the deployment to three easy steps. Head over to GitHub and follow the instructions there: https://github.com/ab77/iplayer-proxy. 

However if you would still like to create this system from scratch, please read on..

In essence, the following solution works by selectively proxying certain requests (*.bbc.co.uk and *.bbci.co.uk) via a server located in the UK, thus bypassing BBC iPlayer application level geo-restrictions. This solution, unlike VPN solutions, does not proxy the actual video/audio delivery, which uses different domain(s), so you should still get the benefit of decent bitrates from your local Content Distribution Network PoP.

Note, this won't work on devices which don't support the use of TLS Server Name Indication during SSL handshake (such as WD TV), you'll have to use a more complex system described in this guide or pay for Hola/Unblock-Us/Smart DNS Proxy/etc.



Prerequisites


You'll need the following:
  • A working Internet connection at home
  • A UK based VPS with root access and clean CentOS x86 6.x image
  • Basic knowledge in navigating a Linux console


Configuration


The configuration depends on the following two components:

  • DNS server
  • HTTP/SSL proxy

For the purpose of this demo, we assume your VPS's public Internet IP is 178.62.x.y and your host is called myhost.

We also assume your ISP has assigned 86.50.x.y to your home router. You can check what your IP is by going to http://www.whatismyip.com/.

Note, if you ISP assigns you a dynamic IP, you will need to update your configuration every time this changes. Better ask them for a static IP address.

When creating your configuration files, please change the place holder IPs to your actual IPs.


Base VPS Deployment


This step depends on the VPS provider you choose, but broadly consists of the following steps:
  • Create account with VPS provider
  • Deploy a CentOS x64 6.x image to a UK based data centre.

Once your VPS server is created, you will get a public IP address and SSH credentials to use to login. First, establish an SSH session to your server and install additional repositories:
[root@myhost ~]# yum -y install epel-release
[root@myhost ~]# yum -y update

Install certain pre-requisites to make your life a little easier:
[root@myhost ~]# yum install openssl vim git wget curl bind-utils telnet -y

Install development tools to enable you to build packages later:
[root@myhost ~]# yum groupinstall "Development Tools" -y
[root@myhost ~]# yum install rpmbuild autoconf automake curl libev libev-devel pcre pcre-devel perl udns-devel -y


IP Tables (firewall)


Since our VPS is on a public Internet, we are going to firewall it off from all the unpleasantness out there.

Add rules to allow DNS, HTTP and HTTPS inbound from your home's public IP:
[root@myhost ~]# iptables -A INPUT -s 86.50.x.y/32 -p udp -m udp --dport 53 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.50.x.y/32 -p tcp -m tcp --dport 80 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.50.x.y/32 -p tcp -m tcp --dport 443 -j ACCEPT
[root@myhost ~]# service iptables save
[root@myhost ~]# service iptables restart



BIND (DNS server)


BIND is going to be our DNS server. It will respond with our VPS IP to DNS requests for bbc.co.uk and bbci.co.uk domains.

Note, we are configuring the DSN server to run in a chroot jail and as a non-root user, to provide added security and minimise the potential effects of a security compromise.

Install BIND:
[root@myhost ~]# yum install bind -y


Edit your /etc/named.conf and make it look as follows:
options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { any; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-recursion { none; };
        allow-transfer  { none; };
        recursion no;
        additional-from-auth no;
        additional-from-cache no;

        auth-nxdomain no;    # conform to RFC1035
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };
};

acl "trusted" {
        any;
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/etc/named/zones.override";


Edit your /etc/named/db.override and make it look as follows:
$TTL  86400

@   IN  SOA ns1 root (
            YYYYMMDD01  ; serial
            604800      ; refresh 1w
            86400       ; retry 1d
            2419200     ; expiry 4w
            86400       ; minimum TTL 1d
            )

    IN  NS  ns1
ns1 IN  A   178.62.x.y
@   IN  A   178.62.x.y
*   IN  A   178.62.x.y

* Note, change the above IP address to your the public IP of your VPS and update the datestamp.
Edit your /etc/named/zones.override and make it look as follows:
zone "bbc.co.uk." {
        type master;
        file "/etc/named/db.override";
};

zone "bbci.co.uk." {
        type master;
        file "/etc/named/db.override";
};


Note, you may need to change the ownership of the new files you created from root to named:
[root@myhost ~]# chown named:named /etc/named/zones.override
[root@myhost ~]# chown named:named /etc/named/db.override


Install BIND (chroot package):
[root@myhost ~]# yum install bind-chroot -y


Finally, start BIND:
[root@myhost ~]# rndc-confgen -a -r /dev/urandom -t /var/named/chroot
[root@myhost ~]# chkconfig named on
[root@myhost ~]# service named start


From one of your home machines, run the following quick test:
$ nslookup www.bbc.co.uk 178.62.x.y
Server:         178.62.x.y
Address:        178.62.x.y#53

Name:   www.bbc.co.uk
Address: 178.62.x.y

$ nslookup www.bbci.co.uk 178.62.x.y
Server:         178.62.x.y
Address:        178.62.x.y#53

Name:   www.bbci.co.uk
Address: 178.62.x.y


You should get the IP address of your VPS in the DNS response.


SNI Proxy (HTTP/SSL proxy)


SNI Proxy is going to be our HTTP/SSL proxy, which supports for TLS SNI extensions.

Build and install SNI Proxy:
[root@myhost ~]# cd /opt
[root@myhost ~]# git clone https://github.com/dlundquist/sniproxy.git
[root@myhost ~]# cd ./sniproxy
[root@myhost ~]# ./autogen.sh && ./configure && make dist
[root@myhost ~]# rpmbuild --define "_sourcedir `pwd`" -ba --nodeps redhat/sniproxy.spec
[root@myhost ~]# yum install $(ls /root/rpmbuild/RPMS/x86_64/sniproxy-[0-9]*.rpm) -y


Note: if the RPM fails to install, check the exact name of the file in the /root/rpmbuild/RPMS/x86_64/ directory.


Also, please review the ./configure command output to make sure LIBUDNS or libudns  detected, if it hasn't, you will not be able to get SNI Proxy to work with the below configuration file. On a Red Hat/CentOS server, the library should be available from EPEL repository (yum install udns-devel), however for other distributions, you will need to build and install it manually.

Edit your sniproxy.conf and make it look as follows:
[root@myhost ~]# vim /etc/sniproxy.conf

user daemon
pidfile /var/tmp/sniproxy.pid

resolver {
        nameserver 127.0.0.1
}

error_log {
        filename /var/log/sniproxy_error.log
        priority notice
}

listener 178.62.x.y 80 {
        proto http
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

listener 178.62.x.y 443 {
        proto tls
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

table {
        .*\.bbc\.co\.uk *
        bbc\.co\.uk *
        .*\.bbci\.co\.uk *
        bbci\.co\.uk *
}

Finally, install the start-up script and start SNI Proxy:
[root@myhost ~]# cp ./redhat/sniproxy.init /etc/init.d/sniproxy
[root@myhost ~]# chmod +x /etc/init.d/sniproxy
[root@myhost ~]# chkconfig sniproxy on
[root@myhost ~]# service sniproxy start


Note, if the proxy complains about ports being in use, check if you got another web server already running and if so, disable it or move it to a different port.

From one of your home machines, run the following quick test:
$ telnet 178.62.x.y 80
Trying 178.62.x.y...
Connected to 178.62.x.y.
Escape character is '^]'.
^C
Connection closed by foreign host.

$ openssl s_client -servername www.bbc.co.uk -connect 178.62.x.y:443
CONNECTED(00000003)
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=GB/ST=London/L=London/O=British Broadcasting Corporation/CN=*.bbc.co.uk
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MII...


You should get a socket connection on port 80 and get an SSL handshake with the BBC.


Putting It All Together


The last step is to change the DNS server on the device you are planning to watch BBC iPlayer on. Set it to your VPS public IP (i.e. 178.62.x.y)

Fire up BBC iPlayer and you should be able to play back all the wonderful content they make available - enjoy!


Note, this won't work on devices which don't support the use of SNI during SSL handshake, you'll have no choice but to use a system described in this guide.


References


I've used the following reference material to prepare the solution described in this article. Many thanks to the respective authors.


Monday 23 June 2014

Pretty dashboards with FortiOS, sFlow and Splunk Storm



Overview


Want to build a pretty dashboard to show bandwidth utilisation on your Internet link or your internal network? Read on..



Prerequisites


You'll need the following:
  • A Fortigate firewall  with sflow support (4.0 MR2+)
  • A Splunk Storm project
  • A Linux server (Raspberry Pi is perfect)
  • Basic knowledge in navigating a Linux console of FortiOS CLI



Configuration


The general approach is as follows:
  • FortiOS sends sflow messages to a Linux server
  • Linux server writes these messages to syslog
  • Syslog is configured to forward messages to Splunk Storm
  • Splunk Storm is configured to extract sflow events and graph them



FortiOS


To configure the firewall, issue the following commands at the FortiOS CLI.

Enable sflow and configure collector IP:

config system sflow
    set collector-ip 
end

Enable sflow on the both internal and external interfaces:
config system interface
    edit "internal"
        set sflow-sampler enable
        set sample-rate 512
        set polling-interval 30
    next
    edit "external"
        set sflow-sampler enable
        set sample-rate 512
        set polling-interval 30
    next 
end
Note: your interface names may differ, adjust appropriately.


Syslog


We need a small Linux server to receive sflow traffic from the firewall and forward it as syslog messages to Splunk. I am using an old Raspberry Pi for this, running Debian 7.4.
First, you need to download, build and install sflowtool as described in the link, but broadly the procedure is:

[pi@rpi ~]# sudo mkdir /opt/sflow
[pi@rpi ~]# cd /opt/sflow
[pi@rpi ~]# sudo wget http://www.inmon.com/bin/sflowtool-3.22.tar.gz
[pi@rpi ~]# sudo tar -xvzf sflowtool-3.22.tar.gz
[pi@rpi ~]# cd sflowtool-3.22
[pi@rpi sflowtool-3.22]# sudo ./configure
[pi@rpi sflowtool-3.22]# sudo make
[pi@rpi sflowtool-3.22]# sudo make install
Next, you need to install and configure rsyslog per the guide prepared by the nice people at Splunk. Again, broadly you need to add a line at the bottom of rsyslog.conf to send all events to Splunk Storm as follows:

[pi@rpi ~]# sudo vim /etc/rsyslog.conf

# logging to Splunk Storm (via TCP)
*.* @@tcp..data.splunkstorm.com:
Note: to find out what is your Splunk Storm host name and TCP port, see this guide.


Supervisord


In order to ensure our sflow collector is always collecting events and forwarding them to syslog, we can use supervisord and a helper script on our Linux server as follows:
[pi@rpi ~]# sudo vim /opt/sflow/sflowlogger.sh

#!/bin/bash
/usr/local/bin/sflowtool -l | logger -t sflow

[pi@rpi ~]# sudo chmod +x /opt/sflow/sflowlogger.sh
If you don't already have supervisord installed, install it by running:
[pi@rpi ~]# sudo apt-get install supervisor
Note: the following guide has a bit more detail on configuring supervisord.

Once installed, configure it to start sflowtool and logger using the helper script:
[pi@rpi ~]# sudo vim /etc/supervisor/conf.d/sflow.conf

[program:sflow]
command=/opt/sflow/sflowlogger.sh
directory=/opt/sflow
process_name=%(program_name)s
autostart=true
autorestart=true
stopasgroup=true
stdout_logfile=/opt/sflow/sflow.log
redirect_stderr=true
stopsignal=KILL

[pi@rpi ~]# sudo supervisorctl reread
[pi@rpi ~]# sudo supervisorctl start sflow
[pi@rpi ~]# sudo supervisorctl status
sflow                            RUNNING    pid 5551, uptime 0:05:49
Note: apparently you need to have at least version 3.0b2-1 of supervisord to support stopasgroup feature, so unless you do you will need to clean-up logger and sflowtool processes if you manually stop the supervised program.


Splunk Storm


Splunk Storm is a hosted Splunk service, which at the time of writing is free to 20GB of log data. this is plenty for a home network sending periodic bandwidth measurements.

First login to your Splunk Storm account and set up a few things:

  1. Click "Explore Data" button under your project.
  2. Click the gear symbol at the top right and click "Event types".
  3. Create a new event type called "sflow" with search string "sourcetype=syslog sflow CNTR" and tag "sflow".
  4. Click the gear symbol at the top right and click "Tags".
  5. Create a new tag called "sflow" with field/value pair "eventtype=sflow".
  6. Click the gear symbol at the top right and click "Fields", then click "Field extractions".
  7. Create a new field extraction called "EXTRACT-sflow-fields" with the following regex:

(?[0-9.]+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+),(?\d+)
Now you should be able to check if there are events coming through from your Linux server and create a test report as follows:
  1. Click on "Search" at the top left
  2. Run the following search:
You should see results resembling this:
tag=sflow | head 10
Splunk search showing sflow events

Save this search as a report called "sflow".


Lastly, create a dashboard with a couple of dynamic drop-down boxes, so you can easily visualise your bandwidth consumption:

  1. Click the gear symbol at the top right and click "Dashboards".
  2. Create a new dashboard called "bandwidth" and paste the following code into it
<form>
  <label>bandwidth</label>
  <fieldset autoRun="true" submitButton="false">
    <input type="time" searchWhenChanged="true">
      <default>Last 60 minutes</default>
    </input>
    <input type="dropdown" token="aggr" searchWhenChanged="true">
      <label>Aggregation</label>
      <default>Minute</default>
      <choice value="1s">Second</choice>
      <choice value="1m">Minute</choice>
      <choice value="1h">Hourly</choice>
      <choice value="1d">Daily</choice>
      <choice value="1w">Weekly</choice>
      <choice value="1mon">Monthly</choice>
      <choice value="1y">Yearly</choice>
    </input>
    <input type="dropdown" token="interface" searchWhenChanged="true">
      <label>Interface</label>
      <default>Internal</default>
      <choice value="1">External</choice>
      <choice value="8">Internal</choice>
    </input>    
  </fieldset>
<row>
    <chart>
      <searchstring>
        tag=sflow ifIndex=$interface$
        | streamstats window=2 current=t global=f first(ifInOctets) as f_ifInOctets last(ifInOctets) as l_ifInOctets first(ifOutOctets) as f_ifOutOctets last(ifOutOctets) as l_ifOutOctets by ifIndex
        | eval ifInOctets=abs(l_ifInOctets - f_ifInOctets)
        | eval ifOutOctets=abs(l_ifOutOctets - f_ifOutOctets)
        | eval in_mbps=ifInOctets * 8 / 2014 /2014
        | eval out_mbps=ifOutOctets * 8 / 2014 /2014
        | timechart span=$aggr$ mean(in_mbps) mean(out_mbps)
      </searchString>
      <earliesttime>$earliest$</earliestTime>
      <latesttime>$latest$</latestTime>
      <option name="charting.axisTitleX.visibility">visible</option>
      <option name="charting.axisTitleY.visibility">visible</option>
      <option name="charting.axisX.scale">linear</option>
      <option name="charting.axisY.scale">linear</option>
      <option name="charting.chart">area</option>
      <option name="charting.chart.nullValueMode">gaps</option>
      <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
      <option name="charting.chart.stackMode">default</option>
      <option name="charting.chart.style">shiny</option>
      <option name="charting.drilldown">all</option>
      <option name="charting.layout.splitSeries">0</option>
      <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
      <option name="charting.legend.placement">right</option>
    </chart>
  </row>
</form>
Note: on my system my external interface has ifIndex=8 and my internal one is ifIndex=1. Adjust your dashboard definition appropriately.

You should get a pretty dashboard, which looks something like this:

References


I've used the following reference material to prepare the solution described in this article. Many thanks to the respective authors.


Friday 7 February 2014

DIY clone of Netflix Tunlr/Unblock-Us/UnoTelly on cheap US based VPS



TL;DR head to netflix-proxy on GitHub or subscribe to Unzoner VPN service to un-block:





(for posterity) If you are in the UK and would like to access Netflix US catalogue, which is about three times bigger, read on.

There are commercial services out there, such as Unblock-Us and Smart DNS Proxy to name a few, so you could pay for the privilege to use their services or you can knock together a similar solution quickly yourself for a cost of a cheap Virtual Private Server (VPS). How cheap, depends on where you decide to host your VPS.
In essence, the following solution works by selectively proxying only certain requests (*.netflix.com) via a server located in the US, thus bypassing Netflix application level geo-restrictions. This solution does not proxy the actual video delivery, which uses different domain(s).

Please note, this basic solution doesn't work on devices which don't support the use of SNI during SSL handshake (e.g. Netflix on Chromecast). To support such devices, you will need to extend this solution to selectively proxy individual Netflix hosts using something akin to HAProxy, go with a commercial provider or switch to black.box unzoner.

Also, if you are looking how to watch BBC iPlayer abroad, please see the flowing guide: http://blog.belodedenko.me/2014/12/watch-bbc-iplayer-in-australia-for-free.html

Update 04/2015: You can also watch HBO NOW on your Apple TV outside the US using this solution, here's how: http://blog.belodedenko.me/2015/04/watch-hbo-now-in-uk-for-free-apple.html

Update 05/2015: You can also watch Hulu Plus on your Apple TV outside the US using this solution, here's how: http://anton.belodedenko.me/watch-hulu-plus-outside-of-the-us/

However, if you still want to build the system manually, perhaps to understand how it all works, please read on...

Prerequisites


You'll need the following:
  • A working Internet connection at home
  • A Netflix subscription
  • A US based VPS with root access and clean CentOS x86 6.x image
  • Basic knowledge in navigating a Linux console
For my hosting I choose Digital Ocean with the VPS hosted in a New York data centre for $5 USD per month. If you sign up with the above link, you will receive $10 USD credit, which is enough for two months hosting. Note, Netflix has blocked Digital Ocean IP address ranges, so don't expect it to work. Other services may still work though.

Configuration


The configuration depends on the following two components:
  • DNS server
  • HTTP/SSL proxy
For the purpose of this demo, we assume your VPS's public Internet IP is 162.243.x.y and your host is called myhost. We also assume your ISP has assigned 86.144.x.y to your home router.

You can check what your IP is by going to http://www.ifconfig.io/. Note, if you ISP assigns you a dynamic IP, you will need to update your configuration every time this changes. Better ask them for a static IP address. When creating your configuration files, please change the place holder IPs to your actual IPs.

Base VPS Deployment


This step depends on the VPS provider you choose, but broadly consists of the following steps:
  • Create account with VPS provider
  • Deploy a CentOS x64 6.x image to a US based data centre, ideally on the east coast.
Once your VPS server is created, you will get a public IP address and SSH credentials to use to login.

First, establish an SSH session to your server and install additional repositories:
[root@myhost ~]# yum -y install epel-release
[root@myhost ~]# yum -y update

Install certain pre-requisites to make your life a little easier:
[root@myhost ~]# yum install openssl vim git wget curl bind-utils telnet -y

Install development tools to enable you to build packages later:
[root@myhost ~]# yum groupinstall "Development Tools" -y
[root@myhost ~]# yum install rpmbuild autoconf automake curl libev libev-devel pcre pcre-devel perl udns-devel

IP Tables (firewall)


Since our VPS is on a public Internet, we are going to firewall it off from all the unpleasantness out there. Add rules to allow DNS, HTTP and HTTPS inbound from your home's public IP:
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p udp -m udp --dport 53 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 80 -j ACCEPT
[root@myhost ~]# iptables -A INPUT -s 86.144.x.y/32 -p tcp -m tcp --dport 443 -j ACCEPT
[root@myhost ~]# service iptables save
[root@myhost ~]# service iptables restart

BIND (DNS server)


BIND is going to be our DNS server. It will respond with our VPS IP to DNS requests for netflix.com domain. Note, we are configuring the DSN server to run in a chroot jail and as a non-root user, to provide added security and minimise the potential effects of a security compromise.

Install BIND:
[root@myhost ~]# yum install bind -y

Edit your /etc/named.conf and make it look as follows:
options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { any; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

        allow-query { trusted; };
        allow-recursion { trusted; };
        allow-transfer  { none; };
        recursion yes;
        additional-from-auth yes;
        additional-from-cache yes;

        auth-nxdomain no;    # conform to RFC1035
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };
};

acl "trusted" {
        any;
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/etc/named/zones.override";

Edit your /etc/named/db.override and make it look as follows:
$TTL  86400

@   IN  SOA ns1 root (
            YYYYMMDD01  ; serial
            604800      ; refresh 1w
            86400       ; retry 1d
            2419200     ; expiry 4w
            86400       ; minimum TTL 1d
            )

    IN  NS  ns1
ns1 IN  A   162.243.x.y
@   IN  A   162.243.x.y
*   IN  A   162.243.x.y

* Note, change the above IP address to your the public IP of your VPS and update the datestamp. 

Edit your /etc/named/zones.override and make it look as follows:
zone "netflix.com." {
    type master;
    file "/etc/named/db.override";
};

Note, you may need to change the ownership of the new files you created from root to named:
[root@myhost ~]# chown named:named /etc/named/zones.override
[root@myhost ~]# chown named:named /etc/named/db.override

Install BIND (chroot package):
[root@myhost ~]# yum install bind-chroot -y

Finally, start BIND:
[root@myhost ~]# rndc-confgen -a -r /dev/urandom -t /var/named/chroot
[root@myhost ~]# chkconfig named on
[root@myhost ~]# service named start

From one of your home machines, run the following quick test:
$ nslookup www.netflix.com 162.243.x.y
Server:         162.243.x.y
Address:        162.243.x.y#53

Name:   www.netflix.com
Address: 162.243.x.y

You should get the IP address of your VPS in the DNS response.

SNI Proxy (HTTP/SSL proxy)


SNI Proxy is going to be our HTTP/SSL proxy, which supports for TLS SNI extensions.

Build and install SNI Proxy:
[root@myhost ~]# cd /opt
[root@myhost ~]# git clone https://github.com/dlundquist/sniproxy.git
[root@myhost ~]# cd ./sniproxy
[root@myhost ~]# ./autogen.sh && ./configure && make dist
[root@myhost ~]# rpmbuild --define "_sourcedir `pwd`" -ba --nodeps redhat/sniproxy.spec
[root@myhost ~]# yum install $(ls /root/rpmbuild/RPMS/x86_64/sniproxy-[0-9]*.rpm) -y

Note: if the RPM fails to install, check the exact name of the file in the /root/rpmbuild/RPMS/x86_64/directory.

Also, please review the ./configure command output to make sure LIBUDNS or libudns is detected, if it hasn't, you will not be able to get SNI Proxy to work with the below configuration file. On a Red Hat/CentOS server, the library should be available from EPEL repository (yum install udns-devel), however for other distributions, you will need to build and install it manually.

Edit your sniproxy.conf and make it look as follows:
[root@myhost ~]# vim /etc/sniproxy.conf

user daemon
pidfile /var/tmp/sniproxy.pid

error_log {
        filename /var/log/sniproxy_error.log
        priority notice
}

listener 162.243.x.y 80 {
        proto http
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

listener 162.243.x.y 443 {
        proto tls
        access_log {
                filename /var/log/sniproxy_access.log
        }
}

table {
        netflix\.com *
}

Finally, install the start-up script and start SNI Proxy:
[root@myhost ~]# cp ./redhat/sniproxy.init /etc/init.d/sniproxy
[root@myhost ~]# chmod +x /etc/init.d/sniproxy
[root@myhost ~]# chkconfig sniproxy on
[root@myhost ~]# service sniproxy start

Note, if the proxy complains about ports being in use, check if you got another web server already running and if so, disable it or move it to a different port.

From one of your home machines, run the following quick test:
$ telnet 162.243.x.y 80
Trying 162.243.x.y...
Connected to 162.243.x.y.
Escape character is '^]'.
^C
Connection closed by foreign host.

$ openssl s_client -servername www.netflix.com -connect 162.243.x.y:443
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=CALIFORNIA/L=Los Gatos/O=Netflix, Inc./OU=Operations/CN=www.netflix.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 1999 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G3
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MII...

You should get a socket connection on port 80 and get an SSL handshake with Netflix.

Putting It All Together


The last step is to change the DNS server on the device you are planning to watch US Netflix on. Set it to your VPS public IP (i.e. 162.243.x.y) Fire up Netflix and you should see the US catalogue and if you do - enjoy!

Note, this won't work on devices which don't support the use of SNI during SSL handshake, you'll have no choice but to use a system described in this guide or use black.box unzoner.

References


I've used the following reference material to prepare the solution described in this article. Many thanks to the respective authors.