Admins eHow SysAdmin Tips & Tricks

August 20, 2017

E2Streamer – Easily Stream from any Enigma2 based STB to your PC

Filed under: Windows — Tags: , , , , — admin @ 1:47 am

I have written a small C# program which makes it very easy to stream from any enigma2 based STB to your PC.
It retrieves the STB bouquets using the STB IP address and then you can stream the channels by double clicking on the channel names.
It uses VLC to stream the channels and VLC should be installed in the default location “C:\Program Files\VideoLAN\VLC\vlc.exe”.

Download link: E2Streamer

August 14, 2017

Download YouTube videos on raspberry pi on a certain time of the day using aria2

Filed under: General — admin @ 11:40 pm

In this post I am going to show you how to setup a system to download YouTube videos on a raspberry pi on a certain time of the day !
I know this is a weird case of usage, but if your internet speed is low and cant watch YouTube videos directly or your daily internet traffic is limited, it may be useful. it can download your favorite YouTube videos for you when you are sleep !
This is actually not a simple system and I am not going through all of the details because the post will become very long and I am lazy :p. I will provide the information which you can not find anywhere else, other steps can be found on other websites.
So here is our design :

Youtube -> Chrome extension -> API (PHP file) on raspberry pi -> a file (/etc/youtube) containing YouTube links
Cronjob 1 -> Process /etc/youtube -> Get download links -> Aria2 (paused mode)
Cronjob 2 -> Start Aria2

Lets start with chrome extension, it is a very simple extension and consists of 2 files.
You can download the extension source by this link : youtube-chrome-ext download
Unzip this file and open sample.js
On line 15 you will see this :

client.get("http://192.168.101.1/ydl.php?url=" + info.linkUrl, function(response) {});

Change 192.168.101.1 to your own raspberry pi IP address.
Now open chrome extensions page chrome://extensions/ and enable developer mode. “Load unpacked extension” button will appear, click on it and browse to extension folder and select it. it will install the extension inside chrome.
Now if you click on any link inside chrome, you would see a new option called “YouTube Downloader”, clicking on it will send the link to our raspberry pi API which we will implement in next step.

Now lets create our PHP API file, needless to say you need to have a web server and PHP installed on your raspberry pi.
Create a file named ydl.php in /var/www/html folder with the following source :

<?php
header('Access-Control-Allow-Origin: *');
$url=$_GET["url"]."\n";
$file = '/etc/youtube';
$current = file_get_contents($file);
$current .= $url;
file_put_contents($file, $current);
?>

As you can see this is a very simple API. it appends the YouTube links which are sent by our chrome extension to a file named /etc/youtube.
As this file does not exist at the first time, lets create it and give it proper permissions. run following commands on raspberry pi :

touch /etc/youtube
chmod 666 /etc/youtube

Now it is time to test our API, open YouTube website, right click on several videos and choose “YouTube Downloader” then check the contents of /etc/youtube on raspberry pi, the links should be there.

Next step is to create the scripts which process /etc/youtube file and send the download links to Aria2.
Create the following files with their respective sources :
/usr/bin/process_youtube :

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        /usr/bin/a2youtube.py $line
done < /etc/youtube

rm /etc/youtube.old
mv /etc/youtube /etc/youtube.old
touch /etc/youtube
chmod 666 /etc/youtube

/usr/bin/a2youtube.py :

#!/usr/bin/python
import xmlrpclib,sys,commands
out=commands.getoutput("/usr/local/bin/youtube-dl -f 'best' -g -e --get-id "+sys.argv[1])
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
if (len(out.splitlines()[0].strip())<10):
        fn=out.splitlines()[1]
else:
        fn=out.splitlines()[0].strip()

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

set proper permissions for both files :

chmod 755 /usr/bin/process_youtube
chmod 755 /usr/bin/a2youtube.py

As you can see we will be using python for second script. so you need to have python installed as well.
There is also another program which is responsible to get the download link for us named youtube-dl.
You should install the latest version from this link : https://rg3.github.io/youtube-dl/

The reason that I chose to use Aria2 is that it is a VERY good and flexible download manager, better than anything else that you can find on Windows or Mac hands down so I highly recommend it. You need to install Aria2 as well : https://aria2.github.io/
here is my aria2 config file :

dir=/media
file-allocation=falloc
continue=true
log-level=notice
check-certificate=false
max-connection-per-server=16
split=16
summary-interval=120
daemon=true
enable-rpc=true
enable-dht=true
max-concurrent-downloads=2
http-auth-challenge=true
log=/var/log/aria2/aria2.log
disable-ipv6=true
disk-cache=25M
timeout=600
retry-wait=30
max-tries=50
save-session=/home/pi/session.gz
input-file=/home/pi/session.gz
seed-time=0
min-split-size=1M
rpc-secret=XXXX
rpc-listen-port=6800
rpc-listen-all=true

Pay attention to last 3 lines of config specially rpc-secret. it is a token that other programs will use to communicate with aria2 daemon. so change XXXX to a password of your choosing. also notice the “token:XXXX” in the /usr/bin/a2youtube.py file. change XXXX to the password that you set in aria2 config file.
You can (should) also install a web user interface for Aria2 from this link : https://github.com/ziahamza/webui-aria2
The webui will act as GUI for aria2 in your web browser so you can see what it is doing and control it as u wish.

If you pay attention to the python code you would see that it adds the links in paused mode :

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

The reason is that if we start to download immediately, youtube-dl may fail to get other links from YouTube website because your download bandwidth is full (thats the point).
so we need 2 more scripts to start/stop Aria2 :
/usr/bin/a2stop.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.pauseAll("token:XXXX")

/usr/bin/a2start.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.unpauseAll("token:XXXX")

dont forget to set proper permissions :

chmod 755 /usr/bin/a2start.py
chmod 755 /usr/bin/a2stop.py

and change XXXX to the secret that you set in aria2 config file.
now you can create the cronjobs :

2 3 * * * /usr/bin/process_youtube
5 3 * * * /usr/bin/a2start.py
55 8 * * * /usr/bin/a2stop.py

It will add YouTube links to aria2 @ 3:02 AM
Starts all downloads in aria2 @ 3:05 AM
Pauses all downloads in aria2 @ 8:55 AM

Good luck on implementing this system, it is not easy. But you will learn a lot if you try and you are persistent.

April 26, 2017

Email to Telegram gateway

Filed under: PHP — Tags: , , , — admin @ 7:21 am

1.Create your own Telegram bot based on this tutorial :
https://www.domoticz.com/wiki/Telegram_Bot
or this one :
https://www.forsomedefinition.com/automation/creating-telegram-bot-notifications/
2.Create an email forwarder in cPanel and pipe it into a PHP script. make sure the script is executable (755 permission).
3.Here is the php script that will forward the email to your telegram bot :
Dont forget to adjust $url variable in telegram function based on first step.

#!/usr/local/bin/php -q
<?php
function mailRead($iKlimit = "")
{
	if ($iKlimit == "") {
		$iKlimit = 1024;
	}
	$sErrorSTDINFail = "Error - failed to read mail from STDIN!";
	$fp = fopen("php://stdin", "r");
	if (!$fp) {
		echo $sErrorSTDINFail;
		exit();
	}
	$sEmail = "";
	if ($iKlimit == - 1) {
		while (!feof($fp)) {
			$sEmail .= fread($fp, 1024);
		}
	}
	else {
		while (!feof($fp) && $i_limit < $iKlimit) {
			$sEmail .= fread($fp, 1024);
			$i_limit++;
		}
	}
	fclose($fp);
	return $sEmail;
}

function telegram($m)
{
	$url = 'https://api.telegram.org/botxxxxxxx:xxxxxxxxx/sendMessage?chat_id=xxxxx&text=';
	$url .= urlencode($m);
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	$response = curl_exec($ch);
	curl_close($ch);
}

$mail = mailRead(4096);
$lines = explode("\n", $mail);

$from = "";
$subject = "";
$headers = "";
$message = "";
$splittingheaders = true;

for ($i = 0; $i < count($lines); $i++) {
	if ($splittingheaders) {
		$headers .= $lines[$i] . "\n";
		if (preg_match("/^Subject: (.*)/", $lines[$i], $matches)) {
			$subject = $matches[1];
		}
		if (preg_match("/^From: (.*)/", $lines[$i], $matches)) {
			$from = $matches[1];
		}
		if (preg_match("/^To: (.*)/", $lines[$i], $matches)) {
			$to = $matches[1];
		}
	}
	else {
		$message .= $lines[$i] . "\n";
	}
	if (trim($lines[$i]) == "") {
		$splittingheaders = false;
	}
}

telegram("From: $from\nSubject: $subject\nMessage: $message");
?>

April 5, 2017

How to monitor bandwidth/traffic usage on a router

Filed under: Networking — Tags: , , , , , — admin @ 4:32 pm

Usually users dont care about their Internet traffic usage, especially on broadband lines as ISP’s provide unmetered bandwidth.
Unfortunately it is not the case for everyone, I have to pay for every single gigabyte of traffic that I use to my ISP. Can you believe that !?
Recently I noticed my ISP usage statistics are a tad high, so I decided to log my usage myself and compare it to my ISP stats.
I have a Mikrotik router which is somewhat advanced, so I expected an easy straight forward solution to measure my daily traffic, but to my surprise I was wrong.
Most solutions available are inaccurate -like query speed of interface every 5 minutes and calculate an estimate- or hard to implement and overkill.
So I came up with a novel idea. Here is the principle:
Most routers keep inbound & outbound bandwidth counters of their network interfaces. If we can query these numbers through SNMP on predefined intervals, we can calculate interface traffic precisely.
(more…)

AdminseHow BitcoinUnlimited and ElectrumX servers online !

Filed under: General — Tags: , , — admin @ 11:31 am

It’s been quite a while that I’ve been running a bitcoin unlimited node and an electrumx server as a contribution to bitcoin network.
I bought all of my bitcoins in 2010-2011 for $10 each and have already made a quite nice 120.000% profit on my investment so a contribution was due.
Both servers provide auto discovery and as a client, you usually don’t need to configure your server or peers manually but technically you can!
If you needed a reliable and fast bitcoin node peer or electrum server, feel free use the following :

Bitcoin Unlimited node:

electrumx.adminsehow.com:8333

Electrum TCP:

electrumx.adminsehow.com:50001

Electrum SSL:

electrumx.adminsehow.com:50002

February 13, 2017

Excellent window manager for Windows/Mac OS X

Filed under: MAC OS X,Windows — Tags: , , , — admin @ 6:33 am

About 1 year ago I bought a 34″ LG ultra wide monitor (21:9). Ultra wide monitors are usually used as a replacement of multiple monitors and thus using a window manager is mandatory with them.
For the past year I was using LG’s crappy “Split Screen” software which comes with monitor until recently I found this great window manager called Divvy. It saved me from all issues I had with “Split Screen” and provides more features and flexibility, so I thought it is necessary to give them a shout-out.
If you are in need of a good window manager don’t hesitate to use Divvy and possibly support them by buying its license. It is worth every penny.
Here is the link :
DIVVY – WINDOW MANAGEMENT AT ITS FINEST

February 4, 2017

Excellent tool for benchmarking disk iops in Linux

Filed under: General — admin @ 1:02 am

https://github.com/cxcv/iops

May 6, 2016

Intel XHCI USB 3.0 drivers for Windows 10

Filed under: Windows — Tags: , , , — admin @ 11:07 pm

Intel has not released official XHCI USB 3.0 drivers for Windows 10 and the one that comes with Windows 10 by default is a generic one provided by MS and buggy. Here you can check intel website : https://downloadcenter.intel.com/product/65855/Intel-USB-3-0-eXtensible-Host-Controller-Driver

Fortunately some people have managed to mod the driver provided by Intel for Windows 7 to Windows 10 : http://www.win-raid.com/t834f25-USB-Drivers-original-and-modded.html

May 5, 2016

Convert whole directory video/audio files using ffmpeg in Windows

Filed under: Windows — Tags: , , , — admin @ 9:05 pm

The following command converts all AVI files in a directory to H264 480p using ffmpeg :

FOR /F "tokens=*" %G IN ('dir /b *.avi') DO ffmpeg -i "%G" -codec:v libx264 -preset slow -b:v 500k -maxrate 500k -bufsize 1000k -vf scale=-1:480 -threads 0 -codec:a aac -b:a 128k "%~nG.mp4"

January 20, 2016

Transparent SSL Socks Proxy using Raspberry pi

Filed under: Debian,Raspberry pi,Security — Tags: , , , — admin @ 8:58 am

A Transparent SSL Socks proxy can be useful to encrypt and secure all TCP connections and/or infiltrate Internet censorship systems.
In order to make this setup, you need two Linux based boxes, one in your local network and one which will act as server in a remote location.
Theoretically what we are going to achieve is to intercept all TCP connections on our local network transparently, encrypt them and then tunnel them to our remote server.
This can be achieved easily using a powerful Linux application called Stunnel.
In my own setup, I am using a raspberry pi 2 for my local device. it is a cheap device and has a very low power usage and can be running 24/7, so it is very suitable to act as a full featured Linux based router. although you can use a PC or a virtual machine to achieve the same.
Debian is my favorite Linux distro, so my guide will be based on Debian.
Enough introduction, lets get started.

Part I: Setting up SSL Socks Proxy
1.Install stunnel on both local and remote devices:
Download and install the latest version of stunnel from stunnel website: https://www.stunnel.org/downloads.html
You may need to compile it from source.

2.Create stunnel config on local device /etc/stunnel/stunnel.conf:

foreground = no
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log
#compression = zlib
syslog = no

[SOCKS Client Direct]
client = yes
PSKsecrets = /etc/stunnel/secrets.txt
accept = 0.0.0.0:LOCAL_PORT
connect = REMOTE_SERVER_IP:REMOTE_PORT
protocol = socks

LOCAL_PORT: The local port which stunnel will listen on.
REMOTE_SERVER_IP: The remote server IP.
REMOTE_PORT: The port which remote server will accept connections on.

Note I: I noticed raspbian prebuilt openssl package doesn’t support zlib compression, so in my case I had to recompile openssl with zlib support. If you have an openssl package with zlib support you can uncomment “compression = zlib” line.
Note II: If you need to see the log messages for debug purposes, you can set “foreground = yes” temporarily. “foreground = no” makes stunnel to run in daemon mode.

3.Create local PreShareKey secret file /etc/stunnel/secrets.txt:

USERNAME:PASSWORD_MORE_THAN_20_CHARS

pick your username and password accordingly.

4.Create stunnel config on remote server /etc/stunnel/stunnel.conf:

foreground = no
socket = l:TCP_NODELAY=1
#compression = zlib

[SOCKS Server]
PSKsecrets = /etc/stunnel/secrets.txt
accept = 0.0.0.0:REMOTE_PORT
protocol = socks

REMOTE_PORT: The port which remote server will accept connections on.
The notes which I mentioned in previous section also apply to this section.

5.Create /etc/stunnel/secrets.txt on remote server identical to secrets file on local device.

6.Run both stunnel instances on local device and remote server.

stunnel /etc/stunnel/stunnel.conf

Part II: Making the SSL Socks Proxy Transparent
In order to make the SSL Socks Proxy Transparent, we need to setup our local device as the router and gateway of our local network and intercept all TCP connections.

The following settings should be made on local device.
1.Enable IPv4 forwarding in /etc/sysctl.conf:

net.ipv4.ip_forward=1

and apply the changes:

sysctl -p

2.Redirect all TCP connections to socks proxy:

iptables -t nat -A PREROUTING -p tcp -d LOCAL_IP_ADDRESS_CLASS -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -j REDIRECT --to-ports LOCAL_PORT

iptables -t nat -A OUTPUT -p tcp -d REMOTE_SERVER_IP --dport REMOTE_PORT -j ACCEPT
iptables -t nat -A OUTPUT -o lo -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -j REDIRECT --to-ports LOCAL_PORT

LOCAL_IP_ADDRESS_CLASS: It is very important to set this option correctly otherwise you will lose your network access to your local device. it should be the network address of your local network for example : 192.168.0.0/16 or 10.0.0.0/8 or 192.168.1.0/24
LOCAL_PORT: The local port which stunnel is listening on.
REMOTE_SERVER_IP: The remote server IP.
REMOTE_PORT: The port which remote server will accept connections on.

The PREROUTING rules redirects all TCP connections of other clients on the LAN to Socks Proxy and The OUTPUT rules redirects all TCP connections of local device to Socks Proxy.

3.Set the gateway of network devices on local network to local device IP address. you can configure it manually or configure your DHCP server to assign the new gateway to DHCP clients. if you have an advanced router, you can probably set it up in your router configuration, otherwise you may need to disable your router DHCP server and install a full featured DHCP server on your local device.

Part III: Setting up DHCP server (OPTIONAL)
1.Install ISC DHCP server:

apt-get install isc-dhcp-server

2.Create /etc/dhcp/dhcpd.conf:

ddns-update-style none;
option domain-name-servers 192.168.1.2;
authoritative;
log-facility local7;
subnet 192.168.1.0 netmask 255.255.255.0 {
        range 192.168.1.50 192.168.1.100;
        option routers 192.168.1.2;
        option broadcast-address 192.168.1.255;
        default-lease-time 86400;
        max-lease-time 172800;
}

It is the exact configuration of my DHCP server, I think it is self explanatory.
192.168.1.2 is my local device (raspberry pi), you need to change it to your local device IP address and also change ranges and broadcast address accordingly.
If you are wondering why I am using my local device as my DNS server, you need to read the next section. you may want to use google public DNS servers 8.8.8.8 and 8.8.4.4.

Part IV: Forward DNS queries on a different port than 53(OPTIONAL)
One of common DNS attacks is called DNS Hijacking.
It is a very easy attack and is performed by hijacking your DNS requests which are sent on port 53 UDP, then the hijacker can send you any reply that he wants and you can not verify if it is a legitimate response or not. Unfortunately in most operating systems you can not configure DNS client to use a different port which is not hijacked or it is very hard to do.
I have invented a very simple, yet brilliant solution for this problem. it can be performed by doing two DNAT operations on our local device and remote server to reach a safe DNS server.
This solution does not encrypt or secure the request, but it changes the DNS port transparently to a non-hijacked port, which works fine 🙂
Here is how it is done:

1.Run the following commands on your local device:

iptables -t nat -A PREROUTING -p udp --dport 53 -d LOCAL_DEVICE_IP -j DNAT --to REMOTE_SERVER_IP:5353
iptables -t nat -A POSTROUTING -p udp --dport 5353 -d REMOTE_SERVER_IP -j SNAT --to LOCAL_DEVICE_IP

Change LOCAL_DEVICE_IP & REMOTE_SERVER_IP accordingly.

Edit: Although the above solution works, I found out it is better to install a caching DNS server on local device rather than forwarding every DNS queries to remote server.

1.Install pdnsd on local device:

apt-get install pdnsd

2.Make following changes to /etc/pdnsd.conf:
in global section:

server_ip = 0.0.0.0

in server section:

ip = REMOTE_SERVER_IP;
port = 5353;

3.Set “START_DAEMON=yes” in /etc/default/pdnsd

4.Restart pdnsd:

service pdnsd restart

5.Run the following commands on your remote server:

iptables -t nat -A PREROUTING -p udp --dport 5353 -d REMOTE_SERVER_IP -j DNAT --to 209.244.0.3:53
iptables -t nat -A POSTROUTING -p udp --dport 53 -d 209.244.0.3 -j SNAT --to REMOTE_SERVER_IP

Change REMOTE_SERVER_IP accordingly.

209.244.0.3 is the IP address of Level3 public DNS server. I used it because it had a very good ping to my remote server. you can use a different DNS server.
If you do it properly, your local device can be used as a DNS server and in my case it would act as Level3 public DNS server. not susceptible to port 53 DNS hijacking.

Part V: Setting up an encrypted TCP based DNS caching server (Solution II for secure DNS – OPTIONAL)
In my experience I noticed forwarding DNS queries on port 5353 can be unreliable and cause problems for web browsing, although there is no technical reason for this problem and it could be just my ISP dropping such UDP packets. also my previous solution was not encrypted and was just used to bypass DNS hijackers listening for DNS packets on port 53.
I tried a new solution and it works much more reliably. I encrypted local device (raspberry pi) TCP connections using 3 new iptables OUTPUT rules (which is explained above) and then set up my DNS caching server to resolve queries only on TCP. using TCP is several times more slower than UDP but it will be encrypted and reliable.
Here is how it is done:

1.Install pdnsd on local device:

apt-get install pdnsd

2.Make following changes to /etc/pdnsd.conf:
in global section:

server_ip = 0.0.0.0
query_method = tcp_only;
min_ttl = 86400;

in server section:

ip = 209.244.0.3;

Note: The DNS server you choose should support resolving DNS queries on TCP port 53. 209.244.0.3 is Level3 public DNS server and supports TCP DNS queries.
Note: “min_ttl = 86400” overrides the default TTL of DNS queries to 24 hours which is OK for 99.999% of websites. But if a website changes its DNS records meanwhile it is in pdnsd cache, you may need to flush the cache manually or wait up to 24 hours for the record to be updated.

3.Set “START_DAEMON=yes” in /etc/default/pdnsd

4.Restart pdnsd:

service pdnsd restart
Older Posts »

Powered by WordPress