Admins eHow SysAdmin Tips & Tricks

August 14, 2017

Download YouTube videos on raspberry pi on a certain time of the day using aria2

Filed under: General — admin @ 11:40 pm

In this post I am going to show you how to setup a system to download YouTube videos on a raspberry pi on a certain time of the day !
I know this is a weird case of usage, but if your internet speed is low and cant watch YouTube videos directly or your daily internet traffic is limited, it may be useful. it can download your favorite YouTube videos for you when you are sleep !
This is actually not a simple system and I am not going through all of the details because the post will become very long and I am lazy :p. I will provide the information which you can not find anywhere else, other steps can be found on other websites.
So here is our design :

Youtube -> Chrome extension -> API (PHP file) on raspberry pi -> a file (/etc/youtube) containing YouTube links
Cronjob 1 -> Process /etc/youtube -> Get download links -> Aria2 (paused mode)
Cronjob 2 -> Start Aria2

Lets start with chrome extension, it is a very simple extension and consists of 2 files.
You can download the extension source by this link : youtube-chrome-ext download
Unzip this file and open sample.js
On line 15 you will see this :

client.get("http://192.168.101.1/ydl.php?url=" + info.linkUrl, function(response) {});

Change 192.168.101.1 to your own raspberry pi IP address.
Now open chrome extensions page chrome://extensions/ and enable developer mode. “Load unpacked extension” button will appear, click on it and browse to extension folder and select it. it will install the extension inside chrome.
Now if you click on any link inside chrome, you would see a new option called “YouTube Downloader”, clicking on it will send the link to our raspberry pi API which we will implement in next step.

Now lets create our PHP API file, needless to say you need to have a web server and PHP installed on your raspberry pi.
Create a file named ydl.php in /var/www/html folder with the following source :

<?php
header('Access-Control-Allow-Origin: *');
$url=$_GET["url"]."\n";
$file = '/etc/youtube';
$current = file_get_contents($file);
$current .= $url;
file_put_contents($file, $current);
?>

As you can see this is a very simple API. it appends the YouTube links which are sent by our chrome extension to a file named /etc/youtube.
As this file does not exist at the first time, lets create it and give it proper permissions. run following commands on raspberry pi :

touch /etc/youtube
chmod 666 /etc/youtube

Now it is time to test our API, open YouTube website, right click on several videos and choose “YouTube Downloader” then check the contents of /etc/youtube on raspberry pi, the links should be there.

Next step is to create the scripts which process /etc/youtube file and send the download links to Aria2.
Create the following files with their respective sources :
/usr/bin/process_youtube :

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        /usr/bin/a2youtube.py $line
done < /etc/youtube

rm /etc/youtube.old
mv /etc/youtube /etc/youtube.old
touch /etc/youtube
chmod 666 /etc/youtube

/usr/bin/a2youtube.py :

#!/usr/bin/python
import xmlrpclib,sys,commands
out=commands.getoutput("/usr/local/bin/youtube-dl -f 'best' -g -e --get-id "+sys.argv[1])
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
if (len(out.splitlines()[0].strip())<10):
        fn=out.splitlines()[1]
else:
        fn=out.splitlines()[0].strip()

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

set proper permissions for both files :

chmod 755 /usr/bin/process_youtube
chmod 755 /usr/bin/a2youtube.py

As you can see we will be using python for second script. so you need to have python installed as well.
There is also another program which is responsible to get the download link for us named youtube-dl.
You should install the latest version from this link : https://rg3.github.io/youtube-dl/

The reason that I chose to use Aria2 is that it is a VERY good and flexible download manager, better than anything else that you can find on Windows or Mac hands down so I highly recommend it. You need to install Aria2 as well : https://aria2.github.io/
here is my aria2 config file :

dir=/media
file-allocation=falloc
continue=true
log-level=notice
check-certificate=false
max-connection-per-server=16
split=16
summary-interval=120
daemon=true
enable-rpc=true
enable-dht=true
max-concurrent-downloads=2
http-auth-challenge=true
log=/var/log/aria2/aria2.log
disable-ipv6=true
disk-cache=25M
timeout=600
retry-wait=30
max-tries=50
save-session=/home/pi/session.gz
input-file=/home/pi/session.gz
seed-time=0
min-split-size=1M
rpc-secret=XXXX
rpc-listen-port=6800
rpc-listen-all=true

Pay attention to last 3 lines of config specially rpc-secret. it is a token that other programs will use to communicate with aria2 daemon. so change XXXX to a password of your choosing. also notice the “token:XXXX” in the /usr/bin/a2youtube.py file. change XXXX to the password that you set in aria2 config file.
You can (should) also install a web user interface for Aria2 from this link : https://github.com/ziahamza/webui-aria2
The webui will act as GUI for aria2 in your web browser so you can see what it is doing and control it as u wish.

If you pay attention to the python code you would see that it adds the links in paused mode :

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

The reason is that if we start to download immediately, youtube-dl may fail to get other links from YouTube website because your download bandwidth is full (thats the point).
so we need 2 more scripts to start/stop Aria2 :
/usr/bin/a2stop.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.pauseAll("token:XXXX")

/usr/bin/a2start.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.unpauseAll("token:XXXX")

dont forget to set proper permissions :

chmod 755 /usr/bin/a2start.py
chmod 755 /usr/bin/a2stop.py

and change XXXX to the secret that you set in aria2 config file.
now you can create the cronjobs :

2 3 * * * /usr/bin/process_youtube
5 3 * * * /usr/bin/a2start.py
55 8 * * * /usr/bin/a2stop.py

It will add YouTube links to aria2 @ 3:02 AM
Starts all downloads in aria2 @ 3:05 AM
Pauses all downloads in aria2 @ 8:55 AM

Good luck on implementing this system, it is not easy. But you will learn a lot if you try and you are persistent.

April 5, 2017

AdminseHow BitcoinUnlimited and ElectrumX servers online !

Filed under: General — Tags: , , — admin @ 11:31 am

It’s been quite a while that I’ve been running a bitcoin unlimited node and an electrumx server as a contribution to bitcoin network.
I bought all of my bitcoins in 2010-2011 for $10 each and have already made a quite nice 120.000% profit on my investment so a contribution was due.
Both servers provide auto discovery and as a client, you usually don’t need to configure your server or peers manually but technically you can!
If you needed a reliable and fast bitcoin node peer or electrum server, feel free use the following :

Bitcoin Unlimited node:

electrumx.adminsehow.com:8333

Electrum TCP:

electrumx.adminsehow.com:50001

Electrum SSL:

electrumx.adminsehow.com:50002

February 4, 2017

Excellent tool for benchmarking disk iops in Linux

Filed under: General — admin @ 1:02 am

https://github.com/cxcv/iops

March 22, 2015

Linux dig utility for Windows x64

Filed under: dns,General,linux,Windows — Tags: , , , — admin @ 5:00 pm

I have created an installer for Linux DNS dig utility for Windows x64. it is extracted from BIND 9.10.2.x64.
It installs dig into system32 folder of Windows so it is already included in PATH and can be invoked from anywhere in command prompt.

Download : DIG_9.10.2.x64

October 20, 2014

How to watch Twitch streams without lag or stutter in Source quality

Filed under: General — Tags: , , , , — admin @ 1:16 am

As a gamer, I am also a fan of watching other people playing games on Twitch. I mostly watch pro players playing Dota 2.
If you have ever tried watching Twitch, you would know how much it matters to be able to watch the streams in Source quality. Good streamers usually stream in Full HD (1920×1080 30 FPS or 60 FPS) which may require a consistent 4-6 mps of bandwidth. well you may think 4-6 mps is not much, your broadband connection supports multiple times of this number but in the reality is not that simple and you may get lag and stutter while watching in Source quality.
The reason behind this is the complexity of Internet. Twitch stream may not be routed to you through an optimal route, Also many broadband ISPs play all kind shenanigans with users traffic in order to save bandwidth and make more money.
Unfortunately for me thats the case. Normally I am not able to watch Twitch streams in Source quality and I hate anything less than Full HD.
But being a system administrator has its own privileges and I have found two ways to be able to watch Twitch streams in Source quality which I am going to share with you :

  1. Use a VPN or Proxy : Using a good VPN or Proxy may actually improve your Internet speed. The reason is that your ISP may not have the best routes to all other networks but may have good routes to few other networks, now if you can get a VPN or Proxy on one of those good networks, it causes all of your traffic to be routed through those good routes and it improves your Internet speed. Also using a VPN or Proxy will save you from shenanigans of your ISP because it is usually encrypted and they can not tamper with it.
    But there is a trick, using any VPN or Proxy will not help you and may even degrade your Internet speeds. You should use a VPN or Proxy which has a good route to you and its quality is decent so forget about free ones. What I recommend to you is to get a service from a reputable VPN provider – like StrongVPN – they provide many VPN servers in diffenet locations which enables you to find the one which works for you by trial and error. you can also speed test their servers and find out which one has the best route to you.
    Another advantage of using a VPN or Proxy is that you can use it in your mobile device. Sometimes I like to watch streams lying in bed on my tablet and without a VPN, I am not able to watch in source quality.
  2. Use Livestreamer : I found this solution just last night when even my beloved Proxy servers could not help me. Livestreamer is a software which allows you to watch online streams in external media players like VLC. it has many features like saving the stream and etc which I am not going to explain here and you can read the documentation if you are interested : http://livestreamer.readthedocs.org/en/latest/cli.html
    First thing which you need to do is to download and install Livestreamer from this link : http://livestreamer.readthedocs.org/en/latest/install.html
    Also if you dont already have VLC installed on your PC, go ahead and download and install it : http://www.videolan.org/
    Now we need to configure Livestreamer, find Livestreamer configuration file and open it in a text editor. It is located under “%APPDATA%\livestreamer\livestreamerrc” in Windows and under “~/.livestreamerrc” for MACOSX and Linux.
    First uncomment the appropriate player line in configuration file. For me it is :

    player="C:\Program Files\VideoLAN\VLC\vlc.exe" --file-caching=5000

    It specifies the location of vlc.exe so Livestreamer can launch it.
    Now scroll to the end of file and enable these two options with following values :

    hls-segment-threads=10
    hds-segment-threads=10

    These two lines will do the trick for us in order to be able to watch stream without stutter. The reason is that in HLS (Twitch Streaming Protocol) and HDS, unlike other streaming protocols which stream is sent in one continuous stream, it is sliced and chunked in several pieces and sent over HTTP protocol. So it is possible to download these chunks simultaneously with multiple connections exactly like how download managers work.
    Save the config file and our setup is complete. We can use it now to watch Twitch stream in source quality in VLC by following command :

    Livestreamer.exe LINK_TO_TWITCH_STREAM best

    For example :

    Livestreamer.exe http://www.twitch.tv/sing_sing best

    It takes about 15 seconds for Livestreamer to establish connections and launch VLC.

Update : I have written a small utility to comfortably launch livestreamer on Windows. It needs .Net framework 4.5 to work.
You can download it here : LiveStream Launcher

October 2, 2014

Minima wallpaper pack for Mobile / Tablets

Filed under: Android,Apple,General — Tags: , , — admin @ 3:50 am

Here are some beautiful minimalist wallpapers for your mobile device. Preview :

wallpaper-preview

Download : Minima-WP-Pack

September 25, 2014

The Internet’s hilarious reaction to #bendgate

Filed under: General — admin @ 9:34 am

Very well deserve it #Apple 😀 #GETREKT


(more…)

July 23, 2014

How to block ongoing DDOS attack on Linux Server

Filed under: General — admin @ 10:44 am

DDOS attacks are one of hardest types of network attacks to encounter and stop. Usually the attacker uses many different IPs to request legitimate resources from your network to the point of exhaustion of your system resources and takes it down.
If you can somehow filter the IP addresses of the attacker on your system, then it is possible to block them in iptables easily and stop the attack.
In my case the attacker was attacking a website hosted on a dedicated IP address, so I was easily able to filter the attacker IP addresses by following command :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq

a.b.c.d : IP address of my server which the victim website was hosted on
You may do all kinds of filtering using grep and awk.
After I identified attacker IP addresses, blocking them was easy. first create a file named block and put it in /usr/bin with following contents :

#!/bin/bash
iptables -I INPUT -s $1/32 -j DROP

make it executable :

chmod +x /usr/bin/block

then run the following command :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq | xargs -n1 block

It will automatically block all attacker IPs in server firewall.
You may run the command every 5-10 minutes until the attack stops completely.
The problem of this approach is that you may end up blocking some legitimate users mixed with attacker IPs, but it is still better than having your whole server down indefinitely.
Also after the attack stops, you can remove all firewall rules or simply reboot your server and everything will be good 🙂

Edit :
In fact you can turn this into a real one liner without creating block file :D, here it is :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq | xargs -n1 -I {} iptables -I INPUT -s {}/32 -j DROP

November 6, 2013

Installing rtorrent+rutorrent on Debian 7 Wheezy

Filed under: General — admin @ 4:00 am

Here is a simple guide on how to install rtorrent/rutorrent on Debian 7 Wheezy , It may also work on Ubuntu and other Debian based Linux distros.

Install prerequisite packages :

apt-get install gcc pkg-config libssl-dev g++ make libncurses5-dev libsigc++-2.0-dev libcurl4-openssl-dev subversion screen nano

Install XMLRPC-C , it is required for rutorrent communication with rtorrent :

svn co http://svn.code.sf.net/p/xmlrpc-c/code/advanced xmlrpc-c
cd xmlrpc-c
./configure
make
make install

Install LibTorrent required by rtorrent :

wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.13.3.tar.gz
tar zxvf libtorrent-0.13.3.tar.gz
cd libtorrent-0.13.3
./configure
make
make install

Install rtorrent client :

wget http://libtorrent.rakshasa.no/downloads/rtorrent-0.9.3.tar.gz
tar zxvf rtorrent-0.9.3.tar.gz
cd rtorrent-0.9.3
./configure --with-xmlrpc-c
make
make install
ldconfig

Now, we have to make a user for rtorrent and configure it :

useradd user1
mkdir -p /home/user1/rtorrent
mkdir -p /home/user1/rtorrent/.session
mkdir -p /home/user1/rtorrent/download
chown -R user1:user1 /home/user1

Copy rtorrent sample config from rtorrent source directory to user1 home directory :

cp rtorrent-0.9.3/doc/rtorrent.rc /home/user1/.rtorrent.rc

Now you can customize the configuration :

nano /home/user1/.rtorrent.rc

But what you need to customize are following options :

directory = /home/user1/rtorrent/download
session = /home/user1/rtorrent/.session
scgi_port = localhost:5000

It is time to run rtorrent, This command runs rtorrent as user1 :

su - user1 -c 'screen -fa -d -m rtorrent'

Now we can install Apache + php5 which is required by rutorrent :

apt-get install libapache2-mod-php5

Enable auth_digest module which is required for rutorrent authentication :

a2enmod auth_digest

Install rutorrent+pluins :

wget http://dl.bintray.com/novik65/generic/rutorrent-3.6.tar.gz
tar zxvf rutorrent-3.6.tar.gz
mv rutorrent /var/www

wget http://dl.bintray.com/novik65/generic/plugins-3.6.tar.gz
tar zxvf plugins-3.6.tar.gz
mv plugins /var/www/rutorrent/

Tip : The only plugin which you need is httprpc. you can disable or delete all the rest.

Configure user1 on rutorrent :

mkdir -p /var/www/rutorrent/conf/users/user1
cp /var/www/rutorrent/conf/config.php /var/www/rutorrent/conf/users/user1
nano /var/www/rutorrent/conf/users/user1/config.php

Make sure $scgi_port in config.php matches scgi_port in rtorrent config file :

$scgi_port = 5000;

For rutorrent web authentication create .htaccess file in rutorrent directory :

nano /var/www/rutorrent/.htaccess

Copy and paste the following inside .htaccess :

AuthName "Restricted Area"
AuthType Basic
AuthUserFile /etc/.htpasswd
AuthGroupFile /dev/null
require valid-user

Create password file for Apache :

htdigest -c /etc/.htpasswd "Restricted Area" user1

Now we need to configure Apache to allow .htaccess override :

nano /etc/apache2/sites-enabled/000-default

Change :

<Directory /var/www/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
</Directory>

To :

<Directory /var/www/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    allow from all
</Directory>

And finally restart apache :

/etc/init.d/apache2 restart

Now you should be able to access your rtorrent/rutorrent on this address : http://IP_SERVER/rutorrent

October 12, 2012

How to install transmission 2.71 on Debian 6.0 Squeeze

Filed under: General — admin @ 6:41 am

Installing transmission 2.71 on Debian 6 is a little tricky and is not as straight forward as installing by apt-get.
Here is how you can do it.

First you need to manually install libevent 2.0 :

apt-get install make gcc
wget https://github.com/downloads/libevent/libevent/libevent-2.0.20-stable.tar.gz 
tar zxvf libevent-2.0.20-stable.tar.gz 
cd libevent-2.0.20-stable
./configure
make
make install

then some prerequisite packages for later installation :

apt-get install libssl-dev pkg-config libcurl4-openssl-dev intltool tar bzip2

now it is time to download and configure transmission :

wget http://download.transmissionbt.com/files/transmission-2.71.tar.bz2
tar jxvf transmission-2.71.tar.bz2
cd transmission-2.71
./configure --enable-lightweight --enable-daemon

now if you try to compile (make) the package you will get the following error :

tr-utp.c:65: error: conflicting types for ‘UTP_Write’
../third-party/libutp/utp.h:116: note: previous declaration of ‘UTP_Write’ was here
make[1]: *** [tr-utp.o] Error 1
make[1]: Leaving directory `/root/transmission-2.71/libtransmission’
make: *** [all-recursive] Error 1

in order to fix this error open this file in transmission directory :

nano ./third-party/libutp/utypes.h

and comment following lines :

// #ifndef __cplusplus
// typedef uint8 bool;
// #endif

and add following line just after above lines :

 
#ifndef __cplusplus
#ifdef HAVE_STDBOOL_H
#include <stdbool.h>
#else
typedef uint8 bool;
#endif
#endif

save file and exit. return to transmission root directory and make and install.

make
make install
Older Posts »

Powered by WordPress