Admins eHow SysAdmin Tips & Tricks

In this post I am going to show you how to setup a system to download YouTube videos on a raspberry pi on a certain time of the day !
I know this is a weird case of usage, but if your internet speed is low and cant watch YouTube videos directly or your daily internet traffic is limited, it may be useful. it can download your favorite YouTube videos for you when you are sleep !
This is actually not a simple system and I am not going through all of the details because the post will become very long and I am lazy :p. I will provide the information which you can not find anywhere else, other steps can be found on other websites.
So here is our design :

Youtube -> Chrome extension -> API (PHP file) on raspberry pi -> a file (/etc/youtube) containing YouTube links
Cronjob 1 -> Process /etc/youtube -> Get download links -> Aria2 (paused mode)
Cronjob 2 -> Start Aria2

Lets start with chrome extension, it is a very simple extension and consists of 2 files.
You can download the extension source by this link : youtube-chrome-ext download
Unzip this file and open sample.js
On line 15 you will see this :

client.get("http://192.168.101.1/ydl.php?url=" + info.linkUrl, function(response) {});

Change 192.168.101.1 to your own raspberry pi IP address.
Now open chrome extensions page chrome://extensions/ and enable developer mode. “Load unpacked extension” button will appear, click on it and browse to extension folder and select it. it will install the extension inside chrome.
Now if you click on any link inside chrome, you would see a new option called “YouTube Downloader”, clicking on it will send the link to our raspberry pi API which we will implement in next step.

Now lets create our PHP API file, needless to say you need to have a web server and PHP installed on your raspberry pi.
Create a file named ydl.php in /var/www/html folder with the following source :

<?php
header('Access-Control-Allow-Origin: *');
$url=$_GET["url"]."\n";
$file = '/etc/youtube';
$current = file_get_contents($file);
$current .= $url;
file_put_contents($file, $current);
?>

As you can see this is a very simple API. it appends the YouTube links which are sent by our chrome extension to a file named /etc/youtube.
As this file does not exist at the first time, lets create it and give it proper permissions. run following commands on raspberry pi :

touch /etc/youtube
chmod 666 /etc/youtube

Now it is time to test our API, open YouTube website, right click on several videos and choose “YouTube Downloader” then check the contents of /etc/youtube on raspberry pi, the links should be there.

Next step is to create the scripts which process /etc/youtube file and send the download links to Aria2.
Create the following files with their respective sources :
/usr/bin/process_youtube :

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        /usr/bin/a2youtube.py $line
done < /etc/youtube

rm /etc/youtube.old
mv /etc/youtube /etc/youtube.old
touch /etc/youtube
chmod 666 /etc/youtube

/usr/bin/a2youtube.py :

#!/usr/bin/python
import xmlrpclib,sys,commands
out=commands.getoutput("/usr/local/bin/youtube-dl -f 'best' -g -e --get-id "+sys.argv[1])
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
if (len(out.splitlines()[0].strip())<10):
        fn=out.splitlines()[1]
else:
        fn=out.splitlines()[0].strip()

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

set proper permissions for both files :

chmod 755 /usr/bin/process_youtube
chmod 755 /usr/bin/a2youtube.py

As you can see we will be using python for second script. so you need to have python installed as well.
There is also another program which is responsible to get the download link for us named youtube-dl.
You should install the latest version from this link : https://rg3.github.io/youtube-dl/

The reason that I chose to use Aria2 is that it is a VERY good and flexible download manager, better than anything else that you can find on Windows or Mac hands down so I highly recommend it. You need to install Aria2 as well : https://aria2.github.io/
here is my aria2 config file :

dir=/media
file-allocation=falloc
continue=true
log-level=notice
check-certificate=false
max-connection-per-server=16
split=16
summary-interval=120
daemon=true
enable-rpc=true
enable-dht=true
max-concurrent-downloads=2
http-auth-challenge=true
log=/var/log/aria2/aria2.log
disable-ipv6=true
disk-cache=25M
timeout=600
retry-wait=30
max-tries=50
save-session=/home/pi/session.gz
input-file=/home/pi/session.gz
seed-time=0
min-split-size=1M
rpc-secret=XXXX
rpc-listen-port=6800
rpc-listen-all=true

Pay attention to last 3 lines of config specially rpc-secret. it is a token that other programs will use to communicate with aria2 daemon. so change XXXX to a password of your choosing. also notice the “token:XXXX” in the /usr/bin/a2youtube.py file. change XXXX to the password that you set in aria2 config file.
You can (should) also install a web user interface for Aria2 from this link : https://github.com/ziahamza/webui-aria2
The webui will act as GUI for aria2 in your web browser so you can see what it is doing and control it as u wish.

If you pay attention to the python code you would see that it adds the links in paused mode :

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

The reason is that if we start to download immediately, youtube-dl may fail to get other links from YouTube website because your download bandwidth is full (thats the point).
so we need 2 more scripts to start/stop Aria2 :
/usr/bin/a2stop.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.pauseAll("token:XXXX")

/usr/bin/a2start.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.unpauseAll("token:XXXX")

dont forget to set proper permissions :

chmod 755 /usr/bin/a2start.py
chmod 755 /usr/bin/a2stop.py

and change XXXX to the secret that you set in aria2 config file.
now you can create the cronjobs :

2 3 * * * /usr/bin/process_youtube
5 3 * * * /usr/bin/a2start.py
55 8 * * * /usr/bin/a2stop.py

It will add YouTube links to aria2 @ 3:02 AM
Starts all downloads in aria2 @ 3:05 AM
Pauses all downloads in aria2 @ 8:55 AM

Good luck on implementing this system, it is not easy. But you will learn a lot if you try and you are persistent.

https://github.com/cxcv/iops

Very well deserve it #Apple 😀 #GETREKT

Bend to those who are worthy 😀 #bendgate #bendghazi #iPhone6 pic.twitter.com/CTWMzT5Wxf

— FiFtHeLeMeNt (@FiFtHeLeMeNt) September 25, 2014

#Apple released certified #iPant for #iPhone6 users. #BendGate #bendghazi pic.twitter.com/c1UqGihSrk

— FiFtHeLeMeNt (@FiFtHeLeMeNt) September 25, 2014

(more…)

DDOS attacks are one of hardest types of network attacks to encounter and stop. Usually the attacker uses many different IPs to request legitimate resources from your network to the point of exhaustion of your system resources and takes it down.
If you can somehow filter the IP addresses of the attacker on your system, then it is possible to block them in iptables easily and stop the attack.
In my case the attacker was attacking a website hosted on a dedicated IP address, so I was easily able to filter the attacker IP addresses by following command :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq

a.b.c.d : IP address of my server which the victim website was hosted on
You may do all kinds of filtering using grep and awk.
After I identified attacker IP addresses, blocking them was easy. first create a file named block and put it in /usr/bin with following contents :

#!/bin/bash
iptables -I INPUT -s $1/32 -j DROP

make it executable :

chmod +x /usr/bin/block

then run the following command :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq | xargs -n1 block

It will automatically block all attacker IPs in server firewall.
You may run the command every 5-10 minutes until the attack stops completely.
The problem of this approach is that you may end up blocking some legitimate users mixed with attacker IPs, but it is still better than having your whole server down indefinitely.
Also after the attack stops, you can remove all firewall rules or simply reboot your server and everything will be good 🙂

Edit :
In fact you can turn this into a real one liner without creating block file :D, here it is :

netstat -n | grep a.b.c.d | awk '{print $5}' | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq | xargs -n1 -I {} iptables -I INPUT -s {}/32 -j DROP

Here is a simple guide on how to install rtorrent/rutorrent on Debian 7 Wheezy , It may also work on Ubuntu and other Debian based Linux distros.

Install prerequisite packages :

apt-get install gcc pkg-config libssl-dev g++ make libncurses5-dev libsigc++-2.0-dev libcurl4-openssl-dev subversion screen nano

Install XMLRPC-C , it is required for rutorrent communication with rtorrent :

svn co http://svn.code.sf.net/p/xmlrpc-c/code/advanced xmlrpc-c
cd xmlrpc-c
./configure
make
make install

Install LibTorrent required by rtorrent :

wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.13.3.tar.gz
tar zxvf libtorrent-0.13.3.tar.gz
cd libtorrent-0.13.3
./configure
make
make install

Install rtorrent client :

wget http://libtorrent.rakshasa.no/downloads/rtorrent-0.9.3.tar.gz
tar zxvf rtorrent-0.9.3.tar.gz
cd rtorrent-0.9.3
./configure --with-xmlrpc-c
make
make install
ldconfig

Now, we have to make a user for rtorrent and configure it :

useradd user1
mkdir -p /home/user1/rtorrent
mkdir -p /home/user1/rtorrent/.session
mkdir -p /home/user1/rtorrent/download
chown -R user1:user1 /home/user1

Copy rtorrent sample config from rtorrent source directory to user1 home directory :

cp rtorrent-0.9.3/doc/rtorrent.rc /home/user1/.rtorrent.rc

Now you can customize the configuration :

nano /home/user1/.rtorrent.rc

But what you need to customize are following options :

directory = /home/user1/rtorrent/download
session = /home/user1/rtorrent/.session
scgi_port = localhost:5000

It is time to run rtorrent, This command runs rtorrent as user1 :

su - user1 -c 'screen -fa -d -m rtorrent'

Now we can install Apache + php5 which is required by rutorrent :

apt-get install libapache2-mod-php5

Enable auth_digest module which is required for rutorrent authentication :

a2enmod auth_digest

Install rutorrent+pluins :

wget http://dl.bintray.com/novik65/generic/rutorrent-3.6.tar.gz
tar zxvf rutorrent-3.6.tar.gz
mv rutorrent /var/www

wget http://dl.bintray.com/novik65/generic/plugins-3.6.tar.gz
tar zxvf plugins-3.6.tar.gz
mv plugins /var/www/rutorrent/

Tip : The only plugin which you need is httprpc. you can disable or delete all the rest.

Configure user1 on rutorrent :

mkdir -p /var/www/rutorrent/conf/users/user1
cp /var/www/rutorrent/conf/config.php /var/www/rutorrent/conf/users/user1
nano /var/www/rutorrent/conf/users/user1/config.php

Make sure $scgi_port in config.php matches scgi_port in rtorrent config file :

$scgi_port = 5000;

For rutorrent web authentication create .htaccess file in rutorrent directory :

nano /var/www/rutorrent/.htaccess

Copy and paste the following inside .htaccess :

AuthName "Restricted Area"
AuthType Basic
AuthUserFile /etc/.htpasswd
AuthGroupFile /dev/null
require valid-user

Create password file for Apache :

htdigest -c /etc/.htpasswd "Restricted Area" user1

Now we need to configure Apache to allow .htaccess override :

nano /etc/apache2/sites-enabled/000-default

Change :

<Directory /var/www/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
</Directory>

To :

<Directory /var/www/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    allow from all
</Directory>

And finally restart apache :

/etc/init.d/apache2 restart

Now you should be able to access your rtorrent/rutorrent on this address : http://IP_SERVER/rutorrent

Installing transmission 2.71 on Debian 6 is a little tricky and is not as straight forward as installing by apt-get.
Here is how you can do it.

First you need to manually install libevent 2.0 :

apt-get install make gcc
wget https://github.com/downloads/libevent/libevent/libevent-2.0.20-stable.tar.gz 
tar zxvf libevent-2.0.20-stable.tar.gz 
cd libevent-2.0.20-stable
./configure
make
make install

then some prerequisite packages for later installation :

apt-get install libssl-dev pkg-config libcurl4-openssl-dev intltool tar bzip2

now it is time to download and configure transmission :

wget http://download.transmissionbt.com/files/transmission-2.71.tar.bz2
tar jxvf transmission-2.71.tar.bz2
cd transmission-2.71
./configure --enable-lightweight --enable-daemon

now if you try to compile (make) the package you will get the following error :

tr-utp.c:65: error: conflicting types for ‘UTP_Write’
../third-party/libutp/utp.h:116: note: previous declaration of ‘UTP_Write’ was here
make[1]: *** [tr-utp.o] Error 1
make[1]: Leaving directory `/root/transmission-2.71/libtransmission’
make: *** [all-recursive] Error 1

in order to fix this error open this file in transmission directory :

nano ./third-party/libutp/utypes.h

and comment following lines :

// #ifndef __cplusplus
// typedef uint8 bool;
// #endif

and add following line just after above lines :

 
#ifndef __cplusplus
#ifdef HAVE_STDBOOL_H
#include <stdbool.h>
#else
typedef uint8 bool;
#endif
#endif

save file and exit. return to transmission root directory and make and install.

make
make install