1000umbrellas.com

This is definitely a newb-level topic, but I think it is important to understand how to redirect output from your command-line processes. This becomes important because regularly-schedule processes run via cron will email you whatever the output is. This is completely confusing when you first encounter it, and the complete solution isn’t exactly obvious.

StdOut

When you run a process on the command line, it will probably output right back to the terminal. This is StdOut, or “standard output.” In scripts and cron jobs, however, you might want to redirect output to a file instead, so you can review the output later.

You can redirect output to a new file by using the “>” operator.

$ ls -l > ~/directory_listing.txt

You can redirect output and append it to a file by using the “>>” operator. This method will create a file if it does not already exist.

StdErr

Error messages are output to StdErr, which by default is redirected to StdOut. This is why you see normal messages and error messages in the terminal when you run a command. StdErr does not have to be redirected to StdOut, however, and has to be treated separately from StdOut if you are redirecting StdOut.

/dev/null and 2>&1

Redirecting output to /del/null sends it nowhere, which effectively gets rid of it. You often see output redirected to /dev/null, followed by a cryptic construction 2>&1. Here is what that construction means:

In the command
$ tvnamer --batch > ~/log/tvnamer.log 2>&1

• 2 refers to StdErr; and
• &1 refers to the first redirection argument; in this case, /log/tvnamer.log.

My example uses a normal file, instead of /dev/null, to show that you can use  2>&1 with any old file.

This construction is very useful if you find that your cron jobs are sending you emails, even after you redirected their output to log files. Cron will automatically email the owner of the cron job any output (on StdOut) from them. This is why you sometimes are notified that you have emails when you log into your system. Effective redirection will eliminate these emails.

The most popular post on my blog is how to set up the Ubuntu Server firewall via ufw, the Uncomplicated Firewall. I recommend UFW for any standard server, but you might want to move to something more advanced to expand your firewall knowledge, to set up a network gateway, router, or firewall, or to conform with other systems on your network. If UFW is for beginners, then Firehol is for intermediate sysadmins, or those wishing to run proxy servers and the like.

Before I started using ufw, I used to use Firehol to set up the iptables firewall. Firewall does the same thing that UFW does, and a lot more, but in a completely different way. That’s because Firehol is not just a script, but a language, for defining firewalls. You write your rules in a configuration file, and Firehol will convert them into iptables commands, and issue them at boot time or on demand. It can only create “drop/reject everything, allow explicitly” type firewalls, but luckily, that’s the most appropriate type of firewall to create most of the time.

Firehol is useful not only for standard file and web servers, but also for routers and gateways, and can set up rules for transparent proxies, NAT, and so on. Home users such as myself have a hardware router for that, and only need to set up firewalls on home servers as a defense-in-depth measure.

I think that Firehol is more flexible than ufw, can implement more advanced features such as transparent proxies better than ufw, but it is a little to a lot harder to set up. This tutorial covers how you would lock down a simple home server that you wish to access via the Internet, but not leave completely open to the outside world.

Installation

Install the firehol package.

$ sudo apt-get install firehol

Enabling Firehol to Start on Boot

Firehol must be enabled before the rules you define will go into effect. To enable it, you must edit a configuration file.

$ sudo nano /etc/default/firehol

The file will look like this after a clean install:

#To enable firehol at startup set START_FIREHOL=YES
START_FIREHOL=NO
#If you want to have firehol wait for an iface to be up add it here
WAIT_FOR_IFACE=""

Simply change the second line to

START_FIREHOL=YES

and save the file. That will enable the firewall upon reboot. What that means is that Firehol will run its bash scripts, generate iptables rules, and then the iptables will be activated. Firehol does not run continuously as a service.

If you wish to disable the firewall, be sure to edit this file again, and reset the START_FIREHOL variable to NO.

Configuring the Firewall Rules

The configuration file containing the firewall rules is located in /etc/firehol.conf. This is what the default configuration file looks like. It will set your server to block everything coming in, and accept everything going out.

#
# $Id: client-all.conf,v 1.2 2002/12/31 15:44:34 ktsaou Exp $
#
# This configuration file will allow all requests originating from the
# local machine to be send through all network interfaces.
#
# No requests are allowed to come from the network. The host will be
# completely stealthed! It will not respond to anything, and it will
# not be pingable, although it will be able to originate anything
# (even pings to other hosts).
#

version 5

# Accept all client traffic on any interface
interface any world
client all accept

You could edit this file and specify your own services, or you could auto-generate a configuration file. I will walk you through auto-generating a confguration file in the next section.

Bear in mind, though, that a fully-working firewall can be made with a configuration file as simple as this:

#!/sbin/firehol
version 5
interface any world
protection strong
server "ssh http samba webmin" allow
client all accept


If you are only running standard services, it’s best to edit the file manually. If you have custom services, or are curious about Firehol’s abilities, read on.

Auto-generating a Configuration File

Firehol can automatically generate a configuration file for you to get you started. The firehol helpme command identifies aany running services on your server and builds a configuration file that will allow all traffic between them, and block all traffic outside those parameters. After the file is generated, you are then expected to review and modify it, to ensure the rules are correct, prior to activating the firewall.

First, use the firehol helpme command to generate a configuration file and put it into the /tmp folder.

$ sudo firehol helpme > /tmp/firehol.conf

Then, review the automatically generated configuration file.

$ nano /tmp/firehol.conf

The automatically generated configuration file is pretty verbose. This is what mine looks like, for a server that runs SSH, Samba, Transmission, Dropbox, and Lighttpd web server (it could be Apache or any other web server, really).

Note that there are two interfaces set up: one for my LAN (10.0.0.0/24) and one for the Internet (i.e, anything outside my router’s subnet). When I reviewed it, I notice that the peer ports for my transmission-daemon service are not all mapped, just the one currently in use. Therefore, you definitely have to tweak this file to suit your needs.

#!/sbin/firehol
# : firehol.sh,v 1.273 2008/07/31 00:46:41 ktsaou Exp $
#
# This config will have the same effect as NO PROTECTION!
# Everything that found to be running, is allowed.
# YOU SHOULD NEVER USE THIS CONFIG AS-IS.
#
# Date: Fri May 14 10:43:37 EDT 2010 on host thor
#
# IMPORTANT:
# The TODOs bellow, are *YOUR* to-dos!
#

# INFO: Processing interface 'eth0'
# INFO: Processing IP 10.0.0.3 of interface 'eth0'
# INFO: Is 10.0.0.3 part of network 10.0.0.0/24? yes

# Interface No 1.
# The purpose of this interface is to control the traffic
# on the eth0 interface with IP 10.0.0.3 (net: "10.0.0.0/24").
# TODO: Change "interface1" to something with meaning to you.
# TODO: Check the optional rule parameters (src/dst).
# TODO: Remove 'dst 10.0.0.3' if this is dynamically assigned.
interface eth0 interface1 src "10.0.0.0/24" dst 10.0.0.3

# The default policy is DROP. You can be more polite with REJECT.
# Prefer to be polite on your own clients to prevent timeouts.
policy drop

# If you don't trust the clients behind eth0 (net "10.0.0.0/24"),
# add something like this.
# > protection strong

# Here are the services listening on eth0.
# TODO: Normally, you will have to remove those not needed.
client dhcp accept
server http accept
server ICMP accept
server ms_ds accept
server ntp accept
server samba accept
server ssh accept
server webmin accept

# The following eth0 services are not known by FireHOL:
# tcp/17500 tcp/20550 tcp/9091 udp/10000 udp/17500 udp/20550

# Custom service definitions for the above unknown services.
server custom if1_tcp_17500 tcp/17500 any accept
server custom if1_tcp_20550 tcp/20550 any accept
server custom if1_tcp_9091 tcp/9091 any accept
server custom if1_udp_10000 udp/10000 any accept
server custom if1_udp_17500 udp/17500 any accept
server custom if1_udp_20550 udp/20550 any accept

# The following means that this machine can REQUEST anything via eth0.
# TODO: On production servers, avoid this and allow only the
#       client services you really need.
client all accept

# INFO: Is 10.0.0.1  part of network 10.0.0.0/24? yes
# INFO: Default gateway 10.0.0.1  is part of network 10.0.0.0/24

# Interface No 2.
# The purpose of this interface is to control the traffic
# from/to unknown networks behind the default gateway 10.0.0.1 .
# TODO: Change "interface2" to something with meaning to you.
# TODO: Check the optional rule parameters (src/dst).
# TODO: Remove 'dst 10.0.0.3' if this is dynamically assigned.
interface eth0 interface2 src not "${UNROUTABLE_IPS} 10.0.0.0/24" dst 10.0.0.3

# The default policy is DROP. You can be more polite with REJECT.
# Prefer to be polite on your own clients to prevent timeouts.
policy drop

# If you don't trust the clients behind eth0 (net not "${UNROUTABLE_IPS} 10.0.0.0/24"),
# add something like this.
# > protection strong

# Here are the services listening on eth0.
# TODO: Normally, you will have to remove those not needed.
client dhcp accept
server http accept
server ICMP accept
server ms_ds accept
server ntp accept
server samba accept
server ssh accept
server webmin accept

# The following eth0 services are not known by FireHOL:
# tcp/17500 tcp/20550 tcp/9091 udp/10000 udp/17500 udp/20550

# Custom service definitions for the above unknown services.
server custom if2_tcp_17500 tcp/17500 any accept
server custom if2_tcp_20550 tcp/20550 any accept
server custom if2_tcp_9091 tcp/9091 any accept
server custom if2_udp_10000 udp/10000 any accept
server custom if2_udp_17500 udp/17500 any accept
server custom if2_udp_20550 udp/20550 any accept

# The following means that this machine can REQUEST anything via eth0.
# TODO: On production servers, avoid this and allow only the
#       client services you really need.
client all accept

# INFO: Processing interface 'wlan0'

# IMPORTANT:
# Ignoring interface 'wlan0' because does not have an IP or route.

# The above 2 interfaces were found active at this moment.
# Add more interfaces that can potentially be activated in the future.
# FireHOL will not complain if you setup a firewall on an interface that is
# not active when you activate the firewall.
# If you don't setup an interface, FireHOL will drop all traffic from or to
# this interface, if and when it becomes available.
# Also, if an interface name dynamically changes (i.e. ppp0 may become ppp1)
# you can use the plus (+) character to match all of them (i.e. ppp+).

# No router statements have been produced, because your server
# is not configured for forwarding traffic.

Customizing the Configuration File

The auto-generated configuration file lists “TODO” items for you to do as you review the file. You can rename the interfaces to something meaningful, such as “lan” and “internet”. You can remove any of the named or unnamed (“custom”) services that it includes rules for. A few more of the rules that Firehol generated for my server bear some explanation:

• UDP/17500 is used by the Dropbox service to identify other Dropbox clients on the LAN. This rule cannot be removed
• TCP and UDP/25500 is one of the peer ports I set up for transmission-daemon. This I actually have to change, because I have a port range open for those services.
• UDP/10000 can be removed, because webmin runs on TCP/10000.

Server vs. Client Rules

Firehol considers all incoming requests “server” requests and all outgoing requests “client” requests. You might see client accept all in a configuration file. That means that client requests from any service are allowed. If you only want to allow certain services to be allowed, you can specify them much like server services are specified.

client http allow

Setting up Custom Services

To clean up custom services—for Transmission or Dropbox in this case—you can either give them names and define them inline…

server custom transmission-daemon tcp/9091 any accept
server custom dropbox-lan-peers udp/17500 any accept
server custom transmission-peers “tcp/20550:20599 udp/20550:20599” any accept

…or define them at the top of the script as variables…

# Transmission front-end
server_txf_ports="tcp/9091"
client_txf_ports="default"

# Transmission peers
server_txp_ports="tcp/20500:20599 udp/20500:20599"
client_txp_ports="default"

# Dropbox-related LAN functions
server_dpbx_ports="udp/17500"
client_dpbx_ports="default"

…and then invoke them very simply:

# custom services
server txf accept
server txp accept
server dpbx accept

To specify a port range, use a colon instead of a dash to indicate “through.” To specify both tcp and udf ports, use a space-delimited list, enclosed in quotation marks.

Different Rules for LAN vs. Internet

If you run Samba for file sharing, you’ll want to allow your LAN users to use Samba file shares, and block Internet users from doing so. (It’s true that the Samba ports should also be blocked on your router/gateway, if you have one, but bear with me.) To do so, you’ll want to include rules to allow Samba on your LAN interface (interface1 in the automatically-generated file above), and not include those Samba-related rules in the Internet interface (interface2, above). You might wish to remove webmin and other services from the Internet-side, too. I do both in my configuration file, which is reproduced below.

My Configuration File

Here is my configuration file. It implements kinder rules for the LAN and stricter rules for the Internet.

#!/sbin/firehol
# -----------------------------------------------
# Set up custom services
# -----------------------------------------------

# Transmission front-end
server_txf_ports="tcp/9091"
client_txf_ports="default"

# Transmission peers
server_txp_ports="tcp/20500:20599 udp/20500:20599"
client_txp_ports="default"

# Dropbox-related LAN functions
server_dpbx_ports="udp/17500"
client_dpbx_ports="default"

# -----------------------------------------------
# Set up interfaces
# -----------------------------------------------

# LAN Interface
# The purpose of this interface is to control the traffic
# on the eth0 interface with IP 10.0.0.3 (net: "10.0.0.0/24").
interface eth0 lan src "10.0.0.0/24" dst 10.0.0.3

	# policies
	policy reject

	# standard services
	server dhcp accept
	server http accept
	server ICMP accept
	server ms_ds accept
	server ntp accept
	server samba accept
	server ssh accept
	server webmin accept

	# custom services
	server txf accept
	server txp accept
	server dpbx accept

	# client section
	client all accept

# Internet Interface
# The purpose of this interface is to control the traffic
# from/to unknown networks behind the default gateway 10.0.0.1.
interface eth0 internet src not "${UNROUTABLE_IPS} 10.0.0.0/24" dst 10.0.0.3

	# policies
	policy drop
	protection strong

	# Here are the services listening on eth0.
	# TODO: Normally, you will have to remove those not needed.
	server http accept
	server ICMP accept
	server ntp accept
	server ssh accept

	# custom services
	server txf accept
	server txp accept

	# client section
	client http accept
	client https accept
	client ICMP accept
	client ntp accept
	client ssh accept
	client txp accept
	client txf accept

Note that I have cleaned it up considerably and defined custom services at the top of the file. This is not the most condensed configuration file I could make, but it is easy to understand.

Testing Your Configuration File

To test the configuration file you have edited, run the firehol test command.

$ sudo firehol test /tmp/firehol.conf

This will do a syntax check of your configuration file, then allow you to commit the changes to the firewall by typing "commit" at a prompt. I recommend that you don't type commit if your configuration file is not in the right place. Let's move the tested, working configuration file there first.

Enabling Your Configuration

Once you have customized and tested your configuration file, copy it to where it belongs.

$ sudo cp /tmp/firehol.conf /etc/firehol/firehol.conf

And then initialize the firewall.

$ sudo /etc/init.d/firehol start

And you're done!

It isn’t obvious at first how to list all the members of a group from a Linux command line (I’m using Ubuntu Server 10.04 Lucid Lynx).

Listing A User’s Groups

It is very easy to list the groups that a user belongs to. Simply use the groups command.

$ groups user

This command will output the user name and a space-delimited list of all the groups that user belongs to, as follows:

user : user dialout cdrom floppy audio dip vido plugdev users fuse lp admin sambashare

Listing a Group’s Members (Users)

If you want to do the opposite, and list the members of a group, you can use the members command. The catch is: the command is not installed by default. Fortunately, it is simple to install:

$ sudo apt-get install members

Once the package is installed, you can issue the members command similarly to how you issue the groups command.

$ members groupname

This will output a space-delimited list of all the members of the group, as follows:

member1 member2 member3

Unlike the groups command, it does not output the name of the group. You may use the optional arguments --primary and --secondary to list only the primary or secondary users within the group.

When I decided that I wanted an easy way to sync files with my Ubuntu 10.04 Lucid Lynx server, I thought that Canonical’s home-grown solution, Ubuntu One, would be the best option. To my surprise, I learned that Ubuntu One is not supported on the server; it requires a GUI to work.

Fortunately, it is possible to run Dropbox, a service I was already using on my Mac and Windows machines, on a headless Ubuntu (or other Linux) server. It is important to stress, however, that Dropbox isn’t “meant” to be run on a headless server, per se. It will work fine, but there is no special server package to install. You can either install the entire package, which installs all manner of Nautilus dependencies, or you can install the server daemon manually. I am sure that the proprietary nature of the server is why no one has repackaged it as a server install.

I did not want to install all sorts of graphical dependencies, however, so I followed Dropbox’s instructions for a text-based Linux install. At the top of the top of the instructions there is a notice that the instructions are no longer necessary, and a link to a forum post with instructions for a “normal install”. I didn’t know who to believe. I ended up taking bits from both sets of instructions, and omitting a lot of steps that simply aren’t necessary anymore, and I got everything to work. I’m reproducing my work here, so that you don’t have to go through the trial and error that I did.

Installing Dropbox

To install Dropbox with no GUI (specifically, Nautilus) requirements, download the most current Dropbox archive, extract it to your home directory, and run the dropbox command.

$ wget http://dl-web.dropbox.com/u/17/dropbox-lnx.x86-0.7.110.tar.gz
$ tar xzf dropbox-lnx.x86-0.7.110.tar.gz
$ .dropbox-dist/dropbox

Dropbox will spit out numerous messages informing you that you need to link your server to your Dropbox account.

This client is not linked to any account...
Please visit https://www.dropbox.com/cli_link?host_id=blahblahblahblahblah23445435365 to link this machine.

Copy and paste that link into a browser on your client machine. You will then have to log in to Dropbox’s website using your Dropbox account. It won’t be obvious at first that anything happened. If you review the “Events” tab on the website, however, you will see a message indicating that your server has been linked to your account. Now you can go back to the command line.

Kill the Dropbox process with Control-Z.

Creating a Service to Run at Boot

I don’t want to bother running Dropbox manually. It’s a service, and should be set up as such. Dropbox provides excellent instructions on how to do this. I will replicate them concisely here.

First, create a script file.

$ sudo nano /etc/init.d/dropbox

Then, paste in the full dropbox script. which I pulled from the Dropbox web site. (I am in no way claiming authorship of this script.) You have to make one change to the script file. On the second line, set the DROPBOX_USERS variable to your user name, or a space-delimited list of all Dropbox users on your server. After that change is made, save the script and exit nano.

Next, make the script executable.

$ sudo chmod +x /etc/init.d/dropbox

Finally, set up the script to run at boot.

$ sudo update-rc.d dropbox defaults

Note that this script will actually run an instance of the Dropbox daemon under the account of every user specified in the DROPBOX_USERS variable. I have only installed Dropbox for myself (a single user on the system). If you wish to sync multiple users with Dropbox, you will have to go through the steps in the “Installing Dropbox” section for each Dropbox user on your system, prior to running it as a service.

Starting, Stopping, Restarting, and Checking the Status of the Dropbox Service

Now that Dropbox is a service, you can use Ubuntu’s standard commands to manipulate it. First off, you’ll want to start it.

To start the service:

$ sudo service start dropbox

When you first start the service, it will create a “Dropbox” folder (note the capital “D”) right beneath your home directory. Personally, I don’t see a need to change this folder location. If you do, Dropbox provides instructions (http://wiki.dropbox.com/TipsAndTricks/TextBasedLinuxInstall).

To check the status of the service:

$ sudo service dropbox status

To stop the service:

$ sudo service dropbox stop

To restart the service:

$ sudo service dropbox restart

Other Stuff

Removing Botched Installs

If you have already downloaded Dropbox and it isn’t working, delete all the Dropbox files in your home directory to start again with a clean slate. After running the following commands, you can go through the “Installing Dropbox” instructions.

$ cd ~
$ rm -rf .dropbox* Dropbox

Command Line Utilities

I don’t see a need to peek into the Dropbox process very often. If you do, download the Linux CLI add-on.

Today I learned a different way to configure the firewall on my Ubuntu 10.04 Lucid Lynx Server: the ufw command. UFW stands for “Uncomplicated FireWall,” and it’s just that. It provides a simpler interface to add or remove firewall rules to iptables, the default Linux firewall. It’s installed on Ubuntu Server by default (and has been, since Ubuntu 8.04), and I find a little simpler than the application I used to use to configure my firewall: Firehol. (Here’s how to set up Firehol, if you are interested. It’s more difficult than ufw, in my opinion, but a lot easier than setting up iptables manually!)

A new Ubuntu Server install (as of 10.04) contains a firewall (iptables) that is not enabled. Ubuntu.com has a great tutorial that explains that ufw is the default configuration tool for iptables. After I set up my server, I used ufw to close all ports by default, then open up ports for the services I use. I don’t have complex security needs or run a proxy server, so my rules are simple.

Adding Rules

Before adding rules, it’s best to explicitly set the default behavior. By default, I like to block everything: both incoming and outgoing traffic. After that is done, I selectively open ports to support the services I wish to run. In contrast, UFW, by default, denies all incoming traffic but allows all outgoing traffic. That setup is accomplished manually with the following commands.


$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing


The following commands open ports for named services that I use: namely, SSH (port 22), a web server (port 80), and Webmin (port 10000). Any services named in /etc/services may be identified by name instead of port number.

$ sudo ufw allow ssh
$ sudo ufw allow www
$ sudo ufw allow webmin

UFW also has a list of application presets, for common servers such as Apache, OpenSSH, Lighttpd, and Samba. You can view the list by issuing the command:

$ sudo ufw app list

You can implement firewall rules for Samba and Lighttpd by using the commands below, which specify the application name, not the service name. Note that you must enclose in quotation marks any application names that include spaces.

$ sudo ufw allow Samba

It’s better to limit Samba access to hosts on your LAN. Using ufw’s more complex syntax, you can do just that. Note that you have to add “app” before the application name in this case.

$ sudo ufw allow from 10.0.0.0/8 to 127.0.0.1 app Samba
$ sudo ufw allow to 10.0.0.0/8 from 127.0.0.1 app Samba

The following commands open the ports required by my Transmission-Daemon server. Here I must specify port numbers explicitly. Note that you use a colon instead of a dash to specify port ranges. Plus, when creating rules for port ranges, you must specify whether they apply to TCP or UDP.

$ sudo ufw allow 9091
$ sudo ufw allow 20500:20599/tcp
$ sudo ufw allow 20500:20599/udp


The following command opens up ports needed for MySQL, but only to hosts within the local network.

$ sudo ufw allow from 10.0.0.0/8 to any port 3306/tcp

If you wish to open up MySQL to the world, you could use a simpler syntax.

$ sudo ufw allow mysql

Deleting Rules

Deleting rules is pretty simple. Just use the following syntax, and replace <…> with the entire rule that you wish to delete.

$ sudo ufw delete <...>

For example:

$ sudo ufw delete allow ssh
$ sudo ufw delete allow 10000

You can also delete all the rules with a single command.

$ sudo ufw reset

Enabling the Firewall

The following command enables the firewall rules immediately, and upon subsequent system restarts. This command will also refresh the rules. Run this command each time you update your firewall configuration.

$ sudo ufw enable

Disabling the Firewall

To disable the firewall, simply issue the following command.

$ sudo ufw disable

Checking the Configuration

You can check your configuration by issuing one of the following commands. The “verbose” version shows more information.

$ sudo ufw status
$ sudo ufw status verbose

Firewall Configuration Script

Here is a script that I wrote to set up my firewall. This script resets the firewall to deny everything but the services/applications I have installed on my server. Run it with sudo. You only have to run it once, not on every boot.

#!/bin/sh

# obtain server's IP address
SERVERIP=`hostname --all-ip-addresses | cut --fields 1 --delimiter " "`

# disable firewall
ufw disable

# reset all firewall rules
ufw reset

# set default rules: deny all incoming traffic, allow all outgoing traffic
ufw default deny incoming
ufw default allow outgoing

# open port for SSH
ufw allow OpenSSH

# open port for Webmin
ufw allow webmin

# open ports for Samba file sharing
ufw allow from 10.0.0.0/8 to $SERVERIP app Samba
ufw allow to 10.0.0.0/8 from $SERVERIP app Samba

# open ports for Transmission-Daemon
ufw allow 9091
ufw allow 20500:20599/tcp
ufw allow 20500:20599/udp

# open port for MySQL
ufw allow proto tcp from 10.0.0.0/8 to any port 3306

# open ports for Lighttpd
ufw allow “Lighttpd Full”

# open port for network time protocol (ntpd)
ufw allow ntp

# enable firewall
ufw enable

# list all firewall rules
ufw status verbose

Transmission-daemon does not automatically remove completed torrents. I created a script to do so, and I run it via a cron job (under my own user ID) on my Ubuntu 10.04 Lucid Lynx Server.

This script uses command line utility transmission-remote (which is installed when you install the transmission-daemon package) to check on the progress of all active torrents. Torrents that have completed downloading and seeding will have a status of “Stopped” and a (download) percent done of “100%”. This script will identify such torrents, move their downloaded files to a different directory, and will remove the torrent from transmission-daemon.

This script depends on using a seeding ratio to stop your torrents from seeding after a certain percentage has been seeded. If your torrents seed indefinitely, transmission-daemon will never stop them, and this script will not know that the torrent is completed.

Note that transmission-daemon has a built-in option to store incomplete downloads in one folder, and move them, when the download completes, to another folder. This script is meant to pick the torrents up afterward, and move them from the download folder to somewhere more permanent. If you just want to remove completed torrents, comment out the line “transmission-remote --torrent $TORRENTID --netrc $NETRC --move $MOVEDIR“.

Script: movecompletedtorrents

This script is a modified version of the one found at http://forum.qnap.com/viewtopic.php?f=45&t=25553.

#!/bin/sh

# script to check for complete torrents in transmission folder, then stop and move them

MOVEDIR=/home/user/media
NETRC=/home/user/.netrc

# get torrent list from transmission-remote list
# delete first / last line of output
# remove leading spaces
# get first field from each line
TORRENTLIST=`transmission-remote --list --netrc $NETRC | sed -e '1d;$d;s/^ *//' | cut -s -d " " -f1`

# for each torrent in the list
for TORRENTID in $TORRENTLIST
do
  echo "* * * * * Operations on torrent ID $TORRENTID starting. * * * * *"

  # check if torrent was started
  STARTED=`transmission-remote --torrent $TORRENTID --info --netrc $NETRC | grep "Id: $TORRENTID"`
  # echo " - started state = $STARTED" # debug message

  # check if torrent download is completed
  COMPLETED=`transmission-remote --torrent $TORRENTID --info --netrc $NETRC | grep "Percent Done: 100%"`
  # echo " - completed state = $COMPLETED" # debug message

  # check torrent's current state is "Stopped"
  STOPPED=`transmission-remote --torrent $TORRENTID --info --netrc $NETRC | grep "State: Stopped"`
  # echo " - torrent stopped seeding = $STOPPED" # debug message

  # if the torrent is "Stopped" after downloading 100% and seeding, move the files and remove the torrent from Transmission
  if [ "$STARTED" != "" ] && [ "$COMPLETED" != "" ] && [ "$STOPPED" != "" ]; then
    echo "Torrent #$TORRENTID is completed."
    echo "Moving downloaded file(s) to $MOVEDIR."
    transmission-remote --torrent $TORRENTID --netrc $NETRC --move $MOVEDIR
    echo "Removing torrent from list."
    transmission-remote --torrent $TORRENTID --netrc $NETRC --remove
  else
    echo "Torrent #$TORRENTID is not completed. Ignoring."
  fi

  echo "* * * * * Operations on torrent ID $TORRENTID completed. * * * * *"

done

Scheduling

To run this script every hour, save the script somewhere (/home/user/bin), grant yourself permission to execute it…

$ chmod u+x ~/bin/movecompletedtorrents

…edit your crontab settings…

$ crontab -e

…and add the following line.

@hourly /home/user/bin/movecompletedtorrents

TVNamer is a Python application that will automatically renames TV episode files to a clean, consistent, and useful format. If you rip your DVDs, or download torrents (for shame!) on your server, you might find the “tvnamer” Python script very useful. I am running this on my Ubuntu 10.04 Lucid Lynx server right now, and I am very happy with it.

How It Works

I thought I would have to write a complex regular expressions script to reformat the junky file names I end up with. TVNamer does this behind the scenes, and goes one very useful step beyond that: it compares the file names you pass it to the TVDB database, via the TVDB API, to ensure you get a good match. The TVDB is an open database that stores the show titles, episode numbers, and so on, for nearly any show you can think of. As long as your file name contains the series title (in some format or another), and the season and episode numbers (S01E01, for example), TVNamer will rename them nicely.

Installation

TVNamer requires Python 2.6. If you don’t have it, install it as follows:

apt-get install python2.6

The tvnamer program is simple to install on Ubuntu (I run Ubuntu Server 10.04 Lucid Lynx) via Python’s “easy_install” command. (This probably works on all platforms that Python runs on, too.)

$ sudo easy_install tvnamer

Running Manually

You can run tvnamer on one file…

$ tvnamer seriesname.s01e01.crap.you.do.not.care.about.in.the.file.name.mkv

…or on an entire folder.

$ tvnamer ~/media/tv/

It can also recursively run on any folder tree that you pass to it. I have found this to be the default behavior, though, so the –recursive argument does not appear to be necessary. (I tend to use the argument in my cron jobs, in case future versions of tvnamer start to require it.)

$ tvnamer --recursive ~/media/tv/

Note that when you run the program as written above, it will run in interactive mode. For each file you pass to it, you will have to verify the series name and verify that you wish to rename the file.

Running Automatically on a Schedule

Fortunately, you can also run tvnamer in batch mode. Batch mode will not prompt the user at all, and will automatically select the most-relevant series title and will automatically rename each file. This is the mode to run tvnamer via a cron job or a shell script.

Batch mode:

$ tvnamer --batch ~/media/tv/

To schedule a cron job to run this script hourly, run:

$ crontab -e

and then add the following line to your crontab:

@hourly tvnamer --batch --recursive ~/media/tv/

and finally save the crontab file.

When I read posts on LifeHacker and Gizmodo sometimes, I figure that everybody else in the world is downloading TV episodes via BitTorrent, and doing it using RSS feeds and an automated process. That can’t possibly be true, but I figured that if other people do this, I should be able to figure out how, too. It is pretty easy to find RSS feeds for various TV shows. Therfore, I thought it would be relatively simple to write a script to download episodes from each one. I figured that I would have to figure out how to sort out duplicate episodes and different quality types (720p, 1080p, etc.) somehow, too. It didn’t take me too long to find out that an excellent Python program called FlexGet already does all this. Plus, it runs from the command line, which is perfect for a headless server, like my Ubuntu Lucid Lynx box.

I don’t actually endorse doing this, by the way, but I was really curious to learn how it was done. I wanted to share what I learned, in case anyone was struggling with the configuration.

Installation

FlexGet’s web site has an excellent walkthrough that describes how to install it on various operating systems. My instructions in this section are not meant to improve on that guide, but merely to summarize it for an Ubuntu Server install.

FlexGet requires Python 2.6. If you don’t already have that installed on your server, try:

$ sudo apt-get install python2.6 python-setuptools

You have to download FlexGet from their web site. Choose the version for Python 2.6.

After you download FlexGet, install it via Python’s easy_install script. You’ll have to specify the name of the FlexGet file you downloaded.

$ sudo easy_install <downloaded egg>

That’s it! Flexget is now installed in the /usr/local/bin directory.

Scheduling

FlexGet can be run manually, but really should be scheduled to run periodically via cron. Note that, there is no need to run FlexGet as root. I run it under my own user account. If you want to operate under the principle of least privilege, you could create a “flexget” account, edit its crontab, and grant that account the permissions it needs to get the job done. For my needs, that isn’t necessary. Therefore, to schedule FlexGet to run every hour, I simply edit my own crontab.

crontab -e

Add the following line to the crontab file, and save it.

@hourly /usr/local/bin/flexget --cron /home/user/media/

This cron job will run FlexGet at the top of every hour. FlexGet will read its options from the configuration file in its default location, which is inside your home directory (/home/user/.flexget/config.yml). I actually choose to log FlexGet’s verbose results, so I can be sure that the cron job is running. Here is my crontab entry:

@hourly /usr/local/bin/flexget -v --cron /home/user/media/ > /home/user/log/tvnamer.log

Configuration

Configuring FlexGet is deceptively simple. The configuration file is written in a format I had heard of, but never encountered before: YAML. YAML is simple and powerful, which means that it can be deceptively complex. You can write a config file that does the same thing in a million different ways. FlexGet’s Cookbook shows you a few examples to get you started. It took me a lot of experimentation to find a config file format that I liked. Here is what I ended up with. It’s probably more complex than it needs to be, but I like the idea of using presets to customize different types of downloads. Note that, in YAML, each level of indentation is two spaces (no tabs are allowed).

~/.flexget/config.yml

presets:
  tv:
    series:
      settings:
        720p:
          timeframe: 8 hours
      720p:
        - series title 1
        - series title 2
        - series title 3

    transmissionrpc:
      host: localhost
      port: 9091
      netrc: /home/user/.netrc

feeds:
  TvTorrent1:
    rss: http://tv.rss.url/
    preset: tv

  TvTorrent2:
    rss: http://tv.rss.url/
    preset: tv

This configuration file sets up a preset called “tv” which tells flexget to wait 8 hours before selecting episodes of any of the series listed. As far as I understand it, the 720p refers to the maximum acceptable quality of the episode. If FlexGet finds an episode of that quality, it will download it; if it finds episodes above that quality level, at 1080p for example, it will ignore them. For my needs, 720p (FlexGet’s default setting) is adequate. If no 720p version exists after the “timeframe” of 8 hours is over, flexget will select the entry with the highest quality available. This is pretty cool. FlexGet won’t download the same episode more than once, unless there is a “proper” or “repack” version—a corrected version of a previously released episode—is available.

At the end of the “tv” preset, the “tranmissionrpc” heading instructs flexget to use the transmissionrpc plugin to directly send torrent URLs to transmission-daemon for download. This obviates the need to set a watch directory in transmission-daemon. Note that the “netrc” setting refers to a special username/password file that contains the user name and password required to log into transmission daemon. That file should be in the following format, hidden, in your home directory. Make sure you grant only your account access to read that file (i.e., chmod 600 ~/.netrc).

machine localhost
        login username
        password password123456789

If you do not run transmission-daemon, you could replace the “transmissionrpc” section with a “download” section, or use another plugin. Check out the flexget Cookbook for more information.

The “feeds” section lists all the TV torrent feeds to look in. I have each feed set to use the “tv” preset that I defined above.

Each RSS feed you list can contain a number of TV series (which is the easiest way to set it up), or just one series (you’ll have to find or build custom feeds for this). Flexget will search all the feeds you list for the series titles listed in the “tv” preset.

This configuration file works very well for me. I hope that it helps you get up and running.

This article explains how to install, configure, and run Transmission on a headless Ubuntu Lucid Lynx (10.04) server. Transmission, specifically transmission-daemon, is a BitTorrent client that can be accessed via a web browser and other front-ends, including a desktop app called trasmission-remote-gui and Android apps, including Transdroid (my favorite) and TorrentFu.

Transmission’s daemon is a good choice over rtorrent, an excellent ncurses client that I used to prefer, because Transmission is simpler to configure, and requires no other software to be installed for accessibility via a web browser or external client.

Installation

First, install the Transmission daemon using apt-get. This part is simple.

$ sudo apt-get install transmission-daemon

This installs the app and starts the daemon. The daemon runs under the user “debian-transmission”, which belongs to a group of the same name.

Configuration

Configuring Transmission is a little tricky, due to the placement of the configuration file (out of the box, the daemon doesn’t read /etc/transmission-daemon/settings.json, as you would expect), and due to the service’s method of saving (overwriting) its configuration file on exit. Restarting the service, if it is already running, will result in it saving the configuration file, and overwriting any changes you might have made to it. Instead of restarting the service after editing the configuration file, you should reload the service. The following code sends a SIGHUP signal to transmission-daemon, which causes it to reload its configuration file.

$ sudo service transmission-daemon reload

Edit the settings file using your favorite text editor. Note that the default location is /var/lib/transmission-daemon/info/settings.json.

$ sudo nano /var/lib/transmission-daemon/info/settings.json

Here is my settings.json file. I have almost all the options set. I recommend that you use my file as a starting point. You have to edit the rpc-whitelist setting to connect to the server from anything other than the local host. You should also edit the paths (download-dir, watch-dir, incomplete-dir) as you see fit.

{
"alt-speed-down": 500,
"alt-speed-enabled": true,
"alt-speed-time-begin": 480,
"alt-speed-time-day": 127,
"alt-speed-time-enabled": true,
"alt-speed-time-end": 0,
"alt-speed-up": 10,
"bind-address-ipv4": "0.0.0.0",
"bind-address-ipv6": "::",
"blocklist-enabled": false,
"dht-enabled": true,
"download-dir": "\/home\/user\/dl\/torrent",
"download-limit": 1000,
"download-limit-enabled": 0,
"encryption": 2,
"incomplete-dir": "\/var\/lib\/transmission-daemon\/info\/Incomplete",
"incomplete-dir-enabled": false,
"lazy-bitfield-enabled": true,
"max-peers-global": 200,
"message-level": 2,
"open-file-limit": 32,
"peer-limit-global": 240,
"peer-limit-per-torrent": 60,
"peer-port": 20683,
"peer-port-random-high": 20500,
"peer-port-random-low": 20599,
"peer-port-random-on-start": true,
"peer-socket-tos": 0,
"pex-enabled": true,
"port-forwarding-enabled": false,
"preallocation": 1,
"proxy": "",
"proxy-auth-enabled": false,
"proxy-auth-password": "",
"proxy-auth-username": "",
"proxy-enabled": false,
"proxy-port": 80,
"proxy-type": 0,
"ratio-limit": 0.2500,
"ratio-limit-enabled": true,
"rename-partial-files": true,
"rpc-authentication-required": true,
"rpc-bind-address": "0.0.0.0",
"rpc-enabled": true,
"rpc-password": "password",
"rpc-port": 9091,
"rpc-username": "transmission",
"rpc-whitelist": "127.0.0.1,*.*.*.*",
"rpc-whitelist-enabled": true,
"speed-limit-down": 1500,
"speed-limit-down-enabled": true,
"speed-limit-up": 50,
"speed-limit-up-enabled": true,
"umask": 2,
"upload-slots-per-torrent": 4,
"watch-dir": "\/home\/user\/dl\/torrent",
"watch-dir-enabled": true
}

It is (for the most part) safe to enter a plaintext password in this file. If you do so, when transmission-daemon closes, and it rewrites its settings file, the password will be saved in a SHA1 encrypted format. Just restart the service after completing these instructions, and your password will be encrypted.

I encountered difficulties with file permissions when I had Transmission save downloads within the /var/lib/transmission-daemon/. Therefore, I changed Transmissions setting to download to a folder within my home directory. To ensure that both I and the debian-transmission user had read/write permissions, I added my account to the debian-transmission group (which is created when you install transmission), and I changed the group ownership of my torrent download directory to debian-transmission. Doing so grants the debian-transmission group read/write access to that folder, preserves read/write/execute access for my account, and prevents access to all others.

$ sudo usermod -a -G debian-transmission mjdescy
$ chgrp debian-transmission ~/dl/torrent
$ chmod 770 ~/dl/torrent

I also changed the “umask” setting from the default of 18 to 2. That means transmission-daemon will create files with read/write permissions set at both the user and group level.

After saving any changes, tell the daemon to reload its configuration file.

$ sudo service transmission-daemon reload

That finishes configuration. Note that you will likely have to set up port forwarding on your router to access the web app from outside your network, and to seed properly.

Web Access

To access the server, point your web browser to http://your.server.ip.address:9091. (9091 is the default port, which can be changed, under “rpc-port”, in the settings file.) The web interface is pretty simple. Note that you can adjust some settings, such as transfer speed and download directory, by clicking the gear on the lower right of the page.