After about a year of enjoying how well my Ubuntu Linux home server downloads, organizes, and serves video files to my TV, I started to think harder about serving up music to devices around the house, and to my phone and/or PC while I’m at work. In the past, I’ve made my music library available on my home network via MediaTomb, minidlna, and Samba, but I never made use of it. Instead, I always used iTunes, which is good software, but very limiting.

In December, I realized two things:

1. I almost never sync my iPods to my computer anymore, because they charge in my car or in my home stereo’s dock.
2. I also never use my Android phone to listen to music, even though it is perfectly capable, because I never wanted to bother to get the USB cable out for my phone, and I haven’t found any software that I like to sync with.

My problem: syncing. I hate it. It is passé and worthless when everything is network connected and your music collection is vastly larger than what a reasonably-priced amount of flash memory will hold.

Around the same time, I received a Cr-48 from Google. It runs Chrome OS, which can play audio via the browser (Flash or HTML 5), but that’s it. I thought it would be an interesting challenge to find a good way to play my music library on it. I also wanted to store my library on my power-efficient Ubuntu Linux server, so I wouldn’t have to leave my power-hungry desktop on all day.

Searching for a Solution

I found two highly respected options that would allow me to stream music to a browser or other device: Ampache and Subsonic. Ampache is built on top of the LAMP stack, so it requires Apache, MySQL, and PHP, which a ton of Linux servers have installed on them. It has a good number of features, too, including an Android client and front-end support in Amarok and WinAmp. I ended up not liking it, however, for two main reasons: The system requirements are rather high, mostly because it depends on MySQL as a data store. Also, indexing my 80 GB music library was very slow.

What I ended up preferring, and falling in love with, was Subsonic. Subsonic is a Java app, which you can install either as a standalone Debian package in Ubuntu, or as a WAR file that you can run under TomCat or another Java Server. It runs on any host OS that supports Java, so Windows and Mac users can run it too. Subsonic scanned my music library surprisingly quickly, looks great, and worked right away. I have it installed under TomCat, and configured to use SSL (for security and to disguise my music downloading/streaming from network administrators).

Subsonic

Subsonic

Subsonic offers a somewhat complex, but attractive and skinnable, web interface. It offers playlist support (though not smart playlists as in iTunes), random playlists, search, and support for album art and metadata (including a tag editor). It is far more album-oriented than song oriented, which took some getting used to moving from iTunes. For example, albums can be rated, but not individual songs. Also, the home page on the interface defaults to random albums, which looks nice and pretty, but differs greatly from how I use iTunes. I really like the feature, though, because it allows me to rediscover music that has long been buried by my iTunes smart playlists.

Playback in the browser is done via an open-source Flash plugin, which works fine. Album art must be embedded in the files, or at least stored at the folder level, which was initially a challenge for me due to the way iTues handles album art in its library. I used MediaMonkey to help get my album art into shape. The server can be set up to transcode music in formats such as Flac to MP3 or AAC to support the client player’s capabilities and to limit bandwidth.

Subsonic doesn’t really stream your music; it downloads and caches it. The browser client caches the song you are playing, while the iOS, Android, and Windows Phone 7 clients will cache a number of songs ahead of the current one. Caching is way better than streaming, because you don’t need a constant connection to the server. It works really well. The server has a “Status” window in which you can review all the connected clients and the bandwidth they are using, which is pretty cool.

Subsonic is free and open source software, though it does have a somewhat controversial donation requirement if you wish to use it via a phone app. You can connect to it via a web browser indefinitely, but if you use the Android app, as I do, it will tell you that you must donate to the project/developer after 30 days, or it will refuse to access the server. After seeing how well the Android app worked, I donated right away. Remember, even in the open source world, it is a good idea to support good developers.

By creating multiple user accounts, you can also use Subsonic to share music with friends and family. If you configure the server with SSL, you can do so securely and privately as well. Users can be given access to download and upload music, so it could become a music sharing platform for you and your close friends that flies under the radar.

All in all, I love Subsonic and recommend it to anyone who has a large music library that they wish to bring with them and share with close friends.

Learn Linux, 101: Manage file permissions and ownership [IBM]

Monit is a utility that you can set up to monitor your server’s processes and files, alert you to problems, restart stopped services, and present a web interface to display the server’s current status. I suspect most administrators would configure it to email them when services are down, drives are nearly full, load is too high, and so on. I don’t bother with email alerts, but I really like Monit’s web front-end and service restarting capabilities.

Monit system monitoring (yes, my server is redlining!)

Install Monit

Installing Monit on an Ubuntu server is accomplished in the usual way.

$ sudo apt-get install monit

Configure Monit

You have to edit a configuration file to start monit on your server.

$ sudo nano /etc/defaults/monit

Default configuration file

# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Stefan Alfredsson

# You must set this variable to for monit to start
startup=0

# To change the intervals which monit should run,
# edit the configuration file /etc/monit/monitrc
# It can no longer be configured here.

You have to change “startup=0″ to “startup=1″.

Edit the Control File

Monit’s control file is the place to set up system, service, and file monitoring. The default control file is all commented out. When I set mine up, I uncommented various lines and inserted sections for each service at the bottom of the file.

$ sudo nano /etc/monit/monitrc

Edit the file (uncomment lines), to enable the following options, which set up a monitoring daemon and enables the web interface (http://localhost:2812).

set daemon  120

set logfile syslog facility log_daemon

set httpd port 2812
allow localhost
allow 192.168.1.0/255.255.255.0

check system localhost

if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 75% then alert
if cpu usage (user) > 70% then alert
if cpu usage (system) > 30% then alert
if cpu usage (wait) > 20% then alert

check filesystem root with path /

Add the following sections to the bottom of the file to monitor cron, sshd, samba, mediatomb, lighttpd, and MySQL (which I am running to sync XMBC across multiple PCs). Services that run with a pid file are simple to monitor. MySQL does not run with a pidfile, but can easily be configured to by adding the line "pid-file = /var/run/mysqld/mysqld.pid" to the [mysqlid] section of /etc/mysql/my.cnf.

# cron
check process cron with pidfile /var/run/crond.pid
group system
start program = "/etc/init.d/cron start"
stop  program = "/etc/init.d/cron stop"
if 5 restarts within 5 cycles then timeout
depends on cron_rc

check file cron_rc with path /etc/init.d/cron
group system
if failed checksum then unmonitor
if failed permission 755 then unmonitor
if failed uid root then unmonitor
if failed gid root then unmonitor

# sshd
check process sshd with pidfile /var/run/sshd.pid
start program  "/etc/init.d/ssh start"
stop program  "/etc/init.d/ssh stop"
if failed port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

#  samba
check process smbd with pidfile /var/run/samba/smbd.pid
group samba
start program = "/etc/init.d/smbd start"
stop  program = "/etc/init.d/smbd stop"
if failed host 192.168.1.3 port 139 type TCP  then restart
if 5 restarts within 5 cycles then timeout

# mediatomb
check process mediatomb with pidfile /var/run/mediatomb.pid
group mediatomb
start program = "/etc/init.d/mediatomb start"
stop  program = "/etc/init.d/mediatomb stop"
if 5 restarts within 5 cycles then timeout

# lighttpd
check process lighttpd with pidfile /var/run/lighttpd.pid
group lighttpd
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

# mysql
# note: requires mysql to run with a pid.
check process mysql with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Unfortunately, monit can’t monitor the transmission-daemon service, because transmission-daemon does not run with a pid file, and forks itself sometimes. There is a warning in the transmission-daemon launch script, /etc/init.d/transmission-daemon, which says that pid file support will not work, because the daemon forks. Hopefully this will be resolved in the future.

Using Monit

After editing your configuration file, restart monit.

$ sudo service monit restart

Now you can check the health of your server by pointing a web browser to your server’s IP address, port 2812 (http://server:2812/).

The WD TV LIVE’s external hard drive is NTFS-formatted, and is mounted to my media server’s filesystem via smbmount.

The “syncmedia” Script

This is my script, which I store in /home/mjdescy/bin/syncmedia.

#!/bin/sh
# mirror media folder to server

SOURCE_DIR=/home/mjdescy/media/
DEST_DIR=/home/mjdescy/wdtvlive/

# perform simple, one-way sync using rsync; use setlock to prevent simultaneous instances from being run
/usr/bin/setlock -n /tmp/.mediasync.lock /usr/bin/rsync --size-only --verbose --recursive --stats --delete --exclude=".*" --modify-window=1 --times --bwlimit=1000 $SOURCE_DIR $DEST_DIR

Execute Permissions

Make sure to set execute permissions on the script before you try to run it.

$ chmod u+x /home/mjdescy/bin/syncmedia

Notes on “setlock”

Note the use of “setlock” (which is part of the “daemontools” package) to prevent the rsync command from being run if it is already running. This is necessary if the time it takes to copy the files can exceed the interval at which the script is executed via cron.

Install daemontools in Ubuntu via:

$ sudo apt-get install daemontools

Rsync Options

Using rsync to copy the files to a Samba share, mounted with smbfs, presented some challenges. Because the server and the share are on two different file systems, copying permissions would either not work or produce unexpected results. Therefore, I couldn’t use the rsync “archive” option, as I typically do. Therefore, I selected the options “modifiy-windows” and “times” to help prevent the same files from being copied again when there wasn’t really a change.

I used the “size-only” option because my media files aren’t edited and updated like office documents; they are either downloaded or created once via Handbrake.

I used the “exclude” option to avoid deleting the “.wd_tv” folder that the WD TV LIVE uses to store its library information on the external hard drive.

The “delete” option will ensure that deletions on my media server (the master) are replicated on the WD TV LIVE (the slave).

I used the “bwlimit” option to avoid taking all the upload bandwidth away from my server for the sync. This is entirely optional. After the initial data move, this process should run primarily during the day, when we are not watching the TV, so slow uploads are not a big problem.

Sheduling

I use my account’s crontab to kick off this script every hour.

$ crontab -e

Add the following line and save the crontab.

@hourly /home/mjdescy/bin/syncmedia > /dev/null 2>&1

Now the process will be kicked off at the top of every hour. Rsync will not move files if there have been no changes to the source folder, and will move only new/changed files to the destination folder. If an instance of this process is already running, another one will not be created.

Example of the TV folder tree:

• tv
  • series name 1
    • Season 01
  • series name 2
    • Season 01
    • Season 02
  • series name 3
    • Season 05

Install TVNamer

First, install TVNamer using the instructions found in my prior post on the subject.

Generate and Customize the TVNamer Configuration File

Running TVNamer without a configuration file renames files in place, without too many options as to how it is done. Running TVNamer with a configuration file, on the other hand, unlocks more powerful functionality. I will show you how to configure TVNAmer to move your TV episodes into a well-organized folder tree, using the series name, season number, and other metadata to create the folder- and file names. (A word of caution: While TVNamer works superbly on episodic content, it does not currently work shows broadcast daily, such as “The Daily Show” and “The Colbert Report”, due to how the files are named.)

To begin using a TVNamer configuration file, you must create one. Luckily, it is easy to tell TVNamer to create a default configuration file for you. Run TVNamer, as follows, to create a configuration file in your home folder.

$ tvnamer --save=~/.tvnamerconfig.json

Now, edit that configuration file. The file is in JSON (JavaScript Object Notation) format, which means that you must be sure your syntax is consistent, and all quotation marks and braces are closed.

$ nano ~/.tvnamerconfig.json

Your configuration file will look similar to my file, which is reproduced below. Any lines that I changed from the default are in boldface. I recommend setting “batch” and “recursive” to true. To move renamed files to a new folder, edit the following attributes: “move_files_confirmation”, “move_files_destination” and “move_files”.

You can pick metadata keywords out of the various “filename_*” attributes. The most useful ones will likely be ”seriesname”, “seasonno”, ”episodename”, and “episode”. These can be used not only in file names, but also in the folder path for the “move_files_destination” attribute. Note the syntax used to insert the metadata values into the attribute value: %(keyword)s. When inserting these metadata strings into the attributes you edit, you must keep the control characters–the “%(” that precedes the keyword, and the )s that follows it–intact. Numbers are treated a little differently in terms of the suffix: 02dx instead of s. This follows Python string formatting syntax, and is explained in the “Custom Output Filenames” section of TVNamer’s web site.

The configuration file below sets up TVNamer to run in batch mode, recursively process any folders passed to it on the command line, process only files with video file extensions, and to move any renamed files to a specific folder tree (~/media/tv/(series name)/Season (season number)),  I don’t change how files are named, but you could easily do so by editing the attributes that start with “filename_”. I highly recommend limiting TVNamer’s activities to video files using the “valid_extensions” attribute, because I’ve had non-video files confuse the script. (I never believed that would happen, but it did, and the fix is simple.)

{
“always_rename”: false,
“batch”: true,
“episode_separator”: “-”,
“episode_single”: “%02d”,
“filename_patterns”: [
"^\\[.+?\\][ ]? # group name\n        (?P<seriesname>.*?)[ ]?[-_][ ]?          # show name, padding, spaces?\n        (?P<episodenumberstart>\\d+)              # first episode number\n        ([-_]\\d+)*                               # optional repeating episodes\n        [-_](?P<episodenumberend>\\d+)            # last episode number\n        [^\\/]*$”,
“^\\[.+?\\][ ]? # group name\n        (?P<seriesname>.*) # show name\n        [ ]?[-_][ ]?(?P<episodenumber>\\d+)\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        [Ss](?P<seasonnumber>[0-9]+)             # s01\n        [\\.\\- ]?                                 # separator\n        [Ee](?P<episodenumberstart>[0-9]+)       # first e23\n        ([\\.\\- ]+                                # separator\n        [Ss](?P=seasonnumber)                    # s01\n        [\\.\\- ]?                                 # separator\n        [Ee][0-9]+)*                             # e24 etc (middle groups)\n        ([\\.\\- ]+                                # separator\n        [Ss](?P=seasonnumber)                    # last s01\n        [\\.\\- ]?                                 # separator\n        [Ee](?P<episodenumberend>[0-9]+))        # final episode number\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        [Ss](?P<seasonnumber>[0-9]+)             # s01\n        [\\.\\- ]?                                 # separator\n        [Ee](?P<episodenumberstart>[0-9]+)       # first e23\n        ([\\.\\- ]?                                # separator\n        [Ee][0-9]+)*                             # e24e25 etc\n        [\\.\\- ]?[Ee](?P<episodenumberend>[0-9]+) # final episode num\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        (?P<seasonnumber>[0-9]+)                 # first season number (1)\n        [xX](?P<episodenumberstart>[0-9]+)       # first episode (x23)\n        ([ \\._\\-]+                               # separator\n        (?P=seasonnumber)                        # more season numbers (1)\n        [xX][0-9]+)*                             # more episode numbers (x24)\n        ([ \\._\\-]+                               # separator\n        (?P=seasonnumber)                        # last season number (1)\n        [xX](?P<episodenumberend>[0-9]+))        # last episode number (x25)\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        (?P<seasonnumber>[0-9]+)                 # 1\n        [xX](?P<episodenumberstart>[0-9]+)       # first x23\n        ([xX][0-9]+)*                            # x24x25 etc\n        [xX](?P<episodenumberend>[0-9]+)         # final episode num\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        [Ss](?P<seasonnumber>[0-9]+)             # s01\n        [\\.\\- ]?                                 # separator\n        [Ee](?P<episodenumberstart>[0-9]+)       # first e23\n        (                                        # -24 etc\n             [\\-]\n             [Ee]?[0-9]+\n        )*\n             [\\-]                                # separator\n             (?P<episodenumberend>[0-9]+)        # final episode num\n        [\\.\\- ]                                  # must have a separator (prevents s01e01-720p from being 720 episodes)\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?          # show name\n        (?P<seasonnumber>[0-9]+)                 # 1\n        [xX](?P<episodenumberstart>[0-9]+)       # first x23\n        (                                        # -24 etc\n             [\\-][0-9]+\n        )*\n             [\\-]                                # separator\n             (?P<episodenumberend>[0-9]+)        # final episode num\n        ([\\.\\- ].*                               # must have a separator (prevents 1×01-720p from being 720 episodes)\n        |\n        $)”,

“^(?P<seriesname>.+?)[ \\._\\-]          # show name and padding\n        \\[                                       # [\n            ?(?P<seasonnumber>[0-9]+)            # season\n        [xX]                                     # x\n            (?P<episodenumberstart>[0-9]+)       # episode\n            (- [0-9]+)*\n        -                                        # -\n            (?P<episodenumberend>[0-9]+)         # episode\n        \\]                                       # \\]\n        [^\\/]*$”,

“^(?P<seriesname>.+?)[ \\._\\-]\n        [Ss](?P<seasonnumber>[0-9]{2})\n        [\\.\\- ]?\n        (?P<episodenumber>[0-9]{2})\n        [^0-9]*$”,

“^((?P<seriesname>.+?)[ \\._\\-])?       # show name and padding\n        \\[?                                      # [ optional\n        (?P<seasonnumber>[0-9]+)                 # season\n        [xX]                                     # x\n        (?P<episodenumber>[0-9]+)                # episode\n        \\]?                                      # ] optional\n        [^\\/]*$”,

“^((?P<seriesname>.+?)[ \\._\\-])?\n        [Ss](?P<seasonnumber>[0-9]+)[\\.\\- ]?\n        [Ee]?(?P<episodenumber>[0-9]+)\n        [^\\/]*$”,

“\n        ^((?P<seriesname>.+?)[ \\._\\-])?         # show name\n        (?P<year>\\d{4})                          # year\n        [ \\._\\-]                                 # separator\n        (?P<month>\\d{2})                         # month\n        [ \\._\\-]                                 # separator\n        (?P<day>\\d{2})                           # day\n        [^\\/]*$”,

“^(?P<seriesname>.+?)[ ]?[ \\._\\-][ ]?\n        [Ss](?P<seasonnumber>[0-9]+)[\\.\\- ]?\n        [Ee]?[ ]?(?P<episodenumber>[0-9]+)\n        [^\\/]*$”,

“\n        (?P<seriesname>.+)                       # Showname\n        [ ]-[ ]                                  # -\n        [Ee]pisode[ ]\\d+                         # Episode 1234 (ignored)\n        [ ]\n        \\[                                       # [\n        [sS][ ]?(?P<seasonnumber>\\d+)            # s 12\n        ([ ]|[ ]-[ ]|-)                          # space, or -\n        ([eE]|[eE]p)[ ]?(?P<episodenumber>\\d+)   # e or ep 12\n        \\]                                       # ]\n        .*$                                      # rest of file\n        “,

“^(?P<seriesname>.+)[ \\._\\-]\n        (?P<seasonnumber>[0-9]{1})\n        (?P<episodenumber>[0-9]{2})\n        [\\._ -][^\\/]*$”,

“^(?P<seriesname>.+)[ \\._\\-]\n        (?P<seasonnumber>[0-9]{2})\n        (?P<episodenumber>[0-9]{2,3})\n        [\\._ -][^\\/]*$”,

“^(?P<seriesname>.+?)                  # Show name\n        [ \\._\\-]                                 # Padding\n        [Ee](?P<episodenumber>[0-9]+)            # E123\n        [\\._ -][^\\/]*$                          # More padding, then anything\n        ”

],

“filename_with_episode”: “%(seriesname)s – [%(seasonno)02dx%(episode)s] – %(episodename)s%(ext)s”,
“filename_with_episode_no_season”: “%(seriesname)s – [%(episode)s] – %(episodename)s%(ext)s”,
“filename_without_episode”: “%(seriesname)s – [%(seasonno)02dx%(episode)s]%(ext)s”,
“filename_without_episode_no_season”: “%(seriesname)s – [%(episode)s]%(ext)s”,
“input_filename_replacements”: [],
“language”: “en”,
“move_files_confirmation”: false,
“move_files_destination”: “/home/mjdescy/media/tv/%(seriesname)s/Season %(seasonnumber)02d”,
“move_files_enable”: true,
“move_files_fullpath_replacements”: [],
“multiep_join_name_with”: “, “,
“normalize_unicode_filenames”: false,
“output_filename_replacements”: [],
“recursive”: true,
“replace_invalid_characters_with”: “_”,
“search_all_languages”: true,
“select_first”: false,
“skip_file_on_error”: true,
“valid_extensions”: ["avi","mp4","m4v","wmv","mkv","mov"],
“verbose”: false,
“windows_safe_filenames”: true
}

Scheduling/Putting it All Together

I run TVNamer on my “completed torrents” folder every 15 minutes, via my personal account’s crontab. This process automatically sweeps any TV episodes out of Transmission‘s completed downloads folder and files them into my TV folder, which is shared via Samba and MediaTomb for use by my HTPC front-ends. This set-up is perfect for my WD TV LIVE front-end, XBMC, or for any HTPC front-end setup. For any TV shows that I rip from DVD and encode to MP4 files myself, TVNamer will rename and file them automatically, as long as I name the original files with a S01E01 (series number, episode number) syntax and drop them into the completed downloads folder.

$ crontab -e

Add the following line, being sure to modify the config file and TV folder paths, and save the crontab file.

*/15 * * * * /usr/local/bin/tvnamer -c /home/mjdescy/.tvnamerconfig.json /home/mjdescy/dl/complete >/dev/null 2>&1

When I read posts on LifeHacker and Gizmodo sometimes, I figure that everybody else in the world is downloading TV episodes via BitTorrent, and doing it using RSS feeds and an automated process. That can’t possibly be true, but I figured that if other people do this, I should be able to figure out how, too. It is pretty easy to find RSS feeds for various TV shows. Therfore, I thought it would be relatively simple to write a script to download episodes from each one. I figured that I would have to figure out how to sort out duplicate episodes and different quality types (720p, 1080p, etc.) somehow, too. It didn’t take me too long to find out that an excellent Python program called FlexGet already does all this. Plus, it runs from the command line, which is perfect for a headless server, like my Ubuntu Lucid Lynx box.

I don’t actually endorse doing this, by the way, but I was really curious to learn how it was done. I wanted to share what I learned, in case anyone was struggling with the configuration.

Installation

FlexGet’s web site has an excellent walkthrough that describes how to install it on various operating systems. My instructions in this section are not meant to improve on that guide, but merely to summarize it for an Ubuntu Server install.

FlexGet requires Python 2.6. If you don’t already have that installed on your server, try:

$ sudo apt-get install python2.6 python-setuptools

You have to download FlexGet from their web site. Choose the version for Python 2.6. I prefer to copy the URL and download the file via wget.

$ wget http://download.flexget.com/unstable/FlexGet-1.0r1377-py2.6.egg

After you download FlexGet, install it via Python’s easy_install script. You’ll have to specify the name of the FlexGet file you downloaded.

$ sudo easy_install <downloaded egg>

That’s it! Flexget is now installed in the /usr/local/bin directory.

If you run Transmission like I do, you should also install the the TransmisionRPC plugin.

$ sudo easy_install transmissionrpc

Configuration

Configuring FlexGet is deceptively simple. The configuration file is written in a format I had heard of, but never encountered before: YAML. YAML is simple and powerful, which means that it can be deceptively complex. You can write a config file that does the same thing in a million different ways. FlexGet’s Cookbook shows you a few examples to get you started. It took me a lot of experimentation to find a config file format that I liked. Here is what I ended up with. It’s probably more complex than it needs to be, but I like the idea of using presets to customize different types of downloads. Note that, in YAML, each level of indentation is two spaces (no tabs are allowed).

$ nano ~/.flexget/config.yml

presets:
tv:
series:
settings:
720p:
timeframe: 8 hours
720p:
- series title 1
- series title 2
- series title 3

transmissionrpc:
host: localhost
port: 9091
netrc: /home/mjdescy/.netrc
removewhendone: true
addpaused: no
ratio: 1.00

feeds:
TvTorrent1:
rss: http://tv.rss.url/
preset: tv

TvTorrent2:
rss: http://tv.rss.url/
preset: tv

This configuration file sets up a preset called “tv” which tells FlexGet to wait 8 hours before selecting episodes of any of the series listed under the 720p heading. 720p is the default quality setting for FlexGet, and is perfectly adequate for my needs. If FlexGet finds an episode of that quality, it will download it; if it finds episodes above that quality level, at 1080p for example, it will ignore them. If no 720p version exists after the “timeframe” of 8 hours is over, FlexGet will select the entry with the highest quality available. FlexGet won’t download the same episode more than once, unless there is a “proper” or “repack” version—a corrected version of a previously released episode—is available.

At the end of the “tv” preset, the “tranmissionrpc” heading instructs FlexGet to use the transmissionrpc plugin to directly send torrent URLs to transmission-daemon for download. This obviates the need to set a watch directory in transmission-daemon. Note that the “netrc” setting refers to a special username/password file that contains the user name and password required to log into transmission daemon. That file should be in the following format, hidden, in your home directory. Make sure you grant only your account access to read that file (i.e., chmod 600 ~/.netrc).

machine localhost
login username
password password123456789

If you do not run transmission-daemon, you could replace the “transmissionrpc” section with a “download” section, or use another plugin. Check out the flexget Cookbook for more information.

The “feeds” section lists all the TV torrent feeds to look in. I have each feed set to use the “tv” preset that I defined above.

Each RSS feed you list can contain a number of TV series (which is the easiest way to set it up), or just one series (you’ll have to find or build custom feeds for this). Flexget will search all the feeds you list for the series titles listed in the “tv” preset.

This configuration file works very well for me. I hope that it helps you get up and running.

Scheduling

FlexGet can be run manually for debugging purposes (flexget -v), but it should be scheduled to run periodically via cron. Note that there is no need to run FlexGet as root. I run it under my own user account. If you want to operate under the principle of least privilege, you could create a “flexget” account, edit its crontab, and grant that account the permissions it needs to get the job done. For my needs, that isn’t necessary. Therefore, to schedule FlexGet to run every hour, I simply edit my own crontab.

$ crontab -e

Add the following line to the crontab file, and save it.

0,30 * * * * /usr/local/bin/flexget –cron -c /home/mjdescy/.flexget/config.yml > /dev/null 2>&1

This cron job will run FlexGet at the top of every hour and half past the hour. FlexGet will read its options from the configuration file I specified (/home/mjdescy/.flexget/config.yml). I redirect output to /dev/null to prevent cron from sending me emails regarding the command’s output.

Left alone, these completed torrents will not be removed from the server, and will accumulate as you download more torrents over time. There is no automated way to remove these torrents. Also, there is no automated way to move files to a new folder after seeding is complete, only after downloading is complete.

My ultimate goals are to complete seeding up until my seeding ratio is hit, to move torrents out of the seeding folder once seeding is complete, and to use the TVNamer Python script to automatically move any TV episodes I have downloaded to my media server’s TV folder.

Because I want to use TVNamer to move and rename most of my downloads automatically, but I want them to be seeded up to the ratio I set, I cannot use TVNamer on my “seeding” folder. If I did so, there’s a big chance that files would be removed from my “seeding” folder before seeding was complete. To solve that, I decided to move all completed torrent downloads to another folder, “complete”, and set TVNamer to run on that folder. To move completed torrents, I created a Transmission queue management script, which performs the following actions every 15 minutes:

1. Check for completed downloads that have either been paused manually or have seeded until the seeding ratio.
2. Move the downloaded files to another folder (“complete”).
3. Remove the torrent from the Transmission server.

Removing Completed Torrents Using a Script

Here is the script, which is based on my previous version. Compared to my last script, this one’s largest difference is the elimination of the netrc argument in the transmission-remote calls. I noticed that I did not need to send credentials to the Transmission daemon, presumably because my user account has all the privileges it needs to run transmission-remote.

A commenter on my previous post of this script informed me that Transmission 2.x uses “Finished” instead of “Stopped” to indicate completed torrents. I incorporated that change into the script, so that it will work with the old or the new verbiage.

Be sure to change the value of the MOVEDIR to the path you wish to move the downloads to. If you set it to “%1”, you can specify the path as a command-line argument.

Script file: /home/mjdescy/bin/removecompletedtorrents

#!/bin/sh

# script to check for complete torrents in transmission folder, then stop and move them

# either hard-code the MOVEDIR variable here…
MOVEDIR=/home/mjdescy/media # the folder to move completed downloads to
# …or set MOVEDIR using the first command-line argument
# MOVEDIR=%1

# use transmission-remote to get torrent list from transmission-remote list
# use sed to delete first / last line of output, and remove leading spaces
# use cut to get first field from each line
TORRENTLIST=`transmission-remote –list | sed -e ’1d;$d;s/^ *//’ | cut –only-delimited –delimiter= ” ” –fields=1`

# for each torrent in the list
for TORRENTID in $TORRENTLIST
do
echo “* * * * * Operations on torrent ID $TORRENTID starting. * * * * *”

# check if torrent download is completed
DL_COMPLETED=`transmission-remote –torrent $TORRENTID –info | grep “Percent Done: 100%”`

# check torrent’s current state is “Stopped”, “Finished”, or “Idle”
STATE_STOPPED=`transmission-remote –torrent $TORRENTID –info | grep “State: Stopped\|Finished\|Idle”`

# if the torrent is “Stopped”, “Finished”, or “Idle” after downloading 100%…
if [ "$DL_COMPLETED" != "" ] && [ "$STATE_STOPPED" != "" ]; then
# move the files and remove the torrent from Transmission
echo “Torrent #$TORRENTID is completed.”
echo “Moving downloaded file(s) to $MOVEDIR.”
transmission-remote –torrent $TORRENTID –move $MOVEDIR
echo “Removing torrent from list.”
transmission-remote –torrent $TORRENTID –remove
else
echo “Torrent #$TORRENTID is not completed. Ignoring.”
fi

echo “* * * * * Operations on torrent ID $TORRENTID completed. * * * * *”

done

After creating the script file, be sure to make it executable.

$ sudo chmod u+x /home/mjdescy/bin/removecompletedtorrents

Scheduling the Script

Schedule execution of the script using cron.

$ crontab -e

I run the script every 10 minutes, with a 1-minute offset from the top of the hour. I redirect output to /dev/null so cron will not send me emails regarding the script’s output.

1,11,21,31,41,51 * * * * /home/mjdescy/bin/removecompletedtorrents > /dev/null 2>&1

This article explains how to install, configure, and run the Transmission daemon on a headless Ubuntu Lucid Lynx (10.04) server. Transmission, specifically transmission-daemon, is a BitTorrent client that can be accessed via a web browser and other front-ends, including a desktop app called trasmission-remote-gui and Android apps, including Transdroid (my favorite) and Torrent-Fu.

Installation

First, install the Transmission daemon using apt-get.

$ sudo apt-get install transmission-daemon

This installs the app and starts the daemon. The daemon runs under the user “debian-transmission”, which belongs to a group of the same name.

Preparation of File System and Group Permissions

I wanted all torrent files to be stored somewhere within my home folder (/home/mjdescy), which debian-transmission does not, by default, have access to. That means that I had to create folders for Transmission to use, and then grant debian-transmission read/write/execute access to them (execute is necessary to create folders), while ensuring that my user account (mjdescy) still had full control of the folders and their contents as well.

To do so, first create the folders for downloading and seeding torrents, completed torrents (i.e., seeding complete), and a watch folder to put new torrent files into. These three folders will be used by Transmission directly.
$ mkdir -p ~/dl/downloading # incomplete downloads
$ mkdir ~/dl/seeding # completed downloads/seeding torrents
$ mkdir ~/dl/watch

I also create a folder to move downloaded files to after seeding is complete. Transmission does not use this folder directly; I manipulate it via a cron job.
$ mkdir ~/dl/complete

Then, add your user account to the group debian-transmission. (I’m using my account name, mjdescy, in this example.) Now your user account and the debian-transmission user account are in the same group.

$ sudo usermod -a -G debian-transmission mjdescy

Next, change the group ownership of the folders Transmission will use to debian-transmission (the group, not the user).

$ chgrp debian-transmission ~/dl/downloading
$ chgrp debian-transmission ~/dl/seeding
$ chgrp debian-transmission ~/dl/watch

Finally, modify the permissions on each folder to allow full access (read, write, and execute) for both user (me) and group (debian-transmission, in this case).

$ chmod 770 ~/dl/downloading
$ chmod 770 ~/dl/seeding
$ chmod 770 ~/dl/watch

Doing so grants the debian-transmission group read/write access to that folder, preserves read/write/execute access for your user account, and prevents access to all others.

These changes also require me to “umask” setting in Transmission’s configuration file from the default of 18 to 2. That means transmission-daemon will create files with read/write permissions set at both the user and group level. Not changing the umask will make Transmission create files that your user account cannot delete or access completely.

Configuring Transmission

Configuring Transmission is a little unusual, compared to other daemons such as Samba, due to the daemon’s method of saving (overwriting) its configuration file on exit. Restarting the daemon, if it is already running, will result in it saving the configuration file, and overwriting any changes you might have made to it. Instead of restarting the service after editing the configuration file, you should reload the service. The following code sends a SIGHUP signal to transmission-daemon, which causes it to reload its configuration file. Importantly, it does not stop the the daemon, which would cause it to write over the configuration file.

$ sudo service transmission-daemon reload

After Transmission is installed, the daemon will be started automatically. Without changing the configuration, however, Transmission will not be accessible from other machines, which would pose a problem on a headless server. You must edit the configuration file using the command below, set the proper configuration options, then reload the daemon, to get Transmission going the way you want it to.

$ sudo nano /etc/transmission-daemon/settings.json

Here is my settings.json file, which I recommend that you use as a starting point. It configures Transmission to be available to clients other than localhost, to use separate folders for incomplete and complete (seeding) downloads and for the watch folder.

You have to edit the rpc-whitelist setting to connect to the server from anything other than the local host (and that’s nearly useless on a headless server). You should also edit the paths (download-dir, watch-dir, incomplete-dir) and peer ports as you see fit. Be sure to set the rpc-password, too!

{
"alt-speed-down": 500,
"alt-speed-enabled": true,
"alt-speed-time-begin": 480,
"alt-speed-time-day": 127,
"alt-speed-time-enabled": true,
"alt-speed-time-end": 0,
"alt-speed-up": 10,
"bind-address-ipv4": "0.0.0.0",
"bind-address-ipv6": "::",
"blocklist-enabled": false,
"dht-enabled": true,
"download-dir": "/home/mjdescy/dl/seeding",
"download-limit": 100,
"download-limit-enabled": 0,
"encryption": 2,
"incomplete-dir": "/home/mjdescy/dl/downloading",
"incomplete-dir-enabled": true,
"lazy-bitfield-enabled": true,
"max-peers-global": 200,
"message-level": 2,
"open-file-limit": 32,
"peer-limit-global": 240,
"peer-limit-per-torrent": 60,
"peer-port": 20500,
"peer-port-random-high": 20599,
"peer-port-random-low": 20500,
"peer-port-random-on-start": true,
"peer-socket-tos": 0,
"pex-enabled": true,
"port-forwarding-enabled": false,
"preallocation": 1,
"proxy": "",
"proxy-auth-enabled": false,
"proxy-auth-password": "",
"proxy-auth-username": "",
"proxy-enabled": false,
"proxy-port": 80,
"proxy-type": 0,
"ratio-limit": 0.2500,
"ratio-limit-enabled": true,
"rename-partial-files": true,
"rpc-authentication-required": true,
"rpc-bind-address": "0.0.0.0",
"rpc-enabled": true,
"rpc-password": “password”,
"rpc-port": 9091,
"rpc-username": "transmission",
"rpc-whitelist": "127.0.0.1,*.*.*.*",
"rpc-whitelist-enabled": true,
"speed-limit-down": 1500,
"speed-limit-down-enabled": true,
"speed-limit-up": 50,
"speed-limit-up-enabled": true,
"umask": 2,
"upload-limit": 100,
"upload-limit-enabled": 0,
"upload-slots-per-torrent": 4,
"watch-dir": "/home/mjdescy/dl/watch",
"watch-dir-enabled": true
}

After saving any changes, tell the daemon to reload its configuration file, which applies all the changes you have made.

$ sudo service transmission-daemon reload

Note that if you change the rpc-password to a plaintext value (which is the easiest way to do it), the password will remain unencrypted until transmission-daemon closes and it rewrites its settings file. To ensure that happens, and that the password is encrypted, restart (instead of reload) the service after you have reloaded it to apply the configuration changes. Once transmission-daemon saves its configuration file, the password will be converted from plaintext to a SHA1 encrypted format. You should only have to do this when you are changing the password.

$ sudo service transmission-daemon restart

That finishes configuration. Note that you will likely have to set up port forwarding on your router to access the web app from outside your network, and to seed properly.

Port Forwarding

Be sure to set up port forwarding on your router to expose the Transmission daemon to the outside world. You will have to forward the peer port range in order to upload properly. You can also forward the 9091 port (or whatever you set the RPC port to be) to your server. This is especially handy if you also run a dynamic DNS service, because then you can access your server from anywhere using a URL and a port number. I use Transdroid on my Android phone to check up on my torrents, or add them via an excellent search interface.

Web Access

To access the server, point your web browser to http://your.server.ip.address:9091. (9091 is the default port, which can be changed, under “rpc-port”, in the settings file.) The web interface is pretty simple. Note that you can adjust some settings, such as transfer speed and download directory, by clicking the gear on the lower right of the page. The open dialog supports URLs to torrent files, magnet links, and uploading a torrent file from your computer.

The Watch Folder

Simply move torrent files to the watch folder to start a download. If your watch folder is accessible via Samba share, SFTP, WebDAV, or what have you, you can add torrents to it while on the go, without a browser.

Install MediaTomb

MediaTomb is simple to install on Ubuntu Server because a Debian package for it is in the repositories. To install MediaTomb, simply install the mediatomb package.

$ sudo apt-get install mediatomb

The MediaTomb daemon will start automatically after installation.

Configure Port Number

By default, MediaTomb will pick the first available port starting with 49152. When restarting the server, it may pick a different port, such as 49153. I prefer to lock MediaTomb down to a certain point, so my bookmarks for the web interface are always correct, and so I can set firewall and networking rules consistently.

To lock MediaTomb to a single port, simply edit the XML configuration file.

$ sudo nano /etc/mediatomb/config.xml

Find the  <name> element in the configuration file. (You can rename its contents if you wish to. Personally, I don’t like the word “tomb” displayed on my TV, so I rename the service “MediaServer”.) Underneath the element, insert a
element, as follows:

<port>49152</port>

Save the configuration file, and restart the service.

$ sudo service mediatomb restart

Now MediaTomb will be locked down to port 49152. You can now set up your firewall to allow traffic on that port.

Add Media Files to MediaTomb’s Library

MediaTomb needs to be told where your media files reside in order to serve them. For simplicity, I prefer to keep all my media files in one folder tree, with high-level subfolders corresponding to media type. My media folder tree is within my home directory; your media may be spread out over your hard drive. MediaTomb doesn’t really care; you simply need to add each folder your media is in to the media library.

• media
  • movies
  • music
  • photos
  • tv

Add Media to MediaTomb

To add my media folder to MediaTomb, I use MediaTomb’s web interface. I point my browser to my server’s IP address, plus the port it is running on, as follows: http://serverip:49152.

MediaTomb’s sparse web user interface will appear. You will want to click on the Filesystem link, then browse the file system tree to find your media folder. Click on the media folder you wish to add, then click the icon that looks like a plus sign with two sync arrows, as in (+). This button calls up a form that allows you to add that folder to the media library, and set up automatic updating.

Set the scan mode to Inotify, the initial scan to Full, and check the Recursive checkbox. Then click Set, and you have added the folder. MediaTomb will scan the folder for media files to serve. Because we selected the Inotify scan mode, MediaTomb will monitor this folder tree for file changes, and update the media library accordingly. This allows you to add and remove media using Samba or the command line.

Repeat this process for each media folder on your server. Note that Inotify will not work with media served from remote file shares; you will need to set scanning to a timed interval.

Note: It is possible to add media via the command line, but it does not work well for the packaged Ubuntu install.

Connect to your Server

After setting up the server and adding media to it, you are essentially done. Your front-end device should automatically discover MediaTomb on the network.

Now I have a new wifi network (802.11n), and I can stream almost all content to the WD TV LIVE directly from the media server. Bandwidth and latency are a problem, however, with very large files, such as 1080p video and Blu-Ray rips. To handle this content, I still have to transfer it to local storage on the WD TV LIVE front-end.

Install Samba File System

The most transparent way to connect to a samba share it to mount it to a mount point on your filesystem. To do so, you need to install the Samba File System (smbfs).

$ sudo apt-get install smbfs

Mount the Samba Share

You mount the samba share like any other hard drive. First, create the mount point, which is simply a folder.

$ mkdir ~/wdtvlive

Next, mount the share to the mount point. Note that I’m using the host name, instead of the IP address. You can use the IP address, or configure your server to allow the use of host names. Note the use of options following the -o argument. Your security needs may differ.

$ sudo smbmount //WDTVLIVE/32GB_STICK ~/wdtvlive -o guest,rw

Now you can use the folder ~/wdtvlive to browse the fire share. The file share is now part of the file system, so utilities such as rsync will work with it as it it was a local folder.

You can unmount the share via the following command:

$ sudo smbumount ~/wdtvlive

Optional: Permanently Mount the Samba Share

To mount a Samba share permanently, you have to add a line to the server’s file system table. First, edit the FSTAB:

$ sudo nano /etc/fstab

Next, add a line, as follows. Note that the file system type is “cifs”, for “Common Internet File System”. The other options essentially set security wide open on the drive, which is acceptable for my use case, but certainly not for every one.

//WDTVLIVE/32GB_STICK /home/mjdescy/wdtvlive cifs guest,rw,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

Then, save and close the file. Finally, remount all drives.

$ sudo mount -a

Now your file share will be mounted in the file system, and it will be persistent between server reboots.