1000umbrellas.com

Michael Descy's Personal Website

The Problem

After about a year of enjoying how well my Ubuntu Linux home server downloads, organizes, and serves video files to my TV, I started to think harder about serving up music to devices around the house, and to my phone and/or PC while I’m at work. In the past, I’ve made my music library available on my home network via MediaTomb, minidlna, and Samba, but I never made use of it. Instead, I always used iTunes, which is good software, but very limiting.

In December, I realized two things:

  1. I almost never sync my iPods to my computer anymore, because they charge in my car or in my home stereo’s dock.
  2. I also never use my Android phone to listen to music, even though it is perfectly capable, because I never wanted to bother to get the USB cable out for my phone, and I haven’t found any software that I like to sync with.

My problem: syncing. I hate it. It is passé and worthless when everything is network connected and your music collection is vastly larger than what a reasonably-priced amount of flash memory will hold.

Around the same time, I received a Cr-48 from Google. It runs Chrome OS, which can play audio via the browser (Flash or HTML 5), but that’s it. I thought it would be an interesting challenge to find a good way to play my music library on it. I also wanted to store my library on my power-efficient Ubuntu Linux server, so I wouldn’t have to leave my power-hungry desktop on all day.

Searching for a Solution

I found two highly respected options that would allow me to stream music to a browser or other device: Ampache and Subsonic. Ampache is built on top of the LAMP stack, so it requires Apache, MySQL, and PHP, which a ton of Linux servers have installed on them. It has a good number of features, too, including an Android client and front-end support in Amarok and WinAmp. I ended up not liking it, however, for two main reasons: The system requirements are rather high, mostly because it depends on MySQL as a data store. Also, indexing my 80 GB music library was very slow.

What I ended up preferring, and falling in love with, was Subsonic. Subsonic is a Java app, which you can install either as a standalone Debian package in Ubuntu, or as a WAR file that you can run under TomCat or another Java Server. It runs on any host OS that supports Java, so Windows and Mac users can run it too. Subsonic scanned my music library surprisingly quickly, looks great, and worked right away. I have it installed under TomCat, and configured to use SSL (for security and to disguise my music downloading/streaming from network administrators).

Subsonic

Subsonic

Subsonic

Subsonic offers a somewhat complex, but attractive and skinnable, web interface. It offers playlist support (though not smart playlists as in iTunes), random playlists, search, and support for album art and metadata (including a tag editor). It is far more album-oriented than song oriented, which took some getting used to moving from iTunes. For example, albums can be rated, but not individual songs. Also, the home page on the interface defaults to random albums, which looks nice and pretty, but differs greatly from how I use iTunes. I really like the feature, though, because it allows me to rediscover music that has long been buried by my iTunes smart playlists.

Playback in the browser is done via an open-source Flash plugin, which works fine. Album art must be embedded in the files, or at least stored at the folder level, which was initially a challenge for me due to the way iTues handles album art in its library. I used MediaMonkey to help get my album art into shape. The server can be set up to transcode music in formats such as Flac to MP3 or AAC to support the client player’s capabilities and to limit bandwidth.

Subsonic doesn’t really stream your music; it downloads and caches it. The browser client caches the song you are playing, while the iOS, Android, and Windows Phone 7 clients will cache a number of songs ahead of the current one. Caching is way better than streaming, because you don’t need a constant connection to the server. It works really well. The server has a “Status” window in which you can review all the connected clients and the bandwidth they are using, which is pretty cool.

Subsonic is free and open source software, though it does have a somewhat controversial donation requirement if you wish to use it via a phone app. You can connect to it via a web browser indefinitely, but if you use the Android app, as I do, it will tell you that you must donate to the project/developer after 30 days, or it will refuse to access the server. After seeing how well the Android app worked, I donated right away. Remember, even in the open source world, it is a good idea to support good developers.

By creating multiple user accounts, you can also use Subsonic to share music with friends and family. If you configure the server with SSL, you can do so securely and privately as well. Users can be given access to download and upload music, so it could become a music sharing platform for you and your close friends that flies under the radar.

All in all, I love Subsonic and recommend it to anyone who has a large music library that they wish to bring with them and share with close friends.

Now that 2010 is completely over, I thought it would be fun to look back on the gadgets that meant the most to me over the past year. I’m leaving off the Cr-48 I received last week from Google (too new) and my Ubuntu server, which I rebuilt into a very useful media center backend (too old). I’d consider those gadgets honorable mentions this year.

Motorola Droid

Even though this phone was almost immediately outclassed by the Nexus One and replaced by the Droid 2 after only 6 months, my Droid remains the gadget I use more often and find most useful throughout the day. That it is more useful than the day I bought it is a testament to the Android platform’s growth and the persistence of the open source custom ROM development community that has sprung up around it. Here is a list of my most-used apps.

  • Communication: Google Voice, GMail, Touchdown (Exchange for Android), Facebook, Twidroid Pro, eBuddy
  • Productivity: VitaDroid, Calendar Pad Pro, EverNote
  • Entertainment: NewsRob, Amazon Kindle, TuneIn Radio, Subsonic, Pandora, Act1 Video Player, NFL Network
  • Utilities: USB Tether (a Froyo feature), Gentle Alarm, Lightning Bug
  • Games: Angry Birds, Robo Defense
  • Travel: TripIt, FlightTrack Pro
  • ROMs: CyanogenMod 5 and 6, Sapphire 1

Amazon Kindle (2nd Generation)

I love my Kindle, and prefer to read books on it than on paper now. Part of the appeal is that it is a gadget, but, even more importantly, the Kindle is lighter and smaller than most books, and takes up a finite amount of space as my library grows. The only downside to the Kindle is that its contrast is not great, and screen glare is a problem in mediocre-light conditions. (Oddly, glare is rarely a problem outside or in bright light.)

Boxee Box

I love my Boxee Box. Even though my Amazon.com review, the #1 most helpful for the product, is less than glowing, I love the Boxee Box’s handling of local media, and wait eagerly for it to get apps for Netflix and, to a lesser extent, Hulu Plus.

A Christmas Present from Google

Last night, when opening boxes from the stack that UPS has shipped to me in the past couple days—Christmas presents, all—I found an unexpected Christmas present to me: the Cr-48 netbook from Google. The Cr-48 Chrome Notebook is Google’s reference hardware for Chrome OS, and they selected me for testing. (Truthfully, I applied and kind of begged them for the opportunity.) I’m going to give a shot at making this my main computer for the next two weeks.

Ready for a Web-only Future?

Getting free hardware is pretty cool, and guarantees a lot of goodwill from users like me, but I am not entirely certain that the world is ready for a web-only OS just yet. Business users with web- or Citrix-based apps, who are expensing the Verizon Wireless network charges while they are on the road, appear to be the intended market. A lot of business people watch videos on netbooks on their flights home–I know, because I was a road warrior myself until recently—and this is one ability that this Chrome OS, with no hard drive or USB drive support, notebook appears to lack.

The Cr-48 is not a production product, so it isn’t too fair to review or critique the hardware too much. What I will say is that it runs a little slow, it really does make sense to have a search button, and separate button the touchpad are deservedly passé as well.

Transitioning to a web-only device is an interesting challenge. For the most part, in anticipation for a Chrome OS-type future, I have found web-based replacements for my most frequent computing activities. As someone who has bought into the Google infrastructure pretty fully since buying an Android phone last year, it is no surprise that I use Google services for a lot of these needs.

My Mapping of Computing Needs to Web-based Services

Here is my plan to move what I do away from the desktop and on to web applications. I will record how things go.

  • Email > Gmail
  • Calendar > Google Calendar
  • RSS Reader > Google Reader
  • Instant messaging > Google Talk plugin
  • Twitter/Facebook client > TweetDeck app for Chrome (though their flagship Adobe Air product has more features, including the ability to use it with Identica)
  • Note taking with Evernote > Web-based version of Evernote (The web version of Evernote is a lot slower than the desktop version, but it has a robust feature-set.)
  • Music/podcast playback with iTunes > Subsonic music server, and, to a lesser extent, Pandora
  • Video playback with VLC Player/MPlayer/Boxee > YouTube? (I have to figure out a way to watch my own content on my home network. I plan to install a web server on my machine and try to browse for and play back videos through the browser.)
  • Office document manipulation using MS Office or OpenOffice.org > Google Docs (I use Office all day at work, but not much in my home life. I tend to write everything in Evernote or Gmail these days. I don’t need advanced Excel functionality at home, though it is crucial for me at work.)
  • Photo manipulation and library management with Aperture/Lightroom > Gallery 3.0 and Picasa , online (I see online photo services as pretty weak compared to the pro-sumer tools I have been using for the past couple years. Also, you can’t plug your camera or SD card into the Cr-48; I think Google expects you to take all your photos from your smartphone.)
  • Printing > Google Cloud Print (though it currently requires me to leave a Windows PC running to print)
  • Scanning > I don’t know what I’m going to do. Scanning is very important for me, and, I think, for their target market of road-warrior business people. As a consultant, I had to scan receipts for expense reports every two weeks.
  • Encrypted password storage with KeePass > LastPass (It would be nice if the “binary” version of the LastPass, which can log you out after a time limit expires, was in the Chrome app store, though.)
  • An SSH client, such as Putty, Apple’s SSH client, or OpenSSH, for remote Linux administration > Chrome OS has a command line (hit Control-Alt-T) with a built-in SSH client, which appears to work just fine. It doesn’t appear to support key-based authentication yet, though.
  • A BitTorrent client > I have Transmission-Daemon running on my Linux home server, so I am covered here. I can’t do anything with what I download just yet, though.
  • Access to Samba shares on my home network > I am not sure what to do about this one. I will experiment with adding a web server to my home server and sharing the files via HTTPS and WebDAV.

I will let you, and Google, know how it goes.

IBM just published a great article that describes how to manage Linux file permissions and ownership. That’s a very important topic for managing any server, and is very important in my Transmission-Daemon setup tutorial. This article is one of the most complete and well-written I have come across. Its command line examples are based on Fedora, but will work just fine in Ubuntu, too.

Learn Linux, 101: Manage file permissions and ownership [IBM]

Sick Beard is a (terribly, terribly named) application that downloads TV episodes from newsgroups—as opposed to BitTorrent—moves and renames them, and even fetches artwork and episode information (.NFO files) for use with XBMC. If, for any reason, you want to avoid BitTorrent and go with newsgroups instead, go aheard and install SABnzb+ for downloading purposes, and Sick Beard to get TV content and metadata, and move/organize it how you want it.

Here is a link to the tutorial. Note that it is epic in length, not written for server installs, and—due to SABnzb+’s processing requirements—likely will crawl on underpowered hardware.

Sick Beard install instructions. [http://www.ainer.org]

I received the D-Link Boxee Box yesterday, tested it out, and posted a thorough review on Amazon.com.

I like the product a lot; it completely blows the WD TV LIVE I have out of the water in all areas, and it offers a lot of good content that I otherwise not know anything about. Even the social aspects of the product are pretty smart. Still, I think that their choice to launch with one interface and have a new, shiny, different, and in-some-ways-inferior one appear as soon as you update the firmware was a misstep. That forms the basis of my review.

What’s great is that a Boxee employee has been reading the review (at least the first few dozen or so) and gave me his email address. I wrote him a long email about how I would improve the user interface. I hope that my ideas get incorporated. It is totally cool, however, to be able to talk back.

I hope Boxee continues to update the software and push it to the Boxee Box.

Even though my Ubuntu Lucid Lynx media server runs pretty solidly, I like to monitor it periodically using SSH and command line tools. Some of my favorites are w, htop, ps, bwm-ng, and nmap. After rebuilding my server, I decided to install some lightweight system monitoring tools that exposed themselves via a web interface, so I could check up on my server more quickly. The FLOSS utility Monit fit the bill for point-in-time monitoring.

Monit is a utility that you can set up to monitor your server’s processes and files, alert you to problems, restart stopped services, and present a web interface to display the server’s current status. I suspect most administrators would configure it to email them when services are down, drives are nearly full, load is too high, and so on. I don’t bother with email alerts, but I really like Monit’s web front-end and service restarting capabilities.

Monit system monitoring (yes, my server is redlining!)

Install Monit

Installing Monit on an Ubuntu server is accomplished in the usual way.

$ sudo apt-get install monit

Configure Monit

You have to edit a configuration file to start monit on your server.

$ sudo nano /etc/defaults/monit

Default configuration file

# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Stefan Alfredsson

# You must set this variable to for monit to start
startup=0

# To change the intervals which monit should run,
# edit the configuration file /etc/monit/monitrc
# It can no longer be configured here.

You have to change “startup=0″ to “startup=1″.

Edit the Control File

Monit’s control file is the place to set up system, service, and file monitoring. The default control file is all commented out. When I set mine up, I uncommented various lines and inserted sections for each service at the bottom of the file.

$ sudo nano /etc/monit/monitrc

Edit the file (uncomment lines), to enable the following options, which set up a monitoring daemon and enables the web interface (http://localhost:2812).

set daemon  120

set logfile syslog facility log_daemon

set httpd port 2812
allow localhost
allow 192.168.1.0/255.255.255.0

check system localhost

if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 75% then alert
if cpu usage (user) > 70% then alert
if cpu usage (system) > 30% then alert
if cpu usage (wait) > 20% then alert

check filesystem root with path /

Add the following sections to the bottom of the file to monitor cron, sshd, samba, mediatomb, lighttpd, and MySQL (which I am running to sync XMBC across multiple PCs). Services that run with a pid file are simple to monitor. MySQL does not run with a pidfile, but can easily be configured to by adding the line "pid-file = /var/run/mysqld/mysqld.pid" to the [mysqlid] section of /etc/mysql/my.cnf.
# cron check process cron with pidfile /var/run/crond.pid group system start program = "/etc/init.d/cron start" stop  program = "/etc/init.d/cron stop" if 5 restarts within 5 cycles then timeout depends on cron_rc check file cron_rc with path /etc/init.d/cron group system if failed checksum then unmonitor if failed permission 755 then unmonitor if failed uid root then unmonitor if failed gid root then unmonitor # sshd check process sshd with pidfile /var/run/sshd.pid start program  "/etc/init.d/ssh start" stop program  "/etc/init.d/ssh stop" if failed port 22 protocol ssh then restart if 5 restarts within 5 cycles then timeout #  samba check process smbd with pidfile /var/run/samba/smbd.pid group samba start program = "/etc/init.d/smbd start" stop  program = "/etc/init.d/smbd stop" if failed host 192.168.1.3 port 139 type TCP  then restart if 5 restarts within 5 cycles then timeout # mediatomb check process mediatomb with pidfile /var/run/mediatomb.pid group mediatomb start program = "/etc/init.d/mediatomb start" stop  program = "/etc/init.d/mediatomb stop" if 5 restarts within 5 cycles then timeout # lighttpd check process lighttpd with pidfile /var/run/lighttpd.pid group lighttpd start program = "/etc/init.d/lighttpd start" stop program = "/etc/init.d/lighttpd stop" if failed host 127.0.0.1 port 80 protocol http then restart if 5 restarts within 5 cycles then timeout # mysql # note: requires mysql to run with a pid. check process mysql with pidfile /var/run/mysqld/mysqld.pid group database start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed host 127.0.0.1 port 3306 then restart if 5 restarts within 5 cycles then timeout

Unfortunately, monit can’t monitor the transmission-daemon service, because transmission-daemon does not run with a pid file, and forks itself sometimes. There is a warning in the transmission-daemon launch script, /etc/init.d/transmission-daemon, which says that pid file support will not work, because the daemon forks. Hopefully this will be resolved in the future.

Using Monit

After editing your configuration file, restart monit.

$ sudo service monit restart

Now you can check the health of your server by pointing a web browser to your server’s IP address, port 2812 (http://server:2812/).

This is a simple script I wrote to perform one-way sync to mirror my server’s media folder on my WD TV LIVE‘s external hard drive. I have to run such a script because the wifi-N signal in my TV cabinet is not sufficient to stream HD video, and the WD TV LIVE doesn’t appear to buffer enough of the video to ensure smooth playback. (I even ran into problems running Boxee on my PC, when it was in the cabinet. And yes, no hardwired solution is possible in that room; even powerline networking wasn’t up to snuff.) Therefore, instead of relying on software to cache video streams, I put an old, external hard drive into use to cache everything on my media server. This is not an elegant solution, but it is an effective one.

The WD TV LIVE’s external hard drive is NTFS-formatted, and is mounted to my media server’s filesystem via smbmount.

The “syncmedia” Script

This is my script, which I store in /home/mjdescy/bin/syncmedia.

#!/bin/sh
# mirror media folder to server

SOURCE_DIR=/home/mjdescy/media/
DEST_DIR=/home/mjdescy/wdtvlive/

# perform simple, one-way sync using rsync; use setlock to prevent simultaneous instances from being run
/usr/bin/setlock -n /tmp/.mediasync.lock /usr/bin/rsync --size-only --verbose --recursive --stats --delete --exclude=".*" --modify-window=1 --times --bwlimit=1000 $SOURCE_DIR $DEST_DIR

Execute Permissions

Make sure to set execute permissions on the script before you try to run it.

$ chmod u+x /home/mjdescy/bin/syncmedia

Notes on “setlock”

Note the use of “setlock” (which is part of the “daemontools” package) to prevent the rsync command from being run if it is already running. This is necessary if the time it takes to copy the files can exceed the interval at which the script is executed via cron.

Install daemontools in Ubuntu via:

$ sudo apt-get install daemontools

Rsync Options

Using rsync to copy the files to a Samba share, mounted with smbfs, presented some challenges. Because the server and the share are on two different file systems, copying permissions would either not work or produce unexpected results. Therefore, I couldn’t use the rsync “archive” option, as I typically do. Therefore, I selected the options “modifiy-windows” and “times” to help prevent the same files from being copied again when there wasn’t really a change.

I used the “size-only” option because my media files aren’t edited and updated like office documents; they are either downloaded or created once via Handbrake.

I used the “exclude” option to avoid deleting the “.wd_tv” folder that the WD TV LIVE uses to store its library information on the external hard drive.

The “delete” option will ensure that deletions on my media server (the master) are replicated on the WD TV LIVE (the slave).

I used the “bwlimit” option to avoid taking all the upload bandwidth away from my server for the sync. This is entirely optional. After the initial data move, this process should run primarily during the day, when we are not watching the TV, so slow uploads are not a big problem.

Sheduling

I use my account’s crontab to kick off this script every hour.

$ crontab -e

Add the following line and save the crontab.

@hourly /home/mjdescy/bin/syncmedia > /dev/null 2>&1

Now the process will be kicked off at the top of every hour. Rsync will not move files if there have been no changes to the source folder, and will move only new/changed files to the destination folder. If an instance of this process is already running, another one will not be created.

Note: These instructions are an update of a previous post. I updated my instructions to include the download of the transmissionrpc plugin, and to include the cron job definition I use. These instructions apply to FlexGet 1.0r1377.

When I read posts on LifeHacker and Gizmodo sometimes, I figure that everybody else in the world is downloading TV episodes via BitTorrent, and doing it using RSS feeds and an automated process. That can’t possibly be true, but I figured that if other people do this, I should be able to figure out how, too. It is pretty easy to find RSS feeds for various TV shows. Therfore, I thought it would be relatively simple to write a script to download episodes from each one. I figured that I would have to figure out how to sort out duplicate episodes and different quality types (720p, 1080p, etc.) somehow, too. It didn’t take me too long to find out that an excellent Python program called FlexGet already does all this. Plus, it runs from the command line, which is perfect for a headless server, like my Ubuntu Lucid Lynx box.

I don’t actually endorse doing this, by the way, but I was really curious to learn how it was done. I wanted to share what I learned, in case anyone was struggling with the configuration.

Installation

FlexGet’s web site has an excellent walkthrough that describes how to install it on various operating systems. My instructions in this section are not meant to improve on that guide, but merely to summarize it for an Ubuntu Server install.

FlexGet requires Python 2.6. If you don’t already have that installed on your server, try:

$ sudo apt-get install python2.6 python-setuptools

You have to download FlexGet from their web site. Choose the version for Python 2.6. I prefer to copy the URL and download the file via wget.

$ wget http://download.flexget.com/unstable/FlexGet-1.0r1377-py2.6.egg

After you download FlexGet, install it via Python’s easy_install script. You’ll have to specify the name of the FlexGet file you downloaded.

$ sudo easy_install <downloaded egg>

That’s it! Flexget is now installed in the /usr/local/bin directory.

If you run Transmission like I do, you should also install the the TransmisionRPC plugin.

$ sudo easy_install transmissionrpc

Configuration

Configuring FlexGet is deceptively simple. The configuration file is written in a format I had heard of, but never encountered before: YAML. YAML is simple and powerful, which means that it can be deceptively complex. You can write a config file that does the same thing in a million different ways. FlexGet’s Cookbook shows you a few examples to get you started. It took me a lot of experimentation to find a config file format that I liked. Here is what I ended up with. It’s probably more complex than it needs to be, but I like the idea of using presets to customize different types of downloads. Note that, in YAML, each level of indentation is two spaces (no tabs are allowed).

$ nano ~/.flexget/config.yml

presets:
tv:
series:
settings:
720p:
timeframe: 8 hours
720p:
- series title 1
- series title 2
- series title 3

transmissionrpc:
host: localhost
port: 9091
netrc: /home/mjdescy/.netrc
removewhendone: true
addpaused: no
ratio: 1.00

feeds:
TvTorrent1:
rss: http://tv.rss.url/
preset: tv

TvTorrent2:
rss: http://tv.rss.url/
preset: tv

This configuration file sets up a preset called “tv” which tells FlexGet to wait 8 hours before selecting episodes of any of the series listed under the 720p heading. 720p is the default quality setting for FlexGet, and is perfectly adequate for my needs. If FlexGet finds an episode of that quality, it will download it; if it finds episodes above that quality level, at 1080p for example, it will ignore them. If no 720p version exists after the “timeframe” of 8 hours is over, FlexGet will select the entry with the highest quality available. FlexGet won’t download the same episode more than once, unless there is a “proper” or “repack” version—a corrected version of a previously released episode—is available.

At the end of the “tv” preset, the “tranmissionrpc” heading instructs FlexGet to use the transmissionrpc plugin to directly send torrent URLs to transmission-daemon for download. This obviates the need to set a watch directory in transmission-daemon. Note that the “netrc” setting refers to a special username/password file that contains the user name and password required to log into transmission daemon. That file should be in the following format, hidden, in your home directory. Make sure you grant only your account access to read that file (i.e., chmod 600 ~/.netrc).

machine localhost
login username
password password123456789

If you do not run transmission-daemon, you could replace the “transmissionrpc” section with a “download” section, or use another plugin. Check out the flexget Cookbook for more information.

The “feeds” section lists all the TV torrent feeds to look in. I have each feed set to use the “tv” preset that I defined above.

Each RSS feed you list can contain a number of TV series (which is the easiest way to set it up), or just one series (you’ll have to find or build custom feeds for this). Flexget will search all the feeds you list for the series titles listed in the “tv” preset.

This configuration file works very well for me. I hope that it helps you get up and running.

Scheduling

FlexGet can be run manually for debugging purposes (flexget -v), but it should be scheduled to run periodically via cron. Note that there is no need to run FlexGet as root. I run it under my own user account. If you want to operate under the principle of least privilege, you could create a “flexget” account, edit its crontab, and grant that account the permissions it needs to get the job done. For my needs, that isn’t necessary. Therefore, to schedule FlexGet to run every hour, I simply edit my own crontab.

$ crontab -e

Add the following line to the crontab file, and save it.

0,30 * * * * /usr/local/bin/flexget –cron -c /home/mjdescy/.flexget/config.yml > /dev/null 2>&1

This cron job will run FlexGet at the top of every hour and half past the hour. FlexGet will read its options from the configuration file I specified (/home/mjdescy/.flexget/config.yml). I redirect output to /dev/null to prevent cron from sending me emails regarding the command’s output.