Alex Wetmore is always busy with something…

Hello regular readers. You’ve probably noticed that I haven’t posted in a long time. I’m now one month into a three month vacation, and part of my plan is to get the blog up to date with both sailing and non-sailing activities.

This is going to be a boring post for most of you, but it’s going to really help someone down the line. The next blog post will be about a big sailing trip that I just returned from. There will be a couple of bike and framebuilding posts after that related to some delayed projects.

Raymarine makes a popular line of electronics for boats. I have a Raymarine e7D chart plotter (fancy name for a GPS) on my sailboat. I just finished a great 4 week sailing trip and thought it would be nice to get the route that we took as an online map. It turns out that Raymarine can only save their routes in an undocumented file format called FSH. A few nice guys have reverse engineered it, but it is hard to find their tools online.

I wanted to document the steps to go from having data on your Raymarine GPS to showing it on a Google Map.  All software involved is free, so you don’t need to spend $50 on Raymarine’s Vogage Planner.   The software is also cross platform and will work on a Mac running OS X, or a PC running Windows or Unix.  This is all command line stuff though, so it’s not very user friendly.

Tools needed:

• parsefsh – This tool can read the proprietary FSH file format and convert it the OSM format.  Documentation is on a wiki.
• gpsbabel – This can convert the OSM format to GPX

Steps:

• Export your tracks from the Raymarine MFD to a SD card.  This is done under the My Data menu.
• On the SD card you will find a single file called “archive.fsh”.  This will have all of your tracks.
• Run “parsefsh < archive.fsh > archive.osm” to convert the FSH to a OSM file.  This also converts the tracks to routes, but we’ll convert them back in the next step.
• Run “gpsbabel -i osm -f archive.osm -x transform,trk=rte -o gpx -F archive.gpx”.  This produces archive.GPX which contains waypoints, routes, and tracks for everything that was in the FSH.
• You can now edit the GPX with your normal tools (I used Garmin Basecamp)

And finally, a link to a map that I generated this way.  Sadly the FSH format doesn’t contain point by point times, so you don’t get speeds and can’t where where I was on which day.

If I were buying a new boat GPS I’d give serious consideration to a Garmin instead of a Raymarine.  There is no reason for a manufacturer in 2013 to be using a proprietary format for GPS track data.

Most blog readers probably know that I host/run bikelist.org, phred.org, and many of the bicycle-related mailing lists associated with them such as Internet-BOB, Framebuilders, and Touring. Many years ago I wrote a crude system for searching the archives. It is based on Index Server and IIS and has many limitations such as not knowing what a message thread is, being difficult to backup, and frequently breaking. It was time to rewrite it.

About 3 years ago I started writing a new archival system based on SQL Server and their full text indexing system. I’d work on the new archives for one or two weeks worth of bus rides, then put the project away and ignore it for a year. Amazingly I did make actual progress working this way. For the past two weeks I’ve been putting the final touches on the system. You can see it live at http://search.bikelist.org/beta.

The goals of the new system are:

• Having agood schema that can properly represent discussions instead of just single messages.A thread is all displayed as a single page to make it easier to get context.
• Using an easier platform to innovate on. With the new system it won’t be hard to add features such as posting through the web interface (that isn’t there yet, but likely will be in the next year or so).
• Having a much nicer interface.

Ultimately the goal is to have a clean, simple, fast system for finding all of the great content that has been posted to these lists in the last 5-15 years. I still have a small amount of work to do completely replacing the old system, but I wanted to setup a preview so that people could see how it looks and provide feedback. The internet-BOB and framebuilders lists are being loaded into the system today,so there is a lot of content to explore. Please leave comments here or send me email telling me what you think.

A fun side effect of rewriting the archives is that I often get side tracked by all of the interesting old messages out there.

Currently phred.org/bikelist.org are run off of two servers, one running FreeBSD and one running Windows 2003 and Exchange 2003. We also have a file server for our house, so that is 3 computers powered on 24×7. The total power consumption for all three machines is around 400 watts, which costs us about $25/mo in electricity (“clean” hydro-powered electricity, but that is still a lot of energy).

This month I’m replacing all of those physical servers with a single PC. Using VMWare Server I canrun the operating systems from those three machines at the same time on a single computer. They’ll share a single set of disks, memory, and CPU. This one box will use about the same amount of power as any one of the previous machines.

It’s going okay so far. Last night I spent a lot of time trying to move the FreeBSD machine (phred.org) over to VMWare. This process is called P2V (physical to virtual). There are very good tools for doing this with Windows-based computers, but not much for FreeBSD. Here is what worked for me:

• Make a backup of the FreeBSD box.
• Get vdkand dd for windows.
• Use vmware-vdiskmanager to create a virtual disk that is at least as large as the disk in the FreeBSD machine.
• Move the FreeBSD machine’s drives to a Windows box with those tools. In my case this couldn’t be the machine that I run VMWare on because that machine runs 64-bit Windows and my FreeBSD drives use an old 3ware RAID controller which isn’t supported in 64-bit Windows. I used an old PC that is being retired.
• Mount the disk image created with vmware-vdiskmanager using vdk.
• Use dd to copy from the physical volume that contains the FreeBSD bits to the VMWare disk.
• Dismount the disk using vdk.
• Copy the .vmdk file to your VMWare box and mount it in a new VMWare image.

Hopefully someone finds this via a search engine and it is helpful to them.

If you see that phred.org or bikelist.org lists are down for periods of time in the next week or two this is why. I still have one more machine to move over.

I’m excited for the end results. The machines take up a good amount of space in the basement and a single machine is much easier to hide in a corner. Power consumption will be lower. It’ll be easier for me to build out test servers to experiment with. It makes backup and upgrades easier.

The old computers (some of which are only a couple of years old…all will run Windows XP and Office) are going to Global Technology Academy, a non-profit based out of Garfield High School. GTA sends local high school students around the world to setup computer labs in schools in developing countries. I’m excited about this non-profit because it gives Seattle students exposure to developing countries and gets the technology installed in useful ways instead of just dumping it.

Part one of this series showed how I ripped a large pile of CDs into FLAC image files. The FLAC image files are playable, but most of our hardware wants MP3 files and I’d prefer one file per track instead of one file per album. The FLAC image files also have crummy tags from CDDB so there will be a lot of typos in artist names and track titles.

A friend and I started tackling this problem a few years ago and wrote a pair of Python scripts called tag.py and transcode.py that converted FLAC images intoMP3s. When we wrote these scripts FLAC images were a sort of new concept and we came up with our own unique way of tagging them that no one else used. Tag.py read an EAC-generated CUEsheet (including CDDB tags) and enter them into the FLAC image. We could (painfully)hand edit the tags in the FLAC image after this step. Transcode.py read the FLAC image, including the tags embedded by tag.py and generated MP3s (or any other format that we wanted, including per-track FLAC files).

The process worked, but it was clumsy. We also still relied on CDDB tags for our music and then a lot of hand editting to fix them.

Since writing those scripts a CDDB replacement called MusicBrainz has become mature. MusicBrainz is a much stricter tag database where all entries are reviewed and there are tight relationships between artists, releases (albums) and tracks. This tight relationship means that each artist only has one name, you won’t have problems with one CD being tagged “The Beatles” while another is tagged “Beatles” and a third is tagged “Beatles, The”.

We talked about our old scripts and came up with a better system:

• tag.py would find the CD in the MusicBrainz database and get the MusicBrainz ID for it. This is a unique ID that identifies that album.
• tag.py would embed the MusicBrainzID into the FLAC image.
• transcode.py would read the MusicBrainzID, get the track metadata from MusicBrainz, and encode our MP3s.

Once you’ve run tag.py you can regenerate MP3s (or any other type of music file) just by rerunning transcode.py. This future proofs our music.

I rewrote tag.py to have a small GUI. When opening a FLAC file tag.py first computes the DiscId (a mostly-unique identifier used to find the disc in the MusicBrainz database) and sends this to MusicBrainz. If the DiscId isn’t found then it searches based on the artist and title that we already have in the FLAC image from CDDB. The GUI lets you see each of these matches and pick the right one. If you don’t find the right one then you can search on your own and get the MusicBrainzID and paste that into the GUI. Once you’ve found the right tags for this album you hit save and it writes the ID out the FLAC.

Note that my day job is writing server software, not GUIs, and it shows in the ugliness of this tool. Also,even though this has a few buttons along the bottom it is really designed to be keyboard driven. A lot of output is written to the console that you started the tool from. Consider it a half-GUI/half command line tool.

You start the tagger by running tag.py and passing it some filename globs. For instance “tag.py *.flac” will have it work on all of the FLAC files in the current directory.

The screen is divided into three sections. The top section shows you the important data from the FLAC image (filename,CDDB artist/title, DiscId, embedded MusicBrainz ID and number of tracks). The fields used for a MusicBrainz search will show up in red. The second section show you the results for a current MusicBrainz query. You can enter a new MusicBrainz ID if you find something better on their search page. The last section has buttons that map to some of the key presses and a status bar that tells you what it is doing.

Here are the keys that matter:

• Control-N and Control-P — Go to the next and previous FLAC file
• Alt-N and Alt-P — Go to the next and previous hit from a MusicBrainz search
• Alt-S — Save the current MusicBrainz ID to the current FLAC
• Alt-L — Reload MusicBrainz matches (this searches by MusicBrainz ID, Disc ID, and CDDB artist and title)
• Control-Shift-N — Find the first FLAC without a MusicBrainzID
• Alt-Q — quit

With this tool I can find the right tags for our FLAC images from MusicBrainz at a rate of about 100 CDs every 20 minutes. This includes entering releases into MusicBrainz for CDs that we own, but which they don’t have in their database.

Once tag.py has been run on a bunch of FLAC files you just run transcode.py and walk away. This will transcode each of your FLAC images into seperate MP3 files and put them in the right directory. The exact method for doing this is controlled by the file transcode.cfg. Here is my version of the file:

[GeneralConfig]
Flac: d:/util/bin/flac.exe
Metaflac: d:/util/bin/metaflac.exe
Encoders: mp3
[mp3]
Directory: f:/music-rerip/mp3/New/$P/$T
Filename: %D%n-$t.mp3
FilenameVA: %D%n-$t($p).mp3
Command: d:/util/bin/lame.exe –alt-preset standard –id3v2-only –tt %t –ta %P–tl %T –ty %Y –tn %n –tg Rock %f %F
CommandVA: d:/util/bin/lame.exe –alt-preset standard –id3v2-only –tt %t –ta %p –tl %T –ty %Y –tn %n –tg Rock %f %F

This tells transcode.py that we are going to use one encoder and that it will make mp3 files. The files will go into d:musicmp3new and then be listed under performer and release title. If the album is a compilation (coming from multiple artists) then the second command listed is used to encode them, otherwise the first one is. If we wanted two different qualities of MP3 (high for home use,crappy for portable device use) you could just make another config that looks like this:

[GeneralConfig]
Flac: d:/util/bin/flac.exe
Metaflac: d:/util/bin/metaflac.exe
Encoders: mp3,mp3crappy
[mp3]
Directory: f:/music-rerip/mp3/New/$P/$T
Filename: %D%n-$t.mp3
FilenameVA: %D%n-$t($p).mp3
Command: d:/util/bin/lame.exe –alt-preset standard –id3v2-only –tt %t –ta %P–tl %T –ty %Y –tn %n –tg Rock %f %F
CommandVA: d:/util/bin/lame.exe –alt-preset standard –id3v2-only –tt %t –ta %p –tl %T –ty %Y –tn %n –tg Rock %f %F
[mp3crappy]
Directory: f:/music-rerip/mp3crappy/New/$P/$T
Filename: %D%n-$t.mp3
FilenameVA: %D%n-$t($p).mp3
Command: d:/util/bin/lame.exe –alt-preset 96 –id3v2-only –tt %t –ta %P–tl %T –ty %Y –tn %n –tg Rock %f %F
CommandVA: d:/util/bin/lame.exe –alt-preset 96 –id3v2-only –tt %t –ta %p –tl %T –ty %Y –tn %n –tg Rock %f %F

All of these scripts are at http://www.phred.org/~alex/transcode. Note that they are likely to change a lot in the next couple of weeks. Here is whatyou’ll find there today:

• MusicBrainzHelper.py — A helper class for Python tomake it easier to work with MusicBrainz
• FlacHelper.py — A helperclass for Python to make it easier to work with FLAC files.
• tag.py — The GUI tool show above for adding the DISC_MUSICBRAINZ_ID tag to FLAC images.
• transcode.py — The command line tool to convert FLAC images to MP3 or other files
• cache-tags.py — This will cache the tags for each FLAC image if you want to run transcode.py while disconnected from the internet.
• TODO — Known bugs

alex

This is going to be a multipart blog series on ripping CDs. I’ll start be describing the goals that lead me to this solution:

• Use a hardware changer to rip the CDs so that I don’t need to sit there and feed the computer each disk one at a time.
• Use ripping software which detects and corrects for errors during ripping caused by scratches or fingerprints on the CDs.
• Rip to a lossless format so that the files can be converted to any audio format at any point in the future.
• Each ripped disk should be represented by a single file (instead of a file per track) so that an exact copy of the CD can be burned in the future if that need arises.
• Use the most reliable tagging (metadata) database available so that the CDs end up with consistent and accuate titles.

Part one talks about the first four bullet points. Part two will expand on the last bullet point.

I had some vacation time to burn and rather than doing something useful like going on a long bike ride, remodeling the downstairs bathroom, or even reading a few novels I decided to rerip all of our music. Since I’m a software developer and not a CD changer I decided to buy new hardware and then write software to make it all work.

I ordered a Sony VGP-XL1B2 Media Changerfrom Amazon. This is a 200 disk CD/DVD changer designed for Windows Media Center machines. Itconnects to your computer via firewire (1394). The changer isn’t supposed to work under Windows XP and I didn’t want to use Media Center for ripping, but I figured that I could make it work anyway. What really convinced me was this blog post from Matt Goyerwhich described the interface that the changer’s driver must provide to Windows.

I ordered the changer and it arrived the next day (thanks to Amazon Prime’s $3.99 overnight shipping). When I plugged it into a Windows XP SP2 machine I was surprised to see it work without needing to install any drivers. A service called the Removable Storage Manager (RSM) detected it and seemed to start doing something with the CDs. I wasted about two hours playing with RSM before deciding that it was overly complicated for what I needed. So I decided to write my own tool to control the changer.

Using information from Matt Goyer’s blog post and MSDN I was able to write a small program called MediaChanger(the zip in that directory contains source and a debug binary). MediaChanger supports the following commands:

• MediaChanger mount — This mounts the disk in slot into the drive.
• MediaChanger unmount — This puts the currently mounted disk back to the slot that it came from.
• MediaChanger drivestatus — Reports if the drive is empty or full and where the disk came from.
• MediaChanger next — Unmounts the current disk and mounts the next one.

This provides enough changer support for a shell script to mount each disk one by one and do something with them. That provided the basis for ripping all of our disks.

Now the problem was automating my preferred ripper, Exact Audio Copy (EAC). EAC is the best audio ripper for CDs if you care about getting perfect rips with no errors. It is not always fast, but it always does a good job. It also supports ripping to a format called WAV+CUE which is a WAV file with all of the music and a text file called a CUEsheet that has the table of contents from the disk. We need WAV+CUE support to create FLAC images.

Sadly EAC is not designed to be scripted, which makes using it with my MediaChanger program a little difficult. A couple of friends pointed me to a script called REACT. REACT is written in a scripting language called AutoIt and adds features to EAC by watching dialog boxes and using menu items as a human would. This is a really ugly way to script something, but it is also the only option available for EAC. I modified REACT to add the following features:

• When finished ripping a diskswitch to the next one and start ripping (this is turned on by pressing Alt-F5). This is built on top of REACT’s existing feature of ripping an image by pressing F10.
• When EAC finds multiple hits for a CD in CDDB use the first one.
• If there is no CDDB information for a CD then title the CD after the slot that it came from.

I’m using REACT to generate FLAC images from the WAV+CUE ripped by EAC. This produces a single FLAC file which contains all of the data from the CD. In part two of this series I’ll explainhow Itag and convert my FLAC images into MP3 files.

With these changes REACT was ready for bulk CD ripping from the changer. I loaded up the changer with a drawer full of CDs from our CD cabinet and let it run. The drive in the changer is not very fast at audio extraction, so it takes about 24 hours for it to rip 200 CDs. This is a little longer than I hoped for it, but at least I don’t have to sit there while it runs.

The last article that I wrote about using the Mac has created a lot of emails and discussions in my inbox. I received a lot of helpful hints too.

Here are a few things that I learned:

1,,,,) In the Keyboard control panel you can make Tab switch between all controls (this is on the Keyboard shortcuts tab). In Safari you can also change it so that Tab goes between all fields. A few people besides myself did miss the Alt-underlined letter method of getting into the menus on Windows.

2) If you hold the mouse button down or right click over an item in the Dock you can see all open windows. This is better, although I still prefer the Windows way of just showing me an icon in the taskbar per open window.

3) No one seems to like iPhoto very much, at least based on responses to my entry. A few people said that iPhoto 6 is better, but since this is a loaner machine I’m not going to purchase it to find out.

4) I can drag folders to the Dock if I want menus to get to less used Applications, making it more like the Windows Taskbar.

5) I played more with the fonts and got Terminal to look better, but it still isn’t very good. The font smoothing on Windows when you are using an LCD is much better (they use a technique called sub pixel rendering (marketted as ClearType) which makes a huge difference). At least I can tell a comma and a period apart now.Below is a comparison If you have a LCD display then clean on each one to see what they look like at normal resolution (these are closeups).

Look at what the Mac is doing to the t there, it is very smudgy. The difference is big enough to give me eyestrain after using the Mac for a while. If you want to see a related silly little OS X bug go into the terminal app and turn on both a blinking cursor and anti-aliasing. You’ll see the left most character on the row with the cursor moving back and forth by one pixel.

6) The Mac thing of keeping an application open even after closing all windows seems to confuse a lot of my other multiplatform friends too. Windows and Unix running X11 don’t behave this way. I think they really could fix this just by making the Dock and Finder tell an Application to open a default window if it is being brought back to the foreground and has no open Windows.

Our adventures with the Mac are over because I finally remembered to bring home a new keyboard for the desktop machine. I enjoyed playing with it, but I’m not sad that I’m done playing with it either (at least as our primary desktop machine). I hope this information was useful to others.