rebuilding bikelist.org archives

Most blog readers probably know that I host/run bikelist.org, phred.org, and many of the bicycle-related mailing lists associated with them such as Internet-BOB, Framebuilders, and Touring. Many years ago I wrote a crude system for searching the archives. It is based on Index Server and IIS and has many limitations such as not knowing what a message thread is, being difficult to backup, and frequently breaking. It was time to rewrite it.

About 3 years ago I started writing a new archival system based on SQL Server and their full text indexing system. I’d work on the new archives for one or two weeks worth of bus rides, then put the project away and ignore it for a year. Amazingly I did make actual progress working this way. For the past two weeks I’ve been putting the final touches on the system. You can see it live at http://search.bikelist.org/beta.

The goals of the new system are:

  • Having agood schema that can properly represent discussions instead of just single messages.A thread is all displayed as a single page to make it easier to get context.
  • Using an easier platform to innovate on. With the new system it won’t be hard to add features such as posting through the web interface (that isn’t there yet, but likely will be in the next year or so).
  • Having a much nicer interface.

Ultimately the goal is to have a clean, simple, fast system for finding all of the great content that has been posted to these lists in the last 5-15 years. I still have a small amount of work to do completely replacing the old system, but I wanted to setup a preview so that people could see how it looks and provide feedback. The internet-BOB and framebuilders lists are being loaded into the system today,so there is a lot of content to explore. Please leave comments here or send me email telling me what you think.

A fun side effect of rewriting the archives is that I often get side tracked by all of the interesting old messages out there.

8 Comments

  1. Myrcurial says:

    Thanks again for all of your work in hosting the lists…

    Do you have the capability in the search to pull a random result from the return set. Would be interesting to pull random quotes from some of the more verbose posters to the list. Kinda like the “I”m feeling lucky” button on google :)

    J

  2. AlexWetmore says:

    Pulling a random search result isn”t too hard.

    I might add a feature where it shows a random message on the homepage at some point — click on it to read the rest of the thread. I”ll have to think about how that will look.

  3. christian says:

    hey, this looks great and will be really useful. I got an error when trying a boolean search:

    Server Error in ”/beta” Application.
    Syntax error near ”monkey” in the full-text search condition ”rohloff AND karate monkey”.
    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

    Exception Details: System.Data.SqlClient.SqlException: Syntax error near ”monkey” in the full-text search condition ”rohloff AND karate monkey”.

    Source Error:

    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

    Stack Trace:

    [SqlException (0x80131904): Syntax error near ''monkey'' in the full-text search condition ''rohloff AND karate monkey''.]
    System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) +862234
    System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +739110
    System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +188
    System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +1956
    System.Data.SqlClient.SqlDataReader.HasMoreRows() +150
    System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout) +214
    System.Data.SqlClient.SqlDataReader.Read() +9
    System.Data.Common.DataAdapter.FillLoadDataRow(SchemaMapping mapping) +156
    System.Data.Common.DataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords, DataColumn parentChapterColumn, Object parentChapterValue) +153
    System.Data.Common.DataAdapter.Fill(DataSet dataSet, String srcTable, IDataReader dataReader, Int32 startRecord, Int32 maxRecords) +172
    System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +175
    System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +137
    System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable) +83
    System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments) +1770
    System.Web.UI.DataSourceView.Select(DataSourceSelectArguments arguments, DataSourceViewSelectCallback callback) +17
    System.Web.UI.WebControls.DataBoundControl.PerformSelect() +149
    System.Web.UI.WebControls.BaseDataBoundControl.DataBind() +70
    System.Web.UI.WebControls.GridView.DataBind() +4
    System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound() +82
    System.Web.UI.WebControls.CompositeDataBoundControl.CreateChildControls() +69
    System.Web.UI.Control.EnsureChildControls() +87
    System.Web.UI.Control.PreRenderRecursiveInternal() +41
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Control.PreRenderRecursiveInternal() +161
    System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1360

  4. AlexWetmore says:

    You need to put phrases in quotes when using boolean search, that is why it isn”t the default. At some point I”ll have a more helpful syntax page come up when you do something illegal.

    Try searching for:
    rohloff and “karate monkey”

  5. Brian Loudon says:

    Yay, it lives!

    Body text on message tree view needs some padding. The text is jammed against the edge of the viewport.

  6. Vo Tran says:

    thanks for hosting these lists. i actually like the current archive design and use them alot. but whatever you do please don”t loose or delete any of the great content. it is a great resource and i”m sure i”m not the only one who uses it frequently. Please keep it all and don”t delete any!!!!!! thanks Vo

  7. Alex Yip says:

    Hi Alex, Thanks for running the email lists and archives.
    I use the search function often, and improvements are always welcome.

    In addition to your custom indexing system, you might try to make the archives easily indexable by Google and Yahoo, etc. Their indexers are highly refined for speed and language processing and can be good for augmenting the site’’s custom built-in search funtions.

    - Alex

  8. AlexWetmore says:

    Vo — Nothing will be deleted and I”m going to great extents to make sure that old message URLs will work with the new archives.

    Alex — I”ve had bad luck in the past with Google overindexing my site and killing my available bandwidth. We”ll see how they do with the new system.