rebuilding bikelist.org archives
Most blog readers probably know that I host/run bikelist.org, phred.org, and many of the bicycle-related mailing lists associated with them such as Internet-BOB, Framebuilders, and Touring. Many years ago I wrote a crude system for searching the archives. It is based on Index Server and IIS and has many limitations such as not knowing what a message thread is, being difficult to backup, and frequently breaking. It was time to rewrite it.
About 3 years ago I started writing a new archival system based on SQL Server and their full text indexing system. I’d work on the new archives for one or two weeks worth of bus rides, then put the project away and ignore it for a year. Amazingly I did make actual progress working this way. For the past two weeks I’ve been putting the final touches on the system. You can see it live at http://search.bikelist.org/beta.
The goals of the new system are:
-
Having agood schema that can properly represent discussions instead of just single messages.A thread is all displayed as a single page to make it easier to get context.
-
Using an easier platform to innovate on. With the new system it won’t be hard to add features such as posting through the web interface (that isn’t there yet, but likely will be in the next year or so).
-
Having a much nicer interface.
Ultimately the goal is to have a clean, simple, fast system for finding all of the great content that has been posted to these lists in the last 5-15 years. I still have a small amount of work to do completely replacing the old system, but I wanted to setup a preview so that people could see how it looks and provide feedback. The internet-BOB and framebuilders lists are being loaded into the system today,so there is a lot of content to explore. Please leave comments here or send me email telling me what you think.
A fun side effect of rewriting the archives is that I often get side tracked by all of the interesting old messages out there.
Thanks again for all of your work in hosting the lists…
Do you have the capability in the search to pull a random result from the return set. Would be interesting to pull random quotes from some of the more verbose posters to the list. Kinda like the “I”m feeling lucky” button on google
J
Pulling a random search result isn”t too hard.
I might add a feature where it shows a random message on the homepage at some point — click on it to read the rest of the thread. I”ll have to think about how that will look.
hey, this looks great and will be really useful. I got an error when trying a boolean search:
Server Error in ”/beta” Application.
Syntax error near ”monkey” in the full-text search condition ”rohloff AND karate monkey”.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Data.SqlClient.SqlException: Syntax error near ”monkey” in the full-text search condition ”rohloff AND karate monkey”.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SqlException (0x80131904): Syntax error near ''monkey'' in the full-text search condition ''rohloff AND karate monkey''.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) +862234
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +739110
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +188
System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +1956
System.Data.SqlClient.SqlDataReader.HasMoreRows() +150
System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout) +214
System.Data.SqlClient.SqlDataReader.Read() +9
System.Data.Common.DataAdapter.FillLoadDataRow(SchemaMapping mapping) +156
System.Data.Common.DataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords, DataColumn parentChapterColumn, Object parentChapterValue) +153
System.Data.Common.DataAdapter.Fill(DataSet dataSet, String srcTable, IDataReader dataReader, Int32 startRecord, Int32 maxRecords) +172
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +175
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable) +83
System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments) +1770
System.Web.UI.DataSourceView.Select(DataSourceSelectArguments arguments, DataSourceViewSelectCallback callback) +17
System.Web.UI.WebControls.DataBoundControl.PerformSelect() +149
System.Web.UI.WebControls.BaseDataBoundControl.DataBind() +70
System.Web.UI.WebControls.GridView.DataBind() +4
System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound() +82
System.Web.UI.WebControls.CompositeDataBoundControl.CreateChildControls() +69
System.Web.UI.Control.EnsureChildControls() +87
System.Web.UI.Control.PreRenderRecursiveInternal() +41
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Control.PreRenderRecursiveInternal() +161
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1360
You need to put phrases in quotes when using boolean search, that is why it isn”t the default. At some point I”ll have a more helpful syntax page come up when you do something illegal.
Try searching for:
rohloff and “karate monkey”
Yay, it lives!
Body text on message tree view needs some padding. The text is jammed against the edge of the viewport.
thanks for hosting these lists. i actually like the current archive design and use them alot. but whatever you do please don”t loose or delete any of the great content. it is a great resource and i”m sure i”m not the only one who uses it frequently. Please keep it all and don”t delete any!!!!!! thanks Vo
Hi Alex, Thanks for running the email lists and archives.
I use the search function often, and improvements are always welcome.
In addition to your custom indexing system, you might try to make the archives easily indexable by Google and Yahoo, etc. Their indexers are highly refined for speed and language processing and can be good for augmenting the site’’s custom built-in search funtions.
- Alex
Vo — Nothing will be deleted and I”m going to great extents to make sure that old message URLs will work with the new archives.
Alex — I”ve had bad luck in the past with Google overindexing my site and killing my available bandwidth. We”ll see how they do with the new system.