LINQ to Lucene

New Post: Paul's LINQ to Lucene Questions

CVertex — Sun, 07 Sep 2008 10:08:17 GMT

The issue of reinstantiating the data context is discussed further here - DataContext recycle discussion

Created Issue: DatabaseIndexSet should somehow re-instantiate the

CVertex — Sun, 07 Sep 2008 10:07:42 GMT

see http://www.codeplex.com/linqtolucene/Thread/View.aspx?ThreadId=33663

New Post: Paul's LINQ to Lucene Questions

CVertex — Sun, 07 Sep 2008 10:05:45 GMT

Paul posted some questions about linq to lucene and it's usage on my blog. I responded via email but decided to post here for all interested.

"A couple quick questions. The Lucene docs say that for a web app, you
should 'keep your index open' across postbacks, either via the app
cache or static objects or whatever. Is that true for Linq-to-Lucene
as well?"

Yup. Index<T> contains a property called Context (which is of type
IndexContext). The index context instance contains the IndexModifier
and IndexSearcher lucene instances which do all the magic under the
hood. I've tried to hide this via encapsulation in a sensible way. If
you keep your Index<T> or IndexSet or DatabaseIndexSet<TData> open and
in your Application cache through the life of the asp.net app, you can
safely query/mod the index without instantiating anything you don't
need to.

"Second, how often would you recommend managing re-indexing? I was
contemplating writing the index on application start, and then when
new objects get added, do it through a repository object which then
also writes them to the index, rather than re-indexing the whole db
table again."

Firstly, there is no ability to update documents in an index. To
update a document, you must delete it then add it again.

I've given updating the index alot of thought (and done it twice in
production using the lucene api), but not yet come to a decision about
how to make it easier for linq to lucene users. Ideally, I'd like to
listen to ObjectTrackingManager (or whatever its called) for changes
and progressively update the index as SaveChanges is called. However,
this approach will have definite performance issues. The approach you
pick depends on a balance between performance (you don't want to
update your index too often otherwise searchers are taken out of
action too often) and up-to-datedness.

On the previous two projects I've worked on, the index has been small
enough to reindex the entire database (barring deactivated records,
expired records etc) every hour then swap the index files across. If
you can afford the CPU time and penalties of data being out of date by
an hour, I'd follow this approach. Please let me know if you can't and
we'll think about a smarter interim solution.

I'm working on a batch indexing and update system at the moment but am
running into sync issues I haven't been able to solve. This system
will allow developers to choose "how" an index set is indexed. That
is, if the indexing of tables is done in parallel, sequentially and
how much to buffer in memory during. The updater will do something
similar, contain a queue of documents that are "dirty", then
periodically or manually update each document.

"Third, I wasn't clear from your description why I would want to
tokenize, can you explain or link? "

Tokenization is necessary to let the term vector model do it's magic,
and to allow terms within a field to be searchable. Tokenization is
actually one phase of a process called field analysis. When a
value/query must be converted into lucene field values, the chosen
analyzer for that field/document is executed. Field analysis is a set
of composable functions that convert a string into an array of values.
Analyzers work to create the lowest string representation of a field
such that variants are properly matched. SimpleAnalyzer for example,
contains a whitespace tokenizer and lower caser. Without an analyzer
on the field, Lucene will happily index the string as is, and casing
will be trapped in the index.

e.g. If ContactTitle wasn't analyzed, the following query wouldn't find anything

var managers = from c in index.Get<Customer>
where c.ContactTitle == "marketing manager"
select c;

Why? because all those customers have titles of "Marketing Manager".
Upon linq to object translation, linq to lucene will generate a
TermQuery that tries to match the field value "marketing manager" with
"Marketing Manager". For speed (and usability), Lucene.net uses case
sensitive string equality. This seems very strange for simple text
fields, but becomes important when you need to index really long
fields and strangely formatted fields.

Hope that helps,
-Vijay

Source code checked in, #23910

CVertex — Sun, 07 Sep 2008 08:56:01 GMT

Changed to struct

Closed Issue: Field boosting (on FieldAttribute)

CVertex — Mon, 25 Aug 2008 12:07:46 GMT

This code might already be there, but a unit test should be added
Comments: The codes in there, unit testing is a bit of a pest

New Post: Multiple field searches

CVertex — Thu, 21 Aug 2008 05:42:01 GMT

Just added this issue

Created Issue: Field boosting (on FieldAttribute)

CVertex — Thu, 21 Aug 2008 05:40:59 GMT

This code might already be there, but a unit test should be added

New Post: Multiple field searches

CVertex — Thu, 21 Aug 2008 05:40:15 GMT

Boosting fields happens at Index time. This will naturally give certain fields higher importance over others.

FieldAttribute should have a boost property on it, which is settable on the property itself.

[Field(Type=Blah, Boost = 3.0f)]
object MyField {get;set;}

I remember putting the stub in, and vaguely remember adjusting the indexer to use the supplied boosts. What I definitely did not do is add a Unit test, which should be something to do.

Try this out Paul, if the Boost property isn't there, please add a issue in the tracker and I'll add it later. You're also free to submit a patch.

-V

New Post: Multiple field searches

pvencill — Wed, 20 Aug 2008 16:05:02 GMT

Ok, this was me being an idiot. Seems that I didn't put in metadata mapping info for the Address field, which is what was throwing the error. my mistake, sorry to bother you all. :D

I'd still like to udnerstand how ranking works in these situations, though. Basically, i'd like to be able to do a Boost on a class of terms, rather than on individual terms. e.g. boost a field's weight, vice boosting a term's weight, if I'm clear.

thanks,

Paul

New Post: Multiple field searches

pvencill — Wed, 20 Aug 2008 15:24:00 GMT

This might be my misunderstanding of the docs, but I'm having trouble getting a multiple field query to work. I'm also curious how to rank the results, if I wanted one field to count for more than the other in an intermixed fashion (e.g. tiebreaker kind of boost).

The query I attempted was based on one of the examples (I thought), and looked like so:

var

mmCustomers = dbi.Get<Customer>().Where(c => (c.ContactTitle.Match(new string[]{"marketing","manager"})) || (c.Address.Match(new string[]{"rue"})));

where dbi is an instance of my DatabaseIndexSet<NorthwindDataContext> (for learning off of). the first part of the query works fine by itself, the exception started when I added in the OR logic (||) to try and bring in the other field.

any thoughts?

the exception I got was:
Lucene.Net.QueryParsers.ParseException was unhandled
Message="Encountered \":\" at line 1, column 59.\nWas expecting one of:\n    <AND> ...\n    <OR> ...\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    \"(\" ...\n    \")\" ...\n    \"^\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <FUZZY_SLOP> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    "
Source="Lucene.Net"
StackTrace:
       at Lucene.Net.QueryParsers.QueryParser.Jj_consume_token(Int32 kind) in C:\downloads\lucene\Lucene.Net\QueryParser\QueryParser.cs:line 1412
       at Lucene.Net.QueryParsers.QueryParser.Clause(String field) in C:\downloads\lucene\Lucene.Net\QueryParser\QueryParser.cs:line 884
       at Lucene.Net.QueryParsers.QueryParser.Query(String field) in C:\downloads\lucene\Lucene.Net\QueryParser\QueryParser.cs:line 801
       at Lucene.Net.QueryParsers.QueryParser.Parse(String query) in C:\downloads\lucene\Lucene.Net\QueryParser\QueryParser.cs:line 154
       at Lucene.Linq.Expressions.ParsingQueryTranslator.Translate(String query, String[] defaultFieldNames) in C:\downloads\lucene\Lucene.Linq\Expressions\ParsingQueryTranslator.cs:line 73
       at Lucene.Linq.Expressions.ParsingQueryTranslator.Translate(Expression expression) in C:\downloads\lucene\Lucene.Linq\Expressions\ParsingQueryTranslator.cs:line 51
       at Lucene.Linq.Expressions.QueryProvider.Translate(Expression expression) in C:\downloads\lucene\Lucene.Linq\Expressions\QueryProvider.cs:line 193
       at Lucene.Linq.Expressions.QueryProvider.Execute(Expression expression) in C:\downloads\lucene\Lucene.Linq\Expressions\QueryProvider.cs:line 67
       at Lucene.Linq.Expressions.QueryProvider.System.Linq.IQueryProvider.Execute[S](Expression expression) in C:\downloads\lucene\Lucene.Linq\Expressions\QueryProvider.cs:line 58
       at System.Linq.Queryable.Count[TSource](IQueryable`1 source)
       at First_Lucene.Program.Main(String[] args) in C:\Documents and Settings\pvencill\My Documents\Visual Studio 2008\Projects\First Lucene\First Lucene\Program.cs:line 23
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
InnerException:

New Post: Multiple field searches

pvencill — Wed, 20 Aug 2008 15:23:59 GMT

var

mmCustomers = dbi.Get<Customer>().Where(c => (c.ContactTitle.Match(new string[]{"marketing","manager"})) || (c.Address.Match(new string[]{"rue"})));

New Post: DataContext lifespan

pvencill — Sat, 16 Aug 2008 21:39:47 GMT

Vijay and I were having this discussion on email, moving it here:

> <<< re: MS team's recommendation on DataContexts >>>
>> Good point. I'm still not sure the right thing to do with db data
>> contexts. If you have an official MS guy statement saying that should
>> be done. Without much effort, we could hide recycle/refresh the data
>> context instance inside the DatabaseIndexSet<T>, rather than making
>> the DataContext settable. What do you think we should do?
>
> The recommendation I'm talking about was in the MS Press book
> Programming Microsoft LINQ; it may be online as well, I haven't looked
> for it. I would think that having the DatabaseIndexSet recycle its
> DataContext is a fine idea; I was thinking about it after I wrote the
> question, though, and I can't help but wonder if it wouldn't be a
> tradeoff between the object tracking overhead that the MS guys were
> warning against and the overhead of reflection to create new
> datacontexts by type. Unless you were thinking of a different way to
> recycle the DataContext that I'm not thinking of...

Hmmm, I still have no idea what to do with DataContext instances - how
long they should live and when i should kill them.
The "Unit of work" rule idea seems to be a good one, except not sure
how large units of work can be.
I'm eager to leave the DataContext instance as a singleton (i.e. one
always open instance) and wait until you come across an error during
periodic updates - just as an experiment to see what goes wrong.

I also found this online discussion of why it's best to keep the DataContext as a UoW, and it seems that the main reason has to do w/ avoiding concurrency conflicts:

http://blogs.msdn.com/dinesh.kulkarni/archive/2008/04/27/lifetime-of-a-linq-to-sql-datacontext.aspx

I wonder if it wouldn't be better to offer a choice. If you used an untyped datacontext internally, I think you could take the connection (or connectionstring) of the database and then create and destroy the datacontext on each use, I would think. The developer wouldn't even have to cast on a query, since you coudl still use GetTable<T> under the hood.

Thoughts?

Updated Wiki: Project Roadmap

CVertex — Tue, 22 Jul 2008 13:02:25 GMT

Project Roadmap

This is a list of desired features the contributors would like to add in the future to this library.
If there's anything we're missing please let us know in the issue tracker or in the discussion forum.

LINQ Methods: Any, All
Aggregation
Filters
Performance Optimizations

Compiled Queries
Single expression tree traversal
Configurable batch indexing support using RAMDirectories

OrderBy operation
Nested Queries & Joins
Change Tracking
Automatic synchronization with LINQ to SQL Data Context

Updated Wiki: Project Roadmap

CVertex — Tue, 22 Jul 2008 12:58:19 GMT

Project Roadmap

LINQ Methods: Any, All
Aggregation
Filters
Performance Optimizations

Compiled Queries
Single expression tree traversal
Configurable batch indexing support using RAMDirectories

OrderBy operation
Nested Queries & Joins
Change Tracking
Automatic synchronization with LINQ to SQL Data Context

Updated Wiki: Home

CVertex — Tue, 22 Jul 2008 12:52:53 GMT

What is LINQ to Lucene?

Providing a custom LINQ solution for the Lucene Information Retrieval System, commonly referred to as a search-engine.

Current Release
The Refactored Release
Tuesday, 22nd July 2008
A large set of library changes including an API analogous to LINQ to SQL, more supported LINQ methods, powerful LINQ to SQL Data Context support and more unit tests.

Older Release
The Re-Release
As promised, this release merges the features of previous releases into the one bundle.

Older Release
The Type-Conversion Release
Monday, 19 November 2007
TYPES BEYOND STRINGS NOW SUPPORTED
As promised (albeit out of order), support has been added for type-conversion. If the type of the indexed-property uses a type-converter that supports conversion from a string, the type will automatically convert to its appropriate type. This automatically inlcudes string, bool, int, datetime and a number of other .net types that already implement their own type-converters. Custom types can implement their own type-converters. For more information on type-converters please review either of the following article: MSDN: How to Implement a Type-Converter

Older Release
The Query Release
Tuesday, 13 November 2007
This release provides a real focus on the querying abilities of the LINQ to Lucene project and is the first real 'working release', converting LINQ statements to Lucene queries with deferred query execution and object creation or projection. It culminates the majority of the required querying features for LINQ that Lucene provides natively.

	Lucene Syntax	LINQ to Lucene
Terms & Phrases	"test" or "hello dolly"	c.Match("test") or c.Match("hello dolly")
Fields	title:"The Right way" and text:go	c.Title == "The Right way" or c.Text == "go"
WildCard	amb?r	c.ContactName.Match("amb?r")
Prefix	amber*	c.ContactName.StartsWith("amber")
Fuzzy	roam~ or roam~0.8	c.ContactName.Like("roam") or c.ContactName.Like("roam", 0.8)
Proximity	"jakarta apache"~10	c.ContactName.Like("jakarta apache", 10)
Inclusive Range	mod_date:[20020101 TO 20030101]	c.ModifiedDate.Includes("20020101", "20030101")
Exclusive Range	title:{Aida TO Carmen}	c.Title.Between("Aida", "Carmen")
Boosting	jakarta^4 apache	c.Title.Match("jakarta".Boost(4), apache)
Boolean Or	"jakarta apache" OR jakarta	where c.Match("jakarta apache") \|\| c.Match("jakarta")
Boolean And	"jakarta apache" AND "Apache Lucene"	where c.Match("jakarta apache") && c.Match("Apache Lucene")
Boolean Not	"jakarta apache" NOT "Apache Lucene"	where c.Match("jakarta apache") && !c.Match("Apache Lucene")
Required	+jakarta lucene	c.Title.Match("jakarta".Require(), "lucene")
Grouping	(jakarta OR apache) AND website	where (c.Title == "jakarta" \|\| c.Title == "apache") && (c.Title == "website")
Native Syntax	ie. title:{+return +"pink panther")	c.Search("title:(return +\"pink panther\"")

First Releases
A first release for generating a simple index out of a database for mapping types to Lucene using LINQ is now available.
This is the first small piece of the puzzle.

The Index Release

[Document]
public class Customer 
{
  [Field(Store=FieldStore.Yes, Index=FieldIndex.No)] public string CustomerID { get; set; }
  [Field(Store=FieldStore.Yes, Index=FieldIndex.Tokenized)] public string CustomerName { get; set; }
}
 
IndexContext index = new IndexContext;
index.WriteIndex<Customer>();

Updated Wiki: Home

CVertex — Tue, 22 Jul 2008 12:50:12 GMT

What is LINQ to Lucene?

Providing a custom LINQ solution for the Lucene Information Retrieval System, commonly referred to as a search-engine.

	Lucene Syntax	LINQ to Lucene
Terms & Phrases	"test" or "hello dolly"	c.Match("test") or c.Match("hello dolly")
Fields	title:"The Right way" and text:go	c.Title == "The Right way" or c.Text == "go"
WildCard	amb?r	c.ContactName.Match("amb?r")
Prefix	amber*	c.ContactName.StartsWith("amber")
Fuzzy	roam~ or roam~0.8	c.ContactName.Like("roam") or c.ContactName.Like("roam", 0.8)
Proximity	"jakarta apache"~10	c.ContactName.Like("jakarta apache", 10)
Inclusive Range	mod_date:[20020101 TO 20030101]	c.ModifiedDate.Includes("20020101", "20030101")
Exclusive Range	title:{Aida TO Carmen}	c.Title.Between("Aida", "Carmen")
Boosting	jakarta^4 apache	c.Title.Match("jakarta".Boost(4), apache)
Boolean Or	"jakarta apache" OR jakarta	where c.Match("jakarta apache") \|\| c.Match("jakarta")
Boolean And	"jakarta apache" AND "Apache Lucene"	where c.Match("jakarta apache") && c.Match("Apache Lucene")
Boolean Not	"jakarta apache" NOT "Apache Lucene"	where c.Match("jakarta apache") && !c.Match("Apache Lucene")
Required	+jakarta lucene	c.Title.Match("jakarta".Require(), "lucene")
Grouping	(jakarta OR apache) AND website	where (c.Title == "jakarta" \|\| c.Title == "apache") && (c.Title == "website")
Native Syntax	ie. title:{+return +"pink panther")	c.Search("title:(return +\"pink panther\"")

Future Release

Unit Tests and Xml Comments
Aggregation
Filters
Performance Optimizations
Query Analyzers beyond StandardAnalayzer
OrderBy operation
Nested Queries & Joins
Updating the Index
Change Tracking

[Document]
public class Customer 
{
  [Field(Store=FieldStore.Yes, Index=FieldIndex.No)] public string CustomerID { get; set; }
  [Field(Store=FieldStore.Yes, Index=FieldIndex.Tokenized)] public string CustomerName { get; set; }
}
 
IndexContext index = new IndexContext;
index.WriteIndex<Customer>();

UPDATED RELEASE: 7. The Refactored Release (Jul 22, 2008)

Tue, 22 Jul 2008 12:45:51 GMT

This release includes a large set of changes to the library:

New API that mirrors LINQ to SQL Table<T>
IndexSet for managing a collection of indexes
DatabaseIndexSet for making indexes from a LINQ to SQL Data Context
Centralized document and field details readers
New LINQ methods: Count, Skip, Take, First, Last, ElementAt
Mapping attributes can be added to a separate type
IHit for providing Relevance and DocumentId to domain tier
Lucene.Net Directory agnostic Index<T>
Refactored ObjectTranslation - abstract an interface for swappable translators in the future
Added PagedList for paging hits
An extra demonstration project comparing LINQ to XML to LINQ to Lucene.Net
IndexContext is analogous to the ASP.NET HttpContext with exposed locking semantics
Alot more Unit Tests using MbUnit
Batch Indexing - still a work in progress

CREATED RELEASE: The Refactored Release (Jul 22, 2008)

Tue, 22 Jul 2008 12:45:14 GMT

This release includes a large set of changes to the library:

New API that mirrors LINQ to SQL Table<T>
IndexSet for managing a collection of indexes
DatabaseIndexSet for making indexes from a LINQ to SQL Data Context
Centralized document and field details readers
New LINQ methods: Count, Skip, Take, First, Last, ElementAt
Mapping attributes can be added to a separate type
IHit for providing Relevance and DocumentId to domain tier
Lucene.Net Directory agnostic Index<T>
Refactored ObjectTranslation - abstract an interface for swappable translators in the future
Added PagedList for paging hits
An extra demonstration project comparing LINQ to XML to LINQ to Lucene.Net
IndexContext is analogous to the ASP.NET HttpContext with exposed locking semantics
Alot more Unit Tests using MbUnit
Batch Indexing - still a work in progress

RELEASED: The Refactored Release (Jul 22, 2008)

Tue, 22 Jul 2008 12:45:14 GMT

This release includes a large set of changes to the library:

New API that mirrors LINQ to SQL Table<T>
IndexSet for managing a collection of indexes
DatabaseIndexSet for making indexes from a LINQ to SQL Data Context
Centralized document and field details readers
New LINQ methods: Count, Skip, Take, First, Last, ElementAt
Mapping attributes can be added to a separate type
IHit for providing Relevance and DocumentId to domain tier
Lucene.Net Directory agnostic Index<T>
Refactored ObjectTranslation - abstract an interface for swappable translators in the future
Added PagedList for paging hits
An extra demonstration project comparing LINQ to XML to LINQ to Lucene.Net
IndexContext is analogous to the ASP.NET HttpContext with exposed locking semantics
Alot more Unit Tests using MbUnit
Batch Indexing - still a work in progress

Created Issue: Built-in support for PagedList

CVertex — Tue, 22 Jul 2008 09:29:24 GMT

Calling LINQ_Lucene_expression.ToPagedList() performs the search twice. Once for the Count, and once of the Skip and Take methods. By building in support for PagedList<T>, the double searches may be avoided.