NCache & Full-Text Search

I bet you have used the SQL LIKE operator to find a keyword in a text field. For large amounts of text, that would be slow. Let’s learn how to implement a full-text search with Lucene and NCache.

What is Full-Text Search?

Full-text search is a technique to search not only exact matches of a keyword in some text but for patterns of text, synonyms, or close words in large amounts of text.

To support large amounts of text, searching is divided into two phases: indexing and searching. In the indexing phase, an analyzer processes text to create indexes based on the rules of a spoken language like English to remove stop words and record synonyms and inflections of words. Then, the searching phase only uses the indexes instead of the original text source.

Full-Text Search with Lucene and NCache

1. Why Lucene and NCache?

From its official page, “Apache Lucene.NET is a high performance search library for .NET.” It’s a C# port of Java-based Apache Lucene, an “extremely powerful” and fast search library optimized for full-text search.

NCache gives distributed capabilities to Lucene by implementing the Lucene API on top of its In-Memory Distributed cache. This way, NCache makes Lucene a linearly scalable full-text searching solution for .NET. For more features of Distributed Lucene, check NCache Distributed Lucene page.

2. Create a Lucene Cache in NCache

We have already installed and used NCache as a IDistributedCache provider. This time, let’s use NCache version 5.3 to find movies by title or director name using Lucene’s full-text search.

Lucene stores data in immutable “segments,” which consist of multiple files. We can store these segments in our local file system or in RAM. But, since we’re using Lucene with NCache, we’re storing these segments in NCache.

Before indexing and searching anything, first, we need to create a Distributed Lucene Cache. Let’s navigate to http://localhost:8251 to fire NCache Web Manager and add a New Distributed Cache.

Let’s select “Distributed Lucene” in the Store Type and give it a name. Then, let’s add our own machine and a second node. For write operations, we need at least two nodes. We can stick to the defaults for the other options.

By default, in Windows machines, NCache stores Lucene indexes in C:\ProgramData\ncache\lucene-index.

For more details about these installation options, check NCache official docs.

3. Index Some Movies

After creating the Distributed Lucene cache, let’s populate our Lucene indexes with some movies from a Console app. Later, we will search them from another Console app.

First, let’s create a Console app to load some movies to the Lucene Cache. Also, let’s install the Lucene.Net.NCache NuGet package.

In the Program.cs file, we could load all movies we want to index from a database or another store. For example, let’s use a list of movies from IMDb. Something like this,

using SearchMovies.Shared;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Services;

var searchService = new SearchService(Config.CacheName);
searchService.LoadMovies(SomeMoviesFromImdb());

Console.WriteLine("Press any key to continue...");
Console.ReadKey();

// This list of movies was taken from IMDb dump
// See: https://www.imdb.com/interfaces/
static IEnumerable<Movie> SomeMoviesFromImdb()
{
    return new List<Movie>
    {
        new Movie("Caged Fury", 1983, 3.8f, 89, new Director("Maurizio Angeloni", 1959), new []{ Genre.Crime,Genre.Drama  }),
        new Movie("Bad Posture", 2011, 6.5f, 93, new Director("Jack Smith", 1932), new []{ Genre.Drama,Genre.Romance  }),
        new Movie("My Flying Wife", 1991, 5.5f, 91, new Director("Franz Bi", 1899), new []{ Genre.Action,Genre.Comedy,Genre.Fantasy  }),
        new Movie("Modern Love", 1990, 5.2f, 105, new Director("Sophie Carlhian", 1962), new []{ Genre.Comedy  }),
        new Movie("Sins", 2012, 2.3f, 84, new Director("Pierre Huyghe", 1962), new []{ Genre.Action, Genre.Thriller  })
        // Some other movies here...
    };
}

Notice we used a SearchService to handle the index creation in a method called LoadMovies(). Let’s take a look at it.

using Lucene.Net.Analysis.Standard;
using Lucene.Net.Index;
using Lucene.Net.Store;
using Lucene.Net.Util;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Extensions;

namespace SearchMovies.Shared.Services;

public class SearchService
{
    private const string IndexName = "movies";
    private const LuceneVersion luceneVersion = LuceneVersion.LUCENE_48;

    private readonly string _cacheName;

    public SearchService(string cacheName)
    {
        _cacheName = cacheName;
    }

    public void LoadMovies(IEnumerable<Movie> movies)
    {
        using var indexDirectory = NCacheDirectory.Open(_cacheName, IndexName);
        // 1. Opening directory    ^^^

        var standardAnalyzer = new StandardAnalyzer(luceneVersion);
        var indexConfig = new IndexWriterConfig(luceneVersion, standardAnalyzer)
        {
            OpenMode = OpenMode.CREATE
        };
        using var writer = new IndexWriter(indexDirectory, indexConfig);
        // 2. Creating a writer   ^^^

        foreach (var movie in movies)
        {
            var doc = movie.MapToLuceneDocument();
            writer.AddDocument(doc);
            //     ^^^^^^^^^^^
            // 3. Adding a document
        }

        writer.Commit();
        //     ^^^^^^
        // 4. Writing documents
    }
}

A bit of background first, Lucene uses documents as the unit of search and index. Documents can have many fields, and we don’t need a schema to store them.

We can search documents using any field. Lucene will only return those with that field and matching data. For more details on some Lucene internals, check its Lucene Quick Start guide.

Notice we started our LoadMovies by opening an NCache directory. We needed the same cache name we configured before and an index name. Then we created an IndexWriter with our directory and some configurations, like a Lucene version, an analyzer, and an open mode.

Then, we looped through our movies and created a Lucene document for each one using the MapToLuceneDocument() extension method. Here it is,

using Lucene.Net.Documents;
using SearchMovies.Shared.Entities;

namespace SearchMovies.Shared.Extensions;

public static class MoviesExtensions
{
    public static Document MapToLuceneDocument(this Movie self)
    {
        return new Document
        {
            new TextField("name", self.Name, Field.Store.YES),
            new TextField("directorName", self.Director.Name, Field.Store.YES)
        };
    }
}

To create Lucene documents, we used two fields of type TextField: movie name and director name. For each field, we need a name and a value to index. We will use the field names later to create a response object from search results.

There are two basic field types for Lucene documents: TextField and StringField. The first one has support for Full-Text search and the second one supports searching for exact matches.

Once we called the Commit() method, NCache stored our movies in a distributed index.

4. Full-Text Searching Movies

Now that we populated our index with some movies, to search them, let’s create another Console app to read a Lucene query.

Again, let’s use the same SearchService, this time with a SearchByNames() method passing a Lucene query.

using Lucene.Net.Analysis.Standard;
using Lucene.Net.Index;
using Lucene.Net.QueryParsers.Classic;
using Lucene.Net.Search;
using Lucene.Net.Store;
using Lucene.Net.Util;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Extensions;
using SearchMovies.Shared.Responses;

namespace SearchMovies.Shared.Services;

public class SearchService
{
    // Same SearchService as before...

    public IEnumerable<MovieResponse> SearchByNames(string searchQuery)
    {
        using var indexDirectory = NCacheDirectory.Open(_cacheName, IndexName);
        using var reader = DirectoryReader.Open(indexDirectory);
        //                 ^^^^^^^^^^^^^^^
        // 1. Creating a reader
        var searcher = new IndexSearcher(reader);

        var analyzer = new StandardAnalyzer(luceneVersion);
        var parser = new QueryParser(luceneVersion, "name", analyzer);
        var query = parser.Parse(searchQuery);
        //          ^^^^^^
        // 2. Parsing a Lucene query 

        var documents = searcher.Search(query, 10);
        //              ^^^^^^^^
        // 3. Searching documents

        var result = new List<MovieResponse>();
        for (int i = 0; i < documents.TotalHits; i++)
        {
            var document = searcher.Doc(documents.ScoreDocs[i].Doc);
            result.Add(document.MapToMovieResponse());
            //     ^^^
            // 4. Populating a result object
        }

        return result;
    }
}

This time, instead of creating an IndexWriter, we used a DirectoryReader and a query parser with the same Lucene version and analyzer. Then, we used the Search() method with the parsed query and a result count. The next step was to loop through the results and create a response object.

To create a response object from a Lucene document, we used the MapToMovieResponse(). Here it is,

public static MovieResponse MapToMovieResponse(this Document self)
{
    return new MovieResponse(self.Get("name"), self.Get("directorName"));
}

This time, we used the Get() method with the same field names as before to retrieve fields from documents.

For example, let’s find all movies whose director’s name contains “ca”, with the query directorName:ca*,

Of course, there are more keywords in Lucene Query Syntaxt than the ones we used here.

Voilà! That’s how to use Distributed Lucene with NCache. If we already have an implementation with Lucene.NET, we would need few code changes to migrate it to Lucene with NCache. Also, notice that NCache doesn’t implement all Lucene methods.

To follow along with the code we wrote in this post, check my Ncache Demo repository over on GitHub.

To read more content, check my post Working with ASP.NET Core IDistributedCache Provider for NCache to learn about caching with ASP.NET Core and NCache.

I wrote this post in collaboration with Alachisoft, NCache creators.

Happy coding!

Monday Links: Storytelling, Leet Code and Boredom

In case you find the Monday Links series for the first time: these are five links from past weeks that I found interesting (and worth sharing) while procastinating surfing the Web. This is not a link-building scheme, I only read and liked these articles.

The Secret Art of Storytelling in Programming by Yehonathan Sharvit

This presentation starts with the author sharing his struggle to read books as a kid. And later read code as a programmer and contracts as a consultant.

The main message from this presentation is how memory, attention, and structure spans relate to coding. The author presents three coding style principles that respect mind spans:

1. Use small functions
2. Make every line in a function have the same level of abstraction
3. Use descriptive names instead of comments

Watch full presentation

Stop Interviewing With Leet Code

Interviewing is broken. We all agree. But we don’t know how to fix it. Brain teasers, IQ tests, pair programming, algorithms? This post presents an alternative: inspect the candidate GitHub and public work, ask him to review some piece of code, add unit tests or do some refactors. That sounds like a better idea! Read full article

How to professionally say…

One of the things we learn while working for a company is office politics. Basically, how to say things and to step away from certain situations. I learned from a coworker to say “I don’t have enough information to answer that question” instead of a simple “I don’t know.” You will find more lines like that one in this post. Read full article

How to feel engaged at work: a software engineer’s guide

Let’s be honest. It’s rewarding when we see the impact of our work. But, often, all days look almost the same. Another JIRA ticket for a production issue. Another meeting that could have been an email. This article shows four ideas to spice things up. Read full article

The Toxic Grind

This article talks about success, hard work, and work-life balance. This is my favorite line: “We should glorify the journey of achieving something meaningful, not the dream of wealth and power. Glorify the skills you build along the way, not the shortcuts you take.” Read full article

Voilà! Another Monday Links. What do you do to feel engaged at your work? Have you been asked to solve LeetCode questions during interviews? Would you like to do something different in future interviews?

In the meantime, check my Getting Started with LINQ course where I cover from what LINQ is to its most recent methods and overloads introduced in .NET6. And don’t miss the previous Monday Links on Staging environments, Work and Types.

LINQ Aggregate Method Explained with Pictures

This is not one of the most used LINQ methods. We won’t use it every day. But, it’s handy for some scenarios. Let’s learn how to use the Aggregate method.

The Aggregate method applies a function on a collection carrying the result to the next element. It “aggregates” the result of a function over a collection.

The Aggregate method takes two parameters: a seed and an aggregating function that takes the accumulated value and one element from the collection.

How does Aggregate work?

Let’s reinvent the wheel to understand Aggregate by finding the maximum rating in our movie catalog. Of course, LINQ has a Max method. And, .NET 6 introduced new LINQ methods, among those: MaxBy.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

var maxRating = movies.Aggregate(0f, (maxSoFar, movie) => MaxBetween(maxSoFar, movie.Rating));
//                     ^^^^^^^^^
Console.WriteLine($"Maximum rating on our catalog: {maxRating}");

// Output:
// Comparing 0 and 4.5
// Comparing 4.5 and 4.6
// Comparing 4.6 and 4.7
// Comparing 4.7 and 5
// Comparing 5 and 4
// Comparing 5 and 5
// Maximum rating on our catalog: 5

Console.ReadKey();

float MaxBetween(float maxSoFar, float rating)
{
    Console.WriteLine($"Comparing {maxSoFar} and {rating}");
    return rating > maxSoFar ? rating : maxSoFar;
}

record Movie(string Name, int ReleaseYear, float Rating);

Notice we used Aggregate() with two parameters: 0f as the seed and the delegate (maxSoFar, movie) => MaxBetween(maxSoFar, movie.Rating) as the aggregating function. maxSoFar is the accumulated value from previous iterations, and movie is the current movie while Aggregate iterates over our list. The MaxBetween() method returns the maximum between two numbers.

Notice the order of the debugging messages we printed every time we compare two ratings in the MaxBetween() method.

On the first iteration, the Aggregate() method executes the MaxBetween() aggregating function using the seed (0f) and the first element (“Titanic” with 4.5) as parameters.

Next, it calls MaxBetween() with the previous result (4.5) as the maxSoFar and the next element of the collection (“The Fifth Element” with 4.6f).

In the last iteration, Aggregate() finds the maxSoFar from all previous iterations and the last element (“My Neighbor Totoro” with 5). And it returns the last value of maxSoFar as a result.

In our example, we used Aggregate() with a seed. But, Aggregate() has an overload without it, then it uses the first element of the collection as the seed. Also, Aggregate() has another parameter to transform the result before returning it.

Voilà! That’s how the Aggregate method works. Remember, it returns an aggregated value from a collection instead of another collection. This is one of those methods we don’t use often. I’ve used it only a couple of times. One of them was in my parsing library, Parsinator, to apply a list of modification functions on the same input object here.

If you want to read more about LINQ and its features, check my quick guide to LINQ, five common LINQ mistakes and how to fix them and what’s new in LINQ with .NET6.

Happy coding!

Peeking into LINQ DistinctBy source code

“Don’t use libraries you can’t read their source code.”

That’s a bold statement I found and shared in a past Monday Links episode. Inspired by that, let’s see what’s inside the new LINQ DistinctyBy method.

What DistinctBy does

DistinctBy returns the objects containing unique values based on one of their properties. It works on collections of complex objects, not just on plain values.

DistinctBy is one of the new LINQ methods introduced in .NET 6.

Here’s how to find unique movies by release year.

var movies = new List<Movie>
{
    new Movie("Schindler's List", 1993, 8.9f),
    new Movie("The Lord of the Rings: The Return of the King", 2003, 8.9f),
    new Movie("Pulp Fiction", 1994, 8.8f),
    new Movie("Forrest Gump", 1994, 8.7f),
    new Movie("Inception", 2010, 8.7f)
};

// Here we use the DistinctBy method with the ReleaseYear property
var distinctByReleaseYear = movies.DistinctBy(movie => movie.ReleaseYear);
//                                 ^^^^^^^^^^

foreach (var movie in distinctByReleaseYear)
{
    Console.WriteLine($"{movie.Name}: [{movie.ReleaseYear}]");
}

// Output:
// Schindler's List: [1993]
// The Lord of the Rings: The Return of the King: [2003]
// Pulp Fiction: [1994]
// Inception: [2010]

record Movie(string Name, int ReleaseYear, float Score);

We used the DistinctBy method on a list of movies. We didn’t use it on a list of released years to then find one movie for each unique release year found.

Before looking at DistinctBy source code, how would you implement it?

LINQ DistinctBy source code

This is the source code for the DistinctBy method. Source

Well, it doesn’t look that complicated. Let’s go through it.

1. Iterating over the input collection

First, DistinctBy() starts by checking its parameters and calling DistinctByIterator().

This is a common pattern in other LINQ methods: Checking parameters in one method and then calling a child iterator method to do the actual logic. (See 1. in the image above)

Then, the DistinctByIterator() initializes the underling enumerator of the input collection with a using declaration. The IEnumerable type has a GetEnumerator() method. (See 2.)

The IEnumerator type has:

• a MoveNext() method to advance the enumerator to the next position
• a Current property to hold the element at the current position.

If a collection is empty or if the iterator reaches the end of the collection, MoveNext() returns false. And, when MoveNext() returns true, Current gets updated with the element at that position. Source

Then, to start reading the input collection, the iterator is placed at the initial position of the collection calling MoveNext(). (See 3.) This first if avoids allocating memory by creating a set in the next step if the collection is empty.

2. Keeping only unique values

After that, DistinctByIterator() creates a set with a default capacity and an optional comparer. This set keeps track of the unique keys already found. (See 4.)

The next step is to read the current element and add its key to the set. (See 5.)

If a set doesn’t already contain the same element, Add() returns true and adds it to the set. Otherwise, it returns false. And, when the set exceeds its capacity, the set gets resized. Source

If the current element’s key was added to the set, the element is returned with the yield return keywords. This way, DistinctByIterator() returns one element at a time.

Step 5 is wrapped inside a do-while loop. It runs until the enumerator reaches the end of the collection. (See 6.)

Voilà! That’s the DistinctBy source code. Simple but effective. Not that intimidating, after all. The trick was to use a set.

To learn about LINQ, check my quick guide to LINQ, five common LINQ methods in pictures and what’s new in LINQ with .NET 6.0.

Happy coding!

Four new LINQ methods in .NET 6: Chunk, DistinctBy, Take, XOrDefault

LINQ isn’t a new feature in the C# language. It was released back in C# version 3.0 in the early 2000s.

And, after more than ten years, it was finally updated with the .NET 6 release. These are four of the new LINQ methods and overloads in .NET 6.

.NET 6 introduced new LINQ methods like Chunk, DistinctBy, MinBy, and MaxBy. Also, new overloads to existing methods like Take and FirstOrDefault. Before this update, to use similar features, custom LINQ methods or third-party libraries were needed.

1. Chunk

Chunk splits a collection into buckets or “chunks” of at most the same size. It receives the chunk size and returns a collection of arrays.

Let’s say we want to watch all movies we have in our catalog. But, we can only watch three films on a single weekend. Let’s use the Chunk method for that.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

// Split the movies list into chunks of three movies
var chunksOf3 = movies.Where(movie => movie.Rating > 4.5f)
                      .Chunk(3); // <--

foreach (var chunk in chunksOf3)
{
    PrintMovies(chunk);
}
// Output:
// The Fifth Element,Terminator 2,Avatar
// My Neighbor Totoro

static void PrintMovies(IEnumerable<Movie> movies)
{
    Console.WriteLine(string.Join(",", movies.Select(movie => movie.Name)));
}

record Movie(string Name, int ReleaseYear, float Rating);

We used three recent C# features: the Top-level statements, records from C# 9.0, and global using statements from C# 10.0. We don’t need all the boilerplate code to write a Console application anymore.

Chunk returns “chunks” with at most the given size. It returns fewer elements in the last bucket when there aren’t enough elements in the source collection.

2. DistinctBy

Unlike Distinct, DistinctBy receives a delegate to select the property to use as the comparison key and returns the objects containing the distinct values, not only the distinct values themselves.

Let’s find the movies containing unique ratings, not just the ratings.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

var distinctRatings = movies.DistinctBy(movie => movie.Rating);
PrintMovies(distinctRatings);

// Output:
// Titanic,The Fifth Element,Terminator 2,Avatar,Platoon

Also, there are similar alternatives to existing methods such as MinBy, MaxBy, ExceptBy, IntersectBy, and UnionBy. They work with a delegate to select a property to use as the comparison key and return the “containing” objects, not only the result.

If we take a look at DistinctBy source code to see what’s inside a LINQ method, it’s not that intimidating after all.

3. Take with Ranges

Take receives a range of indexes to pick a slice of the input collection, not only the first consecutive elements. Take with a range replaces Take followed by Skip to choose a slice of elements.

Let’s choose a slice of our catalog using Take with ranges and the index-from-end operator.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

var rangeOfMovies = movies.Take(^5..3);
PrintMovies(rangeOfMovies);

// Output:
// The Fifth Element,Terminator 2

Take(^5..3) selects elements starting from the fifth position from the end (^5) up to the third position from the start (3). We didn’t need to use the Skip method for that.

Now that we have Take with ranges is easier to find the last “n” elements of a collection: Take(^n...).

Let’s find the last three movies on our catalog.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

var lastThreeMovies = movies.Take(^3..);
PrintMovies(lastThreeMovies);

// Output:
// Avatar,Platoon,My Neighbor Totoro

Before this .NET update, we had to use Skip(movies.Count - 3).Take(3). Take did all the work.

4. XOrDefault methods with an optional default value

FirstOrDefault has a new overload to return a default value when the collection is empty or doesn’t have any elements that satisfy the given condition. Other methods with the suffix ‘OrDefault’ have similar overloads.

Let’s find in our catalog of movies a “perfect” film. Otherwise, let’s return our favorite movies from all times.

Before this update, we had to check if FirstOrDefault returned a non-null value or we had to use the LINQ DefaultIfEmpty method. Like this,

var allTimesFavorite = new Movie("Back to the Future", 1985, 5);

// Using the Null-coalescing assignment ??= operator
var favorite = movies.FirstOrDefault(movie => movie.Rating == 10);
favorite ??= allTimesFavorite;
//       ^^^

// Or
// Using the DefaultIfEmpty method
var favorite = movies.Where(movie => movie.Rating == 10)
                     .DefaultIfEmpty(allTimesFavorite)
                     // ^^^^^
                     .First();

With the new overload, we pass a safe default. Like this,

var allTimesFavorite = new Movie("Back to the Future", 1985, 5);

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

// We have a safe default now. See the second parameter
var favorite = movies.FirstOrDefault(
    movie => movie.Rating == 10,
    allTimesFavorite); // <--
    
// We don't need to check for null here
Console.WriteLine(favorite.Name);

// Output:
// Back to the future

Notice the second parameter we passed to the FirstOrDefault method.

To use these new methods and overloads, install the latest version of the .NET SDK from the .NET official page and use as target framework at least .net6 in your project files.

Here’s a sample csproj file for a Console app using .net6,

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <!--             ^^^^^  -->
  </PropertyGroup>

</Project>

Voilà! These are four new LINQ methods released in the .NET 6 updated. FirstOrDefault with a safe default helps us prevent one of the common LINQ mistakes: using the XOrDefault methods without null checking afterwards.

To learn about LINQ and other methods, check my quick guide to LINQ, how to use LINQ GroupBy method and two new LINQ methods in .NET 9.

Happy coding!