Three set-like LINQ methods: Intersect, Union, and Except

So far we have covered some of the most common LINQ methods. This time let’s cover three LINQ methods that work like set operations: Intersect, Union, and Except.

Like the Aggregate method, we don’t use these methods every day, but they will come in handy from time to time.

1. Intersect

Intersect() finds the common elements between two collections.

Let’s find the movies we both have watched and rated in our catalogs.

var mine = new List<Movie>
{
    // We have not exactly a tie here...
    new Movie("Terminator 2", 1991, 4.7f),
    //        ^^^^^^^^^^^^^^
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("My Neighbor Totoro", 1988, 5f)
    //        ^^^^^^^^^^^^^^^^^^^^
};

var yours = new List<Movie>
{
    new Movie("My Neighbor Totoro", 1988, 5f),
    //        ^^^^^^^^^^^^^^^^^^^^
    new Movie("Pulp Fiction", 1994, 4.3f),
    new Movie("Forrest Gump", 1994, 4.3f),
    // We have not exactly a tie here...
    new Movie("Terminator 2", 1991, 5f)
    //        ^^^^^^^^^^^^^^
};

var weBothHaveSeen = mine.Intersect(yours);
Console.WriteLine("We both have seen:");
PrintMovies(weBothHaveSeen);

// Output:
// We both have seen:
// My Neighbor Totoro

static void PrintMovies(IEnumerable<Movie> movies)
{
    Console.WriteLine(string.Join(",", movies.Select(movie => movie.Name)));
}

record Movie(string Name, int ReleaseYear, float Rating);

This time, we have two lists of movies, mine and yours, with the ones I’ve watched and the ones you have watched, respectively. Also, we both have watched “My Neighbor Totoro” and “Terminator 2.”

To find the movies we both have seen (the intersection between our two catalogs), we used Intersect().

But, our example only shows “My Neighbor Totoro.” What happened here?

If we pay close attention, we both have watched “Terminator 2,” but we gave it different ratings. Since we’re using records from C# 9.0, records have member-wise comparison. Therefore, our two “Terminator 2” instances aren’t exactly the same, even though they have the same name. That’s why Intersect() doesn’t return it.

To find the common movies using only the movie name, we can:

• pass a custom comparer to Intersect(),
• override the default Equals() and GetHashCode() methods of the Movie record, or,
• use the new IntersectBy() method introduced in .NET6.

Let’s use the IntersectBy() method.

var weBothHaveSeen = mine.IntersectBy(
        yours.Select(yours => yours.Name),
        //    ^^^^^^
        // Your movie names
        (movie) => movie.Name);
        //               ^^^^
        // keySelector: Property to compare by

Console.WriteLine("We both have seen:");
PrintMovies(weBothHaveSeen);

// Output:
// We both have seen:
// Terminator 2,My Neighbor Totoro

Unlike Intersect(), IntersectBy() expects a “keySelector,” a delegate with the property to use as the comparing key, and a second collection with the same type as the keySelector.

2. Union

Union() finds the elements from both collections without duplicates.

Let’s find all the movies we have in our catalogs.

var mine = new List<Movie>
{
    new Movie("Terminator 2", 1991, 5f),
    //        ^^^^^^^^^^^^^^
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("My Neighbor Totoro", 1988, 5f)
    //        ^^^^^^^^^^^^^^^^^^^^
};

var yours = new List<Movie>
{
    new Movie("My Neighbor Totoro", 1988, 5f),
    //        ^^^^^^^^^^^^^^^^^^^^
    new Movie("Pulp Fiction", 1994, 4.3f),
    new Movie("Forrest Gump", 1994, 4.3f),
    new Movie("Terminator 2", 1991, 5f)
    //        ^^^^^^^^^^^^^^
};

var allTheMoviesWeHaveSeen = mine.Union(yours);
Console.WriteLine("All the movies we have seen:");
PrintMovies(allTheMoviesWeHaveSeen);

// Output:
// All the movies we have seen:
// Terminator 2,Titanic,The Fifth Element,My Neighbor Totoro,Pulp Fiction,Forrest Gump

static void PrintMovies(IEnumerable<Movie> movies)
{
    Console.WriteLine(string.Join(",", movies.Select(movie => movie.Name)));
}

record Movie(string Name, int ReleaseYear, float Rating);

This time we gave the same rating to our shared movies: “Terminator 2” and “My Neighbor Totoro.” And, Union() showed all the movies from both collections, showing duplicates only once.

Union() works the same way as the union operation in our Math classes.

LINQ has a similar method to “combine” two collections into a single one: Concat(). But, unlike Union(), Concat() returns all elements from both collections without removing the duplicated ones.

.NET 6.0 also has a UnionBy() method to “union” two collections with a keySelector. And, unlike IntersectBy(), we don’t need the second collection to have the same type as the keySelector.

3. Except

Except() finds the elements in one collection that are not present in another one.

This time, let’s find the movies only I have watched.

var mine = new List<Movie>
{
    new Movie("Terminator 2", 1991, 5f),
    new Movie("Titanic", 1998, 4.5f),
    //         ^^^^^^^
    new Movie("The Fifth Element", 1997, 4.6f),
    //         ^^^^^^^^^^^^^^^^^
    new Movie("My Neighbor Totoro", 1988, 5f)
};

var yours = new List<Movie>
{
    new Movie("My Neighbor Totoro", 1988, 5f),
    new Movie("Pulp Fiction", 1994, 4.3f),
    new Movie("Forrest Gump", 1994, 4.3f),
    new Movie("Terminator 2", 1991, 5f)
};

var onlyIHaveSeen = mine.Except(yours);
Console.WriteLine();
Console.WriteLine("Only I have seen:");
PrintMovies(onlyIHaveSeen);

// Output:
// Only I have seen:
// Titanic,The Fifth Element

static void PrintMovies(IEnumerable<Movie> movies)
{
    Console.WriteLine(string.Join(",", movies.Select(movie => movie.Name)));
}

record Movie(string Name, int ReleaseYear, float Rating);

With Except(), we found the movies in mine that are not in yours.

When working with Except(), we should pay attention to the order of the collection because this method isn’t commutative. This means, mine.Except(yours) is not the same as yours.Except(mine).

Likewise, we have ExceptBy() that receives a KeySelector and a second collection with the same type as the keySelector type.

Voilà! These are the Intersect(), Union(), and Except() methods. They work like the Math set operations: intersection, union, and symmetrical difference, respectively. Of the three, I’d say Except is the most common method.

If you want to read more about LINQ, check my quick guide to LINQ, five common LINQ mistakes and how to fix them and what’s new in LINQ with .NET6.

Happy coding!

NCache & Full-Text Search

I bet you have used the SQL LIKE operator to find a keyword in a text field. For large amounts of text, that would be slow. Let’s learn how to implement a full-text search with Lucene and NCache.

What is Full-Text Search?

Full-text search is a technique to search not only exact matches of a keyword in some text but for patterns of text, synonyms, or close words in large amounts of text.

To support large amounts of text, searching is divided into two phases: indexing and searching. In the indexing phase, an analyzer processes text to create indexes based on the rules of a spoken language like English to remove stop words and record synonyms and inflections of words. Then, the searching phase only uses the indexes instead of the original text source.

Full-Text Search with Lucene and NCache

1. Why Lucene and NCache?

From its official page, “Apache Lucene.NET is a high performance search library for .NET.” It’s a C# port of Java-based Apache Lucene, an “extremely powerful” and fast search library optimized for full-text search.

NCache gives distributed capabilities to Lucene by implementing the Lucene API on top of its In-Memory Distributed cache. This way, NCache makes Lucene a linearly scalable full-text searching solution for .NET. For more features of Distributed Lucene, check NCache Distributed Lucene page.

2. Create a Lucene Cache in NCache

We have already installed and used NCache as a IDistributedCache provider. This time, let’s use NCache version 5.3 to find movies by title or director name using Lucene’s full-text search.

Lucene stores data in immutable “segments,” which consist of multiple files. We can store these segments in our local file system or in RAM. But, since we’re using Lucene with NCache, we’re storing these segments in NCache.

Before indexing and searching anything, first, we need to create a Distributed Lucene Cache. Let’s navigate to http://localhost:8251 to fire NCache Web Manager and add a New Distributed Cache.

Let’s select “Distributed Lucene” in the Store Type and give it a name. Then, let’s add our own machine and a second node. For write operations, we need at least two nodes. We can stick to the defaults for the other options.

By default, in Windows machines, NCache stores Lucene indexes in C:\ProgramData\ncache\lucene-index.

For more details about these installation options, check NCache official docs.

3. Index Some Movies

After creating the Distributed Lucene cache, let’s populate our Lucene indexes with some movies from a Console app. Later, we will search them from another Console app.

First, let’s create a Console app to load some movies to the Lucene Cache. Also, let’s install the Lucene.Net.NCache NuGet package.

In the Program.cs file, we could load all movies we want to index from a database or another store. For example, let’s use a list of movies from IMDb. Something like this,

using SearchMovies.Shared;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Services;

var searchService = new SearchService(Config.CacheName);
searchService.LoadMovies(SomeMoviesFromImdb());

Console.WriteLine("Press any key to continue...");
Console.ReadKey();

// This list of movies was taken from IMDb dump
// See: https://www.imdb.com/interfaces/
static IEnumerable<Movie> SomeMoviesFromImdb()
{
    return new List<Movie>
    {
        new Movie("Caged Fury", 1983, 3.8f, 89, new Director("Maurizio Angeloni", 1959), new []{ Genre.Crime,Genre.Drama  }),
        new Movie("Bad Posture", 2011, 6.5f, 93, new Director("Jack Smith", 1932), new []{ Genre.Drama,Genre.Romance  }),
        new Movie("My Flying Wife", 1991, 5.5f, 91, new Director("Franz Bi", 1899), new []{ Genre.Action,Genre.Comedy,Genre.Fantasy  }),
        new Movie("Modern Love", 1990, 5.2f, 105, new Director("Sophie Carlhian", 1962), new []{ Genre.Comedy  }),
        new Movie("Sins", 2012, 2.3f, 84, new Director("Pierre Huyghe", 1962), new []{ Genre.Action, Genre.Thriller  })
        // Some other movies here...
    };
}

Notice we used a SearchService to handle the index creation in a method called LoadMovies(). Let’s take a look at it.

using Lucene.Net.Analysis.Standard;
using Lucene.Net.Index;
using Lucene.Net.Store;
using Lucene.Net.Util;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Extensions;

namespace SearchMovies.Shared.Services;

public class SearchService
{
    private const string IndexName = "movies";
    private const LuceneVersion luceneVersion = LuceneVersion.LUCENE_48;

    private readonly string _cacheName;

    public SearchService(string cacheName)
    {
        _cacheName = cacheName;
    }

    public void LoadMovies(IEnumerable<Movie> movies)
    {
        using var indexDirectory = NCacheDirectory.Open(_cacheName, IndexName);
        // 1. Opening directory    ^^^

        var standardAnalyzer = new StandardAnalyzer(luceneVersion);
        var indexConfig = new IndexWriterConfig(luceneVersion, standardAnalyzer)
        {
            OpenMode = OpenMode.CREATE
        };
        using var writer = new IndexWriter(indexDirectory, indexConfig);
        // 2. Creating a writer   ^^^

        foreach (var movie in movies)
        {
            var doc = movie.MapToLuceneDocument();
            writer.AddDocument(doc);
            //     ^^^^^^^^^^^
            // 3. Adding a document
        }

        writer.Commit();
        //     ^^^^^^
        // 4. Writing documents
    }
}

A bit of background first, Lucene uses documents as the unit of search and index. Documents can have many fields, and we don’t need a schema to store them.

We can search documents using any field. Lucene will only return those with that field and matching data. For more details on some Lucene internals, check its Lucene Quick Start guide.

Notice we started our LoadMovies by opening an NCache directory. We needed the same cache name we configured before and an index name. Then we created an IndexWriter with our directory and some configurations, like a Lucene version, an analyzer, and an open mode.

Then, we looped through our movies and created a Lucene document for each one using the MapToLuceneDocument() extension method. Here it is,

using Lucene.Net.Documents;
using SearchMovies.Shared.Entities;

namespace SearchMovies.Shared.Extensions;

public static class MoviesExtensions
{
    public static Document MapToLuceneDocument(this Movie self)
    {
        return new Document
        {
            new TextField("name", self.Name, Field.Store.YES),
            new TextField("directorName", self.Director.Name, Field.Store.YES)
        };
    }
}

To create Lucene documents, we used two fields of type TextField: movie name and director name. For each field, we need a name and a value to index. We will use the field names later to create a response object from search results.

There are two basic field types for Lucene documents: TextField and StringField. The first one has support for Full-Text search and the second one supports searching for exact matches.

Once we called the Commit() method, NCache stored our movies in a distributed index.

4. Full-Text Searching Movies

Now that we populated our index with some movies, to search them, let’s create another Console app to read a Lucene query.

Again, let’s use the same SearchService, this time with a SearchByNames() method passing a Lucene query.

using Lucene.Net.Analysis.Standard;
using Lucene.Net.Index;
using Lucene.Net.QueryParsers.Classic;
using Lucene.Net.Search;
using Lucene.Net.Store;
using Lucene.Net.Util;
using SearchMovies.Shared.Entities;
using SearchMovies.Shared.Extensions;
using SearchMovies.Shared.Responses;

namespace SearchMovies.Shared.Services;

public class SearchService
{
    // Same SearchService as before...

    public IEnumerable<MovieResponse> SearchByNames(string searchQuery)
    {
        using var indexDirectory = NCacheDirectory.Open(_cacheName, IndexName);
        using var reader = DirectoryReader.Open(indexDirectory);
        //                 ^^^^^^^^^^^^^^^
        // 1. Creating a reader
        var searcher = new IndexSearcher(reader);

        var analyzer = new StandardAnalyzer(luceneVersion);
        var parser = new QueryParser(luceneVersion, "name", analyzer);
        var query = parser.Parse(searchQuery);
        //          ^^^^^^
        // 2. Parsing a Lucene query 

        var documents = searcher.Search(query, 10);
        //              ^^^^^^^^
        // 3. Searching documents

        var result = new List<MovieResponse>();
        for (int i = 0; i < documents.TotalHits; i++)
        {
            var document = searcher.Doc(documents.ScoreDocs[i].Doc);
            result.Add(document.MapToMovieResponse());
            //     ^^^
            // 4. Populating a result object
        }

        return result;
    }
}

This time, instead of creating an IndexWriter, we used a DirectoryReader and a query parser with the same Lucene version and analyzer. Then, we used the Search() method with the parsed query and a result count. The next step was to loop through the results and create a response object.

To create a response object from a Lucene document, we used the MapToMovieResponse(). Here it is,

public static MovieResponse MapToMovieResponse(this Document self)
{
    return new MovieResponse(self.Get("name"), self.Get("directorName"));
}

This time, we used the Get() method with the same field names as before to retrieve fields from documents.

For example, let’s find all movies whose director’s name contains “ca”, with the query directorName:ca*,

Of course, there are more keywords in Lucene Query Syntaxt than the ones we used here.

Voilà! That’s how to use Distributed Lucene with NCache. If we already have an implementation with Lucene.NET, we would need few code changes to migrate it to Lucene with NCache. Also, notice that NCache doesn’t implement all Lucene methods.

To follow along with the code we wrote in this post, check my Ncache Demo repository over on GitHub.

To read more content, check my post Working with ASP.NET Core IDistributedCache Provider for NCache to learn about caching with ASP.NET Core and NCache.

I wrote this post in collaboration with Alachisoft, NCache creators.

Happy coding!

Monday Links: Storytelling, Leet Code and Boredom

In case you find the Monday Links series for the first time: these are five links from past weeks that I found interesting (and worth sharing) while procastinating surfing the Web. This is not a link-building scheme, I only read and liked these articles.

The Secret Art of Storytelling in Programming by Yehonathan Sharvit

This presentation starts with the author sharing his struggle to read books as a kid. And later read code as a programmer and contracts as a consultant.

The main message from this presentation is how memory, attention, and structure spans relate to coding. The author presents three coding style principles that respect mind spans:

1. Use small functions
2. Make every line in a function have the same level of abstraction
3. Use descriptive names instead of comments

Watch full presentation

Stop Interviewing With Leet Code

Interviewing is broken. We all agree. But we don’t know how to fix it. Brain teasers, IQ tests, pair programming, algorithms? This post presents an alternative: inspect the candidate GitHub and public work, ask him to review some piece of code, add unit tests or do some refactors. That sounds like a better idea! Read full article

How to professionally say…

One of the things we learn while working for a company is office politics. Basically, how to say things and to step away from certain situations. I learned from a coworker to say “I don’t have enough information to answer that question” instead of a simple “I don’t know.” You will find more lines like that one in this post. Read full article

How to feel engaged at work: a software engineer’s guide

Let’s be honest. It’s rewarding when we see the impact of our work. But, often, all days look almost the same. Another JIRA ticket for a production issue. Another meeting that could have been an email. This article shows four ideas to spice things up. Read full article

The Toxic Grind

This article talks about success, hard work, and work-life balance. This is my favorite line: “We should glorify the journey of achieving something meaningful, not the dream of wealth and power. Glorify the skills you build along the way, not the shortcuts you take.” Read full article

Voilà! Another Monday Links. What do you do to feel engaged at your work? Have you been asked to solve LeetCode questions during interviews? Would you like to do something different in future interviews?

In the meantime, check my Getting Started with LINQ course where I cover from what LINQ is to its most recent methods and overloads introduced in .NET6. And don’t miss the previous Monday Links on Staging environments, Work and Types.

LINQ Aggregate Method Explained with Pictures

This is not one of the most used LINQ methods. We won’t use it every day. But, it’s handy for some scenarios. Let’s learn how to use the Aggregate method.

The Aggregate method applies a function on a collection carrying the result to the next element. It “aggregates” the result of a function over a collection.

The Aggregate method takes two parameters: a seed and an aggregating function that takes the accumulated value and one element from the collection.

How does Aggregate work?

Let’s reinvent the wheel to understand Aggregate by finding the maximum rating in our movie catalog. Of course, LINQ has a Max method. And, .NET 6 introduced new LINQ methods, among those: MaxBy.

var movies = new List<Movie>
{
    new Movie("Titanic", 1998, 4.5f),
    new Movie("The Fifth Element", 1997, 4.6f),
    new Movie("Terminator 2", 1991, 4.7f),
    new Movie("Avatar", 2009, 5),
    new Movie("Platoon", 1986, 4),
    new Movie("My Neighbor Totoro", 1988, 5)
};

var maxRating = movies.Aggregate(0f, (maxSoFar, movie) => MaxBetween(maxSoFar, movie.Rating));
//                     ^^^^^^^^^
Console.WriteLine($"Maximum rating on our catalog: {maxRating}");

// Output:
// Comparing 0 and 4.5
// Comparing 4.5 and 4.6
// Comparing 4.6 and 4.7
// Comparing 4.7 and 5
// Comparing 5 and 4
// Comparing 5 and 5
// Maximum rating on our catalog: 5

Console.ReadKey();

float MaxBetween(float maxSoFar, float rating)
{
    Console.WriteLine($"Comparing {maxSoFar} and {rating}");
    return rating > maxSoFar ? rating : maxSoFar;
}

record Movie(string Name, int ReleaseYear, float Rating);

Notice we used Aggregate() with two parameters: 0f as the seed and the delegate (maxSoFar, movie) => MaxBetween(maxSoFar, movie.Rating) as the aggregating function. maxSoFar is the accumulated value from previous iterations, and movie is the current movie while Aggregate iterates over our list. The MaxBetween() method returns the maximum between two numbers.

Notice the order of the debugging messages we printed every time we compare two ratings in the MaxBetween() method.

On the first iteration, the Aggregate() method executes the MaxBetween() aggregating function using the seed (0f) and the first element (“Titanic” with 4.5) as parameters.

Next, it calls MaxBetween() with the previous result (4.5) as the maxSoFar and the next element of the collection (“The Fifth Element” with 4.6f).

In the last iteration, Aggregate() finds the maxSoFar from all previous iterations and the last element (“My Neighbor Totoro” with 5). And it returns the last value of maxSoFar as a result.

In our example, we used Aggregate() with a seed. But, Aggregate() has an overload without it, then it uses the first element of the collection as the seed. Also, Aggregate() has another parameter to transform the result before returning it.

Voilà! That’s how the Aggregate method works. Remember, it returns an aggregated value from a collection instead of another collection. This is one of those methods we don’t use often. I’ve used it only a couple of times. One of them was in my parsing library, Parsinator, to apply a list of modification functions on the same input object here.

If you want to read more about LINQ and its features, check my quick guide to LINQ, five common LINQ mistakes and how to fix them and what’s new in LINQ with .NET6.

Happy coding!

Peeking into LINQ DistinctBy source code

“Don’t use libraries you can’t read their source code.”

That’s a bold statement I found and shared in a past Monday Links episode. Inspired by that, let’s see what’s inside the new LINQ DistinctyBy method.

What DistinctBy does

DistinctBy returns the objects containing unique values based on one of their properties. It works on collections of complex objects, not just on plain values.

DistinctBy is one of the new LINQ methods introduced in .NET 6.

Here’s how to find unique movies by release year.

var movies = new List<Movie>
{
    new Movie("Schindler's List", 1993, 8.9f),
    new Movie("The Lord of the Rings: The Return of the King", 2003, 8.9f),
    new Movie("Pulp Fiction", 1994, 8.8f),
    new Movie("Forrest Gump", 1994, 8.7f),
    new Movie("Inception", 2010, 8.7f)
};

// Here we use the DistinctBy method with the ReleaseYear property
var distinctByReleaseYear = movies.DistinctBy(movie => movie.ReleaseYear);
//                                 ^^^^^^^^^^

foreach (var movie in distinctByReleaseYear)
{
    Console.WriteLine($"{movie.Name}: [{movie.ReleaseYear}]");
}

// Output:
// Schindler's List: [1993]
// The Lord of the Rings: The Return of the King: [2003]
// Pulp Fiction: [1994]
// Inception: [2010]

record Movie(string Name, int ReleaseYear, float Score);

We used the DistinctBy method on a list of movies. We didn’t use it on a list of released years to then find one movie for each unique release year found.

Before looking at DistinctBy source code, how would you implement it?

LINQ DistinctBy source code

This is the source code for the DistinctBy method. Source

Well, it doesn’t look that complicated. Let’s go through it.

1. Iterating over the input collection

First, DistinctBy() starts by checking its parameters and calling DistinctByIterator().

This is a common pattern in other LINQ methods: Checking parameters in one method and then calling a child iterator method to do the actual logic. (See 1. in the image above)

Then, the DistinctByIterator() initializes the underling enumerator of the input collection with a using declaration. The IEnumerable type has a GetEnumerator() method. (See 2.)

The IEnumerator type has:

• a MoveNext() method to advance the enumerator to the next position
• a Current property to hold the element at the current position.

If a collection is empty or if the iterator reaches the end of the collection, MoveNext() returns false. And, when MoveNext() returns true, Current gets updated with the element at that position. Source

Then, to start reading the input collection, the iterator is placed at the initial position of the collection calling MoveNext(). (See 3.) This first if avoids allocating memory by creating a set in the next step if the collection is empty.

2. Keeping only unique values

After that, DistinctByIterator() creates a set with a default capacity and an optional comparer. This set keeps track of the unique keys already found. (See 4.)

The next step is to read the current element and add its key to the set. (See 5.)

If a set doesn’t already contain the same element, Add() returns true and adds it to the set. Otherwise, it returns false. And, when the set exceeds its capacity, the set gets resized. Source

If the current element’s key was added to the set, the element is returned with the yield return keywords. This way, DistinctByIterator() returns one element at a time.

Step 5 is wrapped inside a do-while loop. It runs until the enumerator reaches the end of the collection. (See 6.)

Voilà! That’s the DistinctBy source code. Simple but effective. Not that intimidating, after all. The trick was to use a set.

To learn about LINQ, check my quick guide to LINQ, five common LINQ methods in pictures and what’s new in LINQ with .NET 6.0.

Happy coding!