Searchkick

We use Searchkick for full text search. Searchkick is a library that integrates ElasticSearch or OpenSearch (ES/OS) with ActiveRecord, enhancing it with many features. You may want to read its docs before keeping on with the reading.

A former knowledge of ElasticSearch or OpenSearch is a great help when dealing with them and with Searchkick. For any questions, don't hesitate to ask around.

Next index management

The Problem

While developing Warehouse products full text search, when we deployed changes to the Searchkick index, we got the products index page, which is the main entrypoint of the Warehouse section, returning 500s because of the newly deployed application trying to use the previous application's Searchkick index version, which didn't contain the settings required by the new application. Searchkick indices of some models (mainly Product) can take quite some time to perform a .reindex (even more than 30 minutes), so it wasn't acceptable having the main entrypoint down during all that time. We needed a way to create the new index (referred as the next index from now on) before actually deploying it.

We tried to implement a solution that creates the next index before the deployment at platform level, but at the time of writing we couldn't figure out an easy way, given the infrastructure setup, to deploy the Docker image of the yet-to-be-deployed application just to create the next index.

The Solution

So we came up with an implementation residing in the SearchkickUtils namespace, whose main character is the SearchkickUtils::WithNextIndexSettings model's concern.

Suppose we have a model calling the .searchkick method to define a Searchkick index:

model Product
  searchkick settings: { ... }, mappings: { ... }
end

Now we want update the settings, but preventing the old application instances using the new index, or the new ones using the old one. We can make a preparatory release with the following code:

model Product
  include SearchkickUtils::WithNextIndexSettings

  current_settings = { ... }
  next_settings = { index_prefix: 'v1', ... }

  searchkick_with_next_index_settings(current_settings, next_settings)
end

The index_prefix value points to the index version. It must comply with the versioning format, that is:

• nil for the first version
• "v1" for the second version
• "v#{i}" for the next versions, where i is an integer number greater by one than the previous version

nil was chosen for the first version to support brand new Searchkick indices, that are usually created without :index_prefix.

You'll get an error if the actual next settings version doesn't match with the expected one.

While working to a pull request containing index changes, you might want to export SEARCHKICK_NEXT=true. This way you can develop against the updated settings without compromising the index used by main.

After its deployment, we can run the maintenance task MaintenanceTask::SearchkickUtils::CreateNextIndex passing the model_name_as_string argument (in this case Product), either by console or UI.

If the task terminated successfully, ES/OS should have one or more indices matching the products_ pattern, that are the current indices, and one index with the v1_products_ pattern, which is the next one. Additionally, if the next index version reached "v3" or more, it cleans up the old indices using the service SearchkickUtils::CleanUpOldIndicesService. This service deletes the indices of two versions ago starting from the current one, preventing the old indices from piling up.

Now we can deploy the codebase pointing to the next index, which is going to be the current one:

model Product
  current_settings = { index_prefix: 'v1', ... }

  searchkick(**current_settings)
end

We should now run a reindex to sync the current index with the changes that happened between the two releases. We can achieve that using the maintenance task MaintenanceTask::SearchkickUtils::Reindex passing the model_name_as_string argument. Optionally, the updated_after argument may be provided to perform a partial reindex that involves only records updated after the updated_after date. This is handy for Searchkick models for which a whole model reindex is too expensive in terms of time and possibly resources, such as Product. The updated_after argument is set by default to prevent accidental reindexing of the whole model. If you need to reindex the whole model, you can set updated_after to nil.

Reverting a release pointing to the previous index

Since the indices pointing to the previous version are always preserved, we can seamlessly deploy a release reverting a pull request containing an index version bump, caring of running the maintenance task MaintenanceTask::SearchkickUtils::Reindex passing the model name argument to sync the index document changes.

F.A.Q.

When does a model need a reindex?

• when the searchkick gem gets updated
• when the .search_data method gets updated

See https://github.com/ankane/searchkick#to-reindex-or-not-to-reindex for further information.

When to bump an index version?

Safe answer: every time a model's .searchkick method arugments get updated. Here in TWPro there isn't much knowledge about ES/OS, so this rule prevents any possible issues due to unexpected breaking changes.

How to deploy Searchkick changes?

Deploy Searchkick index changes

How to perform a reindex?

Deploy Searchkick index changes -> Reindex

How to bump an index version?

See SearchkickUtils::WithNextIndexSettings model's concern to learn how to create the next version of a model's Searchkick index.

How to deploy a new index version?

Deploy Searchkick index changes -> Create next index