In today’s world, applications and services churn out a huge amount of data continuously. And if you’re dealing with time-series data like logs or audit data points then you need to consider how to get rid of it after it becomes old and unwanted. The best way to achieve this is using Elasticsearch Rollover Index.

What is a Rollover Index?

When a new Elasticsearch index is created automatically for write operations as soon as the previous one becomes passive, it’s called a rollover index.
Only the newly created index will be available for write operations and the older indices will become read-only. All these indices have to be under one alias so as to allow the user to read and write on the same Elasticsearch alias without worrying about handling multiple indices.

But the question still remains. How to go about creating these magical indices which will handle all these complex operations automatically?

Let’s go through this process using default Elasticsearch APIs along with its equivalent OpenDistro APIs as its something which is used by many industry level applications because of its ability to provide enterprise-grade features, security and analytics tools.

1. Index lifecycle management (ILM) policy

Creating ILM Policy is the first step for kicking off our rollover index.

ILM policy or Index State Management (ISM) policy in the case of OpenDistro, is a set of rules which defines the behavior of an index over a period of time.
Any ILM Policy can have 5 different phases defined, namely hot, warm, cold, frozen and delete. You can configure these phases according to your needs of the lifecycle of an index.

Below is an example of such a policy :

PUT /_ilm/policies/policy_name
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10gb"
          }
        }
      },
      "warm": {
        "actions": {
          "min_age": "0d",
          "read_only": {},
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age ": "240d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Hot Phase

In the hot phase, a new index will be created once the size of the existing index will reach 10 GB(rollover condition). There are multiple rollover conditions that you can configure and can be referred to from here.
There should be at least one rollover condition in the hot phase.

Warm Phase

Index will be passed onto the warm phase just after coming from the hot phase since the minimum age defined here is 0 days. Once in the warm phase, the index will become read-only as defined in the actions and the number of shards will also decrease to 1.
There are many actions that can be configured in the policy and can be referred to from here.

Delete Phase

Index will be passed onto the delete phase once it’s 240 days old and will be deleted.

Note: The mentioned policy doesn’t have cold or frozen phases configured. You can remove or add these phases according to your needs.

As promised, I have also provided OpenDistro equivalent of the above policy below :

PUT  /_opendistro/_ism/policies/policy_name
{
  "description": "index_lifecycle_policy",
  "default_state": "hot",
  "states": [
    {
      "name": "hot",
      "actions": [
        {
          "rollover": {
            "min_size": "10gb "
          }
        }
      ],
      "transitions": [
        {
          "state_name": "warm"
        }
      ]
    },
    {
      "name": "warm",
      "actions": [
        {
          "read_only": {},
          "shrink": {
            "number_of_shards": 1
          }
        }
      ],
      "transitions": [
        {
          "state_name": "delete",
          "conditions": {
            "min_index _age": "240d"
          }
        }
      ]
    },
    {
      "name": "delete",
      "actions": [
        {
          "delete": {}
        }
      ],
      "transitions": []
    }
  ],
  "ism_template": [
    {
      "index_patterns": [
        "index-name-*"
      ]
    }
  ]
}

Note : There are no predefined names of the phases in OpenDistro’s ISM Policy. Also, there is an extra index name pattern needed here. Any index whose name has this pattern will follow this policy.

2. Index Templates

After configuring an ILM Policy, let’s proceed to create an index template.

Index Template is a set of rules that is referred to when creating an index automatically if rollover happens. An Index Template contains mapping, settings and index name pattern.

Below is an example of Index Pattern :

PUT  /_index_template/index_template_name
{
  "index_patterns": [
    "index-name-*"
  ],
  "settings": {
    "refresh_interval": "5s",
    "number_of_shards": "5",
    "number_of_replicas": "1",
    "index.lifecycle.name ": "lifecycly_policy_name",
    "index.lifecycle.rollover_alias": "index_alias_name"
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "name": {
        "type": "long"
      },
      "version": {
        "type": "integer"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Below is an example of Index Pattern in OpenDistroAPI :

PUT  /_template/vuh_index_template
{
  "index_patterns": [
    "index-name-*"
  ],
  "settings": {
    " refresh_interval": "5s",
    "number_of_shards": "5",
    "number_of_replicas": "1",
    "opendistro.index_state_management.rollover_alias": "index_alias_name"
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "name": {
        "type": "long"
      },
      "version": {
        "type": "integer"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Note : There is an ILM Policy name provided in the default Elasticsearch index template but not in OpenDistro because in OpenDistro the policy is attached to the index based on the name pattern which is defined in the policy.

3. Creating First Index

The first index which will follow the policy has to be created manually by the user and successive indices will be created by elasticsearch automatically based on the ILM policy and index template configured. Although, there are a few important points that we should take care of while creating it.

  • Mentioning alias while creating this first index is required.
  • The name of the index must match the pattern that is provided in the policy or template.
  • The mapping of the index should be the same as provided in the template.
  • The name of the index should end with a numeric value so as to allow elasticsearch to create successive indices with an increased numeric value.

Here is an example of such first index:

PUT /index-name-000001
{
  "aliases": {
    "index_alias_name": {
      "is_write_index": true
    }
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "name": {
        "type": "long"
      },
      "version": {
        "type": "integer"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Voila, we have created our rollover index. Now, it’s up to Elasticsearch to handle these indices.

While reading or writing, use only the alias name as the index name and Elasticsearch will take care of writing only in the currently active index and while reading it will consider all the indices which are existing.

4. Explain Lifecycle

At last, Elasticsearch has provided an API for us to check the current status of indices. It tells us about the current phase and state of all the indices present under ILM policy.

Below mentioned is the Explain Lifecycle API in default Elasticsearch

GET /_ilm/explain

Below mentioned is the Explain Lifecycle API in OpenDistro Elasticsearch

GET /_opendistro/_ism/explain

I tried to ease out the process to create rollover indices but if you want to read more about it, you can always refer to Elasticsearch’s official documentation here or OpenDistro’s official documentation here.

Thanks For Reading!