Mapping Update and Dynamic Mapping Templates in Elasticsearch

Mapping updates and dynamic mapping templates are important concepts in Elasticsearch for managing data structure within an index. The mapping defines how documents are indexed and stored, including the data type of each field. As data evolves, it's important to update mappings to ensure Elasticsearch can accurately interpret and index documents.

Dynamic mapping templates provide a way to define custom mappings based on specific criteria and offers flexibility in how fields are mapped dynamically.

In this article, we will learn about Mapping Updates and dynamic mapping templates by understanding the Mapping Updates, it's Need, How to Perform Mapping Updates and Dynamic and Dynamic Mapping Templates in detail.

What are Mapping Updates?

• Mapping updates in Elasticsearch refer to the process of modifying the mapping of an existing field or adding new fields to an index.
  
• When we index documents into Elasticsearch, it automatically creates a mapping for those documents based on the data it encounters.
  
• As our data evolves, we may need to update the mapping to accommodate new fields or changes in existing fields.

Need of Mapping Updates

• Mapping updates are important for maintaining the integrity and consistency of our data.
  
• Without proper mapping, Elasticsearch may be unable to index our documents correctly, leading to search issues or data loss.
  
• By updating the mapping, we ensure that Elasticsearch understands the structure of our data accurately, enabling more effective searching and analysis.

How to Perform Mapping Updates?

Performing mapping updates in Elasticsearch is crucial when the data schema evolves or when we need to optimize indexing and querying. Let's understand the process of mapping updates:

1. Identify Mapping Changes: Before updating the mapping, identify the changes needed in the mapping. This could include adding new fields, changing field data types, or updating analyzer settings.
2. Update Index Settings: If necessary, update the index settings to allow for mapping changes. For example, you might need to increase the index.mapping.total_fields.limit setting if you are adding many new fields.
3. Update Mapping: Use the Update Mapping API to apply the changes to the mapping. we can update the mapping for a specific index or all indices.

Example of Updating Mapping for a Specific Index:

PUT /my_index/_mapping
{
  "properties": {
    "new_field": {
      "type": "keyword"
    }
  }
}

In this example, we are updating the mapping for the my_index index to include a new field called new_field with the keyword data type.

Example of Updating Mapping for All Indices:

POST /_all/_mapping
{
  "properties": {
    "new_field": {
      "type": "keyword"
    }
  }
}

This example updates the mapping for all indices in the cluster to include the new_field field.

Dynamic Mapping in Elasticsearch

• Dynamic mapping allows Elasticsearch to dynamically determine the data type of fields based on the JSON documents being indexed.
  
• When Elasticsearch encounters a new field, it analyzes the data and assigns a data type to the field based on its content. This process is known as dynamic field mapping.

Example:

Let's say you have a simple index called my_index and you start indexing documents with different fields

{
  "title": "Elasticsearch Dynamic Mapping",
  "author": "John Doe",
  "views": 1000
}
{
  "title": "Introduction to Elasticsearch",
  "date": "2022-05-26"
}
{
  "title": "Advanced Elasticsearch Techniques",
  "tags": ["elasticsearch", "search", "analytics"]
}

Output:

{
  "took": 10,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "my_index",
        "_type": "_doc",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1
      }
    },
    {
      "index": {
        "_index": "my_index",
        "_type": "_doc",
        "_id": "2",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 1,
        "_primary_term": 1
      }
    },
    {
      "index": {
        "_index": "my_index",
        "_type": "_doc",
        "_id": "3",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 2,
        "_primary_term": 1
      }
    }
  ]
}

In this example, Elasticsearch inferred the data types of the fields based on their content. The "title" field was mapped as text, "author" as keyword, and "views" as long.

Dynamic Mapping Templates

• Dynamic mapping templates in Elasticsearch allow you to define custom mappings that are applied dynamically based on certain criteria.
  
• This is useful when we want to apply different mappings to fields that match specific patterns in document keys or values.

Example:

Suppose we have an index named logs where the document keys follow a specific pattern, such as user_*, and we want to apply a custom mapping to these fields.

Dynamic Mapping Template Definition

PUT _index_template/logs_template
{
  "index_patterns": ["logs*"],
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "user_fields": {
            "match_mapping_type": "string",
            "match": "user_*",
            "mapping": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      ]
    }
  }
}

In this template, we define a dynamic template named user_fields that matches string fields (match_mapping_type: "string") with keys starting with user_*. It specifies that these fields should be mapped as text fields with a keyword sub-field for exact matching.

Applying the Template

PUT /logs
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "dynamic_templates": [
      {
        "user_fields": {
          "match_mapping_type": "string",
          "match": "user_*",
          "mapping": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      }
    ]
  }
}

This request applies the logs_template to the logs index, ensuring that any new documents with keys starting with user_* will have their string fields mapped as specified in the template.

Output

When we index a document with a key that matches the pattern user_*, the dynamic mapping template will be applied.

Example Document

PUT logs/_doc/1
{
  "user_id": "123",
  "user_name": "John Doe"
}

Mapping for the Document

GET logs/_mapping

Output:

{
  "logs": {
    "mappings": {
      "dynamic_templates": [
        {
          "user_fields": {
            "path_match": "user_*",
            "mapping": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "text"
            }
          }
        }
      ]
    }
  }
}

Explanation:

• Dynamic Template Matching: The dynamic template matches fields based on their key names (match: "user_*") and their data type (match_mapping_type: "string").
• Custom Mapping: The template specifies a custom mapping for matched fields, in this case, mapping them as text fields with a keyword sub-field for exact matching.
• Field Transformation: When a document is indexed with a key matching the template, Elasticsearch applies the specified mapping transformation to the field

Conclusion

Overall, mapping updates and dynamic mapping templates play a vital role in Elasticsearch's ability to manage evolving data structures effectively. Mapping updates allow for modifications to existing mappings or addition of new fields, ensuring accurate indexing and search capabilities. Dynamic mapping templates provide a powerful mechanism to define custom mappings based on specific patterns, offering good control over field mapping.