Changing Index Mapping & Making Long Texts Keyword For Elasticsearch

[Ethan777100 (Ethan Chia)]

Moving forward, should I find a need to change my index mapping after the data is ingested,

I have this description column which I realised was ingested as Text and not a keyword.

PUT /ats-mainline-logs-2023-01,ats-mainline-logs-2023-02,ats-mainline-logs-2023-03,ats-mainline-logs-2023-04,ats-mainline-logs-2023-05,ats-mainline-logs-2023-06/_mapping
{
    "properties": {
      "description": {
        "type": "text",
        "fields": {
          "raw": { 
            "type":  "keyword"
          }
        }
      }
    }
}

{
  "acknowledged": true
}

The modification succeeded on Console.

Followed fields | Elasticsearch Guide [8.11] | Elastic

Okay - the reason was coz I had idea to do a visualisation that is Time-Based but bar-graphs the number of occurances of that specific event identified via Description

I was intending to use this column to identify specific events to visualise. No other keyword column in my data is useful to identify events.

Sample descrption

Auto Norm: Vehicle overrun detected 
Auto Norm: Vehicle performed handover with one radio only
Auto Norm: Vehicle Readiness State transition failure

My question now is whether the Indexes will take time to re-index my data in the backend - how to check progress? when will it be done.

UPDATE: Even when I reuploaded my index with the updated mapping, I still did not see Description being sortable / filterable.

I did see a description.raw in the Expanded document Json

   "description.raw": [
      "Summary of xxx with xxx"
    ],
    "description": [
      "Summary of xxx with xxx"

I imagined having the capability to do sort / filter based on Description when I prepare my Visualisation

From general research, I am aware that people do ask why aggregate long texts. Would like to understand a little better on the intricacies.

For my case, I feel based on my data, the only column that allows me to identify the category of events/faults is only from this Description column.

[stephenb (Stephen Brown)]

@Ethan777100

Moving forward, should I find a need to change my index mapping after the data is ingested,

In general You can not change mapping after data is added.

@Ethan777100

The modification succeeded on Console

Right, what happened is you just added to the mapping which you can do... You cannot change an existing field mapping.

You can add to the mapping but it didn't change the underlying data. It doesn't automatically make that field available in the new mapping because you've already indexed data. It's just an empty mapping at that point.

You're going to need to re-index your data if you want that to be a keyword.

The mapping you showed above. This was called multi- field and what it means is you have both the text and the keyword

You just access the keyword by using

description.raw

You can filter or aggregate using that.

You can have both texts and keyword. But if you just want it as a keyword then just make it keyword...

If you want to do full text search and filtering an aggregation do the mapping like you showed about. Typically where you have the word raw we would use keyword ... As both the type and the subfield name

[Ethan777100 (Ethan Chia)]

I think I faced incompatability errors when I tried to change the pipeline or index template to accept description as a "keyword"

This was earlier today.

Console rejected me. Unless I was doing smth incorrectly?

[stephenb (Stephen Brown)]

@Ethan777100

I think I faced incompatability errors when I tried to change the pipeline or index template to accept description as a "keyword"

This was earlier today.

Console rejected me. Unless I was doing smth incorrectly?

No clue what that means unless you show me...

Changes the mapping to keyword, the pipeline should not need to change...

This template only changing the description to keyword works fine...

You have to clean up and start over... this is why I keep saying do 1 file until it is perfect then load many.

Changing / adding a mapping DOES NOT change the data that is already ingested.

This template works with no changes to the pipeline

DELETE _index_template/ats-event-template

PUT _index_template/ats-event-template
{
  "index_patterns": [
    "ats-events-*"
  ],
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "strict_date_optional_time||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.S"
        },
        "alarm": {
          "type": "keyword"
        },
        "alarmvalue": {
          "type": "long"
        },
        "description": {
          "type": "keyword"
        },
        "equipment": {
          "type": "keyword"
        },
        "eventtype": {
          "type": "keyword"
        },
        "graphicelement": {
          "type": "long"
        },
        "id": {
          "type": "long"
        },
        "location": {
          "type": "keyword"
        },
        "mmsstate": {
          "type": "long"
        },
        "operator": {
          "type": "keyword"
        },
        "severity": {
          "type": "long"
        },
        "sourcetime": {
          "type": "date",
          "format": "strict_date_optional_time||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.S"
        },
        "state": {
          "type": "long"
        },
        "subsystem": {
          "type": "keyword"
        },
        "system": {
          "type": "keyword"
        },
        "uniqueid": {
          "type": "keyword"
        },
        "value": {
          "type": "keyword"
        },
        "zone": {
          "type": "long"
        }
      }
    }
  }
}


# Ran the logstash

GET ats-events-2023-06

GET ats-events-2023-06/_search

# GET ats-events-2023-06 200 OK
{
  "ats-events-2023-06": {
    "aliases": {},
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "strict_date_optional_time||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.S"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "alarm": {
          "type": "keyword"
        },
        "alarmvalue": {
          "type": "long"
        },
        "description": {
          "type": "keyword"
        },
        "equipment": {
          "type": "keyword"
        },
        "event": {
          "properties": {
            "original": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "eventtype": {
          "type": "keyword"
        },
        "graphicelement": {
          "type": "long"
        },
        "host": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "id": {
          "type": "long"
        },
        "location": {
          "type": "keyword"
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "mmsstate": {
          "type": "long"
        },
        "operator": {
          "type": "keyword"
        },
        "severity": {
          "type": "long"
        },
        "sourcetime": {
          "type": "date",
          "format": "strict_date_optional_time||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.S"
        },
        "state": {
          "type": "long"
        },
        "subsystem": {
          "type": "keyword"
        },
        "system": {
          "type": "keyword"
        },
        "uniqueid": {
          "type": "keyword"
        },
        "value": {
          "type": "keyword"
        },
        "zone": {
          "type": "long"
        }
      }
    },
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "provided_name": "ats-events-2023-06",
        "creation_date": "1700677951207",
        "number_of_replicas": "1",
        "uuid": "p_IVC2cJTHSUe7DrZgbtJQ",
        "version": {
          "created": "8500003"
        }
      }
    }
  }
}
# GET ats-events-2023-06/_search 200 OK
{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 8,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "ats-events-2023-06",
        "_id": "Xu5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:06.644+08:00",
          "subsystem": "DDD",
          "equipment": "xx210/RRR/DDD/ALL",
          "description": "Summary of Status with Closed & Locked",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:06.644+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[1]",
          "location": "xx210",
          "id": 3432487,
          "state": 5,
          "event": {
            "original": "3432487,CCC_3432487,CCC_0,DIAG_IAllDoorModeStatus[1],RRR,DDD,2023-06-01 00:00:06.644,null,0,NOT CLOSED & LOC,xx210/RRR/DDD/ALL,xx210,0,Summary of Status with Closed & Locked,5,0,-1,-1"
          },
          "value": "NOT CLOSED & LOC",
          "uniqueid": "CCC_3432487"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "Xe5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:06.642+08:00",
          "subsystem": "DDD",
          "equipment": "xx210/RRR/DDD/ALL",
          "description": "Summary of Status with Open",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:06.642+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[2]",
          "location": "xx210",
          "id": 3432486,
          "state": 5,
          "event": {
            "original": "3432486,CCC_3432486,CCC_0,DIAG_IAllDoorModeStatus[2],RRR,DDD,2023-06-01 00:00:06.642,null,0,>=1 OPEN,xx210/RRR/DDD/ALL,xx210,0,Summary of Status with Open,5,0,-1,-1"
          },
          "value": ">=1 OPEN",
          "uniqueid": "CCC_3432486"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "W-5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:06.143+08:00",
          "subsystem": "DDD",
          "equipment": "xx030/RRR/DDD/ALL",
          "description": "Summary of Status with Closed & Locked",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:06.143+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[1]",
          "location": "xx030",
          "id": 3432485,
          "state": 5,
          "event": {
            "original": "3432485,CCC_3432485,CCC_0,DIAG_IAllDoorModeStatus[1],RRR,DDD,2023-06-01 00:00:06.143,null,0,CLOSED & LOCKED,xx030/RRR/DDD/ALL,xx030,0,Summary of Status with Closed & Locked,5,0,-1,-1"
          },
          "value": "CLOSED & LOCKED",
          "uniqueid": "CCC_3432485"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "Wu5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:04.923+08:00",
          "subsystem": "AAA",
          "equipment": "GGG/SSS/AAA/CCCAAA_SVR",
          "description": "Command SET ROUTE Received Status with Closed & Locked",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:04.923+08:00",
          "system": "SSS",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "COMMAND_RECEIVED_CCC",
          "location": "GGG",
          "id": 3432483,
          "state": 5,
          "event": {
            "original": "3432483,CCC_3432483,CCC_0,COMMAND_RECEIVED_CCC,SSS,AAA,2023-06-01 00:00:04.923,null,0,,GGG/SSS/AAA/CCCAAA_SVR,GGG,0,Command SET ROUTE Received Status with Closed & Locked,5,0,-1,-1"
          },
          "uniqueid": "CCC_3432483"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "We5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:04.614+08:00",
          "subsystem": "ATC",
          "equipment": "xx022/SSS/ATC/AAA_SYS",
          "description": "Auto Norm: performed handover with one radio only",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:04.614+08:00",
          "system": "SSS",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IHndovr1Radio",
          "location": "xx022",
          "id": 3432482,
          "state": 5,
          "event": {
            "original": "3432482,CCC_3432482,CCC_0,DIAG_IHndovr1Radio,SSS,ATC,2023-06-01 00:00:04.614,null,0,ALARM,xx022/SSS/ATC/AAA_SYS,xx022,0,Auto Norm: performed handover with one radio only,5,0,-1,-1"
          },
          "value": "ALARM",
          "uniqueid": "CCC_3432482"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "XO5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:04.108+08:00",
          "subsystem": "DDD",
          "equipment": "xx140/RRR/DDD/ALL",
          "description": "Summary ofStatus with Closed & Locked",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:04.108+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[1]",
          "location": "xx140",
          "id": 3432481,
          "state": 5,
          "event": {
            "original": "3432481,CCC_3432481,CCC_0,DIAG_IAllDoorModeStatus[1],RRR,DDD,2023-06-01 00:00:04.108,null,0,CLOSED & LOCKED,xx140/RRR/DDD/ALL,xx140,0,Summary ofStatus with Closed & Locked,5,0,-1,-1"
          },
          "value": "CLOSED & LOCKED",
          "uniqueid": "CCC_3432481"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "WO5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:00.439+08:00",
          "subsystem": "DDD",
          "equipment": "xx030/RRR/DDD/ALL",
          "description": "Summary of  Status with Open",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:00.439+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[2]",
          "location": "xx030",
          "id": 3432480,
          "state": 5,
          "event": {
            "original": "3432480,CCC_3432480,CCC_0,DIAG_IAllDoorModeStatus[2],RRR,DDD,2023-06-01 00:00:00.439,null,0,NO DOORS OPEN,xx030/RRR/DDD/ALL,xx030,0,Summary of  Status with Open,5,0,-1,-1"
          },
          "value": "NO DOORS OPEN",
          "uniqueid": "CCC_3432480"
        }
      },
      {
        "_index": "ats-events-2023-06",
        "_id": "X-5O-IsBNa_axXWiIuXa",
        "_score": 1,
        "_source": {
          "severity": 0,
          "log": {
            "file": {
              "path": "/usr/share/logstash/csv_files/events2023-06-01.csv"
            }
          },
          "graphicelement": -1,
          "sourcetime": "2023-06-01T00:00:00.035+08:00",
          "subsystem": "DDDD",
          "equipment": "xxx800/RRR/DDD/ALL",
          "description": "Summary of",
          "operator": "null",
          "mmsstate": 0,
          "@timestamp": "2023-06-01T00:00:00.035+08:00",
          "system": "RRR",
          "zone": -1,
          "alarmvalue": 0,
          "@version": "1",
          "host": {
            "name": "6e71e0dd4966"
          },
          "alarm": "CCC_0",
          "eventtype": "DIAG_IAllDoorModeStatus[2]",
          "location": "xxx",
          "id": 3432479,
          "state": 5,
          "event": {
            "original": "3432479,CCC_3432479,CCC_0,DIAG_IAllDoorModeStatus[2],RRR,DDDD,2023-06-01 00:00:00.035,null,0,NO DOORS OPEN,xxx800/RRR/DDD/ALL,xxx,0,Summary of,5,0,-1,-1"
          },
          "value": "NO DOORS OPEN",
          "uniqueid": "CCC_3432479"
        }
      }
    ]
  }
}