Indexing with deprecated _type field

Elastic Stack Elasticsearch
1

Hello community. I have a question. What happens in elasticsearch 7.10 if a write attempt occurs with deprecated _type field (we use apache metron, that tries to write to elastic), something like this

 "{"index":{"_index":"logstash_index_2021.04.28","_type":"logstash_doc"}}..".

Is it possible to use _index_template in this case? When I try change field types using index template I get this errors:

ip - Elasticsearch exception [type=illegal_argument_exception, reason=mapper [source.ip] cannot be changed from type [ip] to [text]

timestamp - Elasticsearch exception [type=illegal_argument_exception, reason=mapper [timestamp] cannot be changed from type [date] to [long]]
2

The errors seem to be more related to a mapping change attempt than an index template update.

Share what you did

3

Index logstash_index_2021.04.28 doens't exist, I deleted it using elasticvue. Then I uploaded index pattern to elastic using curl, in this pattern I declared:

...
        "timestamp": {
          "type": "date"
        },
        "source": {
          "properties": {
            "ip": {
              "type": "ip"
            }
...

Field types definition I tried separately (one first,another after).And then tried to write to elasticsearch using apache metron. Got those errors from the first message

4

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script is something anyone can copy and paste in Kibana dev console, click on the run button to reproduce your use case. It will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

5 
PUT _index_template/logstash
{
"index_patterns": [
    "logstash_index_*"
],
"template": {
  "mappings": {
      "dynamic_templates": [
          {
              "timestamps_ts": {
                  "mapping": {
                      "format": "epoch_millis",
                      "type": "date"
                  },
                  "match": "*ts",
                  "match_mapping_type": "*"
              }
          },
          {
              "timestamps_milliseconds": {
                  "mapping": {
                      "format": "epoch_millis",
                      "type": "date"
                  },
                  "match": "*_milliseconds",
                  "match_mapping_type": "*"
              }
          },
          {
              "timestamps_seconds": {
                  "mapping": {
                      "format": "epoch_second",
                      "type": "date"
                  },
                  "match": "*_seconds",
                  "match_mapping_type": "*"
              }
        }
      ],
            "properties": {
                
                "source": {
                    "properties": {
                        "ip": {
                            "type": "ip"
                        },
                        "port": {
                            "type": "long"
                        }
                    }
                },
                "timestamp": {
                    "type": "long"
                }
            }
        
    },
    "settings": {
        "index": {
            "refresh_interval": "15s",
                "number_of_shards": "1"
        }
    } 
    }  
}
6

It gives:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[44:37] [template] failed to parse field [mappings]"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[44:37] [index_template] failed to parse field [template]",
    "caused_by" : {
      "type" : "x_content_parse_exception",
      "reason" : "[44:37] [template] failed to parse field [mappings]",
      "caused_by" : {
        "type" : "json_parse_exception",
        "reason" : "Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n at [Source: (org.elasticsearch.common.io.stream.InputStreamStreamInput); line: 45, column: 26]"
      }
    }
  },
  "status" : 400
}
7

Ah, hurried. Fixed request

изображение
изображение1920×781 38.7 KB

It should be ok now

8

So everything is working well, right?

9

I'm talking about indexing, when apache metron starting writing data to elasticsearch. In that moment I got mapping errors.

10

Please provide a script to reproduce the problem. We can't help I think without more information.

11

First I uploaded index pattern for asa_logstash_index_*. Then I tried to write to index using apache metron

изображение
изображение1234×305 18.6 KB

And got this responses:
изображение
изображение1224×247 20.3 KB

Don't know how to make bulk requests with headers using kibana console.

12

You don't need headers in Kibana dev console.

But reproducing with one single document would be enough

13

After index pattern uploading

POST _bulk 
{   "index": { "_index": "logstash_index_2021.04.28", "_type": "logstash_doc" }}
{ "parallelenricher.splitter.end.ts": "1619587111219", "destination": { "port": 123,"ip": "100.100.100.100"}, "source": { "port": 123, "ip": "101.101.101.101"}, "message": "<132>Apr 27 2021 20:21:44: %ASA-4-106023: Deny udp src Ingress_Interface:100.100.100.100/123 dst Egress_Interface:101.101.101.101/123 by access-group \"Rule_Name\" [0x0, 0x0]\n", "parallelenricher.enrich.begin.ts": "1619587111219", "metron.metadata.topic": "asa_after_logstash", "source.type": "asa_logstash", "parallelenricher.splitter.begin.ts": "1619587111219", "observer": { "ingress": { "interface": { "name": "Ingress_Interface" } },"egress": {"interface": {"name": "Egress_Interface"}}},"@timestamp": "2021-04-28T05:18:02.726Z", "original_string": "{\"observer\":{\"ingress\":{\"interface\":{\"name\":\"Ingress_Interface\"}},\"egress\":{\"interface\":{\"name\":\"Egress_Interface\"}}},\"@timestamp\":\"2021-04-28T05:18:02.726Z\",\"organization\":{\"id\":\"0\"},\"destination\":{\"ip\":\"101.101.101.101\",\"port\":123},\"@version\":\"1\",\"source\":{\"ip\":\"100.100.100.100\",\"port\":123},\"message\":\"<132>Apr 27 2021 20:21:44: %ASA-4-106023: Deny udp src Ingress_Interface:100.100.100.100\\/123 dst Egress_Interface:101.101.101.101\\/123 by access-group \\\"Ingress_Interface_access_in_1\\\" [0x0, 0x0]\\n\",\"cisco\":{\"asa\":{\"Egress_Interface\":\"Ingress_Interface_access_in_1\",\"outcome\":\"Deny\",\"network\":{\"transport\":\"udp\"}}}}", "parallelenricher.enrich.end.ts": "1619587111219", "organization": { "id": "0"}, "@version": "1", "guid": "da35ff9a-195f-4e48-9a74-ff07ff9ff087", "cisco": { "asa": { "Egress_Interface": "Rule_Name", "outcome": "Deny", "network": { "transport": "udp"}}},"timestamp": 1619587083600}

изображение
изображение1920×760 85.4 KB

14

Try with this header instead:

{ "index": { "_index": "logstash_index_2021.04.28", "_type": "_doc" }}
15

I can't, header forms apache metron.

16

I see.

Some comments. I can see from Metron – Elasticsearch in Metron that:

Project Metron has retired.

The last version, this project is supporting is 5.6 if I read that correctly.
Here you are trying to use it with a 7.x version. It might work with some luck but I'm almost sure it won't work anymore in the future as types are going away. See Removal of mapping types | Elasticsearch Guide [7.12] | Elastic

Your "template" API call is the new way to create index templates. It does not support the include_type_name option. So you would need to call the legacy template.

This will work:

DELETE logstash_index_test
DELETE _template/logstash
PUT _template/logstash?include_type_name=true
{
  "index_patterns": [
    "logstash_index_*"
  ],
  "mappings": {
    "logstash_doc": {
      "dynamic_templates": [
        {
          "timestamps_ts": {
            "mapping": {
              "format": "epoch_millis",
              "type": "date"
            },
            "match": "*ts",
            "match_mapping_type": "*"
          }
        },
        {
          "timestamps_milliseconds": {
            "mapping": {
              "format": "epoch_millis",
              "type": "date"
            },
            "match": "*_milliseconds",
            "match_mapping_type": "*"
          }
        },
        {
          "timestamps_seconds": {
            "mapping": {
              "format": "epoch_second",
              "type": "date"
            },
            "match": "*_seconds",
            "match_mapping_type": "*"
          }
        }
      ],
      "properties": {
        "source": {
          "properties": {
            "ip": {
              "type": "ip"
            },
            "port": {
              "type": "long"
            }
          }
        },
        "timestamp": {
          "type": "long"
        }
      }
    }
  },
  "settings": {
    "index": {
      "refresh_interval": "15s",
      "number_of_shards": "1"
    }
  }
}
POST _bulk 
{ "index": { "_index": "logstash_index_test", "_type": "logstash_doc" }}
{ "parallelenricher.splitter.end.ts": "1619587111219", "destination": { "port": 123,"ip": "100.100.100.100"}, "source": { "port": 123, "ip": "101.101.101.101"}, "message": "<132>Apr 27 2021 20:21:44: %ASA-4-106023: Deny udp src Ingress_Interface:100.100.100.100/123 dst Egress_Interface:101.101.101.101/123 by access-group \"Rule_Name\" [0x0, 0x0]\n", "parallelenricher.enrich.begin.ts": "1619587111219", "metron.metadata.topic": "asa_after_logstash", "source.type": "asa_logstash", "parallelenricher.splitter.begin.ts": "1619587111219", "observer": { "ingress": { "interface": { "name": "Ingress_Interface" } },"egress": {"interface": {"name": "Egress_Interface"}}},"@timestamp": "2021-04-28T05:18:02.726Z", "original_string": "{\"observer\":{\"ingress\":{\"interface\":{\"name\":\"Ingress_Interface\"}},\"egress\":{\"interface\":{\"name\":\"Egress_Interface\"}}},\"@timestamp\":\"2021-04-28T05:18:02.726Z\",\"organization\":{\"id\":\"0\"},\"destination\":{\"ip\":\"101.101.101.101\",\"port\":123},\"@version\":\"1\",\"source\":{\"ip\":\"100.100.100.100\",\"port\":123},\"message\":\"<132>Apr 27 2021 20:21:44: %ASA-4-106023: Deny udp src Ingress_Interface:100.100.100.100\\/123 dst Egress_Interface:101.101.101.101\\/123 by access-group \\\"Ingress_Interface_access_in_1\\\" [0x0, 0x0]\\n\",\"cisco\":{\"asa\":{\"Egress_Interface\":\"Ingress_Interface_access_in_1\",\"outcome\":\"Deny\",\"network\":{\"transport\":\"udp\"}}}}", "parallelenricher.enrich.end.ts": "1619587111219", "organization": { "id": "0"}, "@version": "1", "guid": "da35ff9a-195f-4e48-9a74-ff07ff9ff087", "cisco": { "asa": { "Egress_Interface": "Rule_Name", "outcome": "Deny", "network": { "transport": "udp"}}},"timestamp": 1619587083600}

But again, this is not going to work in the future.

17

Thank you very much for patience and answers!
P.S. How much time we have until this legacy will be dropped?

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.