ECS fields not applied from index template

Elastic Stack Logstash
1

I'm trying to reformat my index template to migrate to ECS. I think I'm missing something because the field format for the multi-level keys is not taken into account.

My index template looks like this :

    {
      "template": {
        "settings": {
          "index": {
            "lifecycle": {
              "name": "fwlogs"
            },
            "codec": "best_compression",
            "mapping": {
              "ignore_malformed": "true"
            },
            "refresh_interval": "60s",
            "number_of_replicas": "0"
          }
        },
        "mappings": {
          "dynamic_templates": [],
          "properties": {
            "@timestamp": {
              "type": "date"
            },
            "@version": {
              "type": "keyword",
              "index": false
            },
            "event": {
              "properties": {
                "category": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "kind": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "module": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "host": {
              "type": "ip"
            },
            "message": {
              "type": "text"
            }
          }
        },
        "aliases": {}
      }
    }

But my indices show "unknown field" for all multi-level fields :
image

I'm using this logstash filter to translate my fields :

    filter{
            if [host] in ["...", "..."] {
                    mutate {
                            add_field => { "event.category" => "network" }
                            add_field => { "event.kind" => "event" }
                            add_field => { "event.module" => "juniper" }
                    }
                    grok {
                            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp_tmp} - %{DATA:vpn} - \[%{IPV4:src_ip}\] %{DATA:src_user}\(%{WORD:domain}\)\[%{DATA:role}\] - %{GREEDYDATA:juni_message}" ]
                    }
                    if [juni_message] =~ /Key Exchange number [0-9]+ occurred for user with NCIP/ {
                            grok {
                                    match => [ "juni_message", "Key Exchange number %{NUMBER} occurred for user with NCIP %{IP:client.ip}" ]
                            }
                    } else if [juni_message] =~ /is deleted since user does not qualify reevaluated policies/ {
                            mutate {
                                    add_field => { "event.reason" => "User does not qualify reevaluated policies" }
                            }
                    }
                    mutate {
                            remove_field => [ "juni_message" ]
                    }
            }
    }

I'm probably missing a flag or a small tip either in logstash or in the index template, but I can't figure out where.

Any idea ?

2

Could you try this instead?

add_field => { "[event][category]" => "network" }
3

Did the change, restarted logstash. Same symptoms (but I'm still on the same index template).

Should I try explicit nested fields in the template ? I.e. adding "type: nested" as in :

"event": {
    "type": "nested", 
    "properties": {
        "category": {

Or should I just rotate my indices ?

4

No. Don't use nested here.

Did you remove the index first?

If it still does not work, I'd recommend moving this discussion to #elastic-stack:logstash .

5

No. I can't remove the index as this is live data I can't afford to loose. I usually wait for the next log rotation at midnight to have the new template used.

Indeed, the root cause seem to lie in the way logstash is configured. I have another issue there (ilm not applied), which may have the same root cause.

BTW, how can I move this topic or add logstash as label ?

6

I moved the thread to #elastic-stack:logstash

7

OK. Found the culprit. My index template had fields that where conflicting with ECS structure.
More specifically, I had " host" and a "url" keyword fields in the template.

This conflict was not detected by ES when pushing the new index template.

That's why I had this strange behaviour of "everything looks fine, but it fails anyway". I renamed the offending fields in the template and changed the logstash ingest logic and everything is on track now.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.