Logstash loading template, but not applying it?

[AndyS (Andrew Shepherd)]

I'm having a problem where logstash is loading a template file, but either it's not applying it, or this isn't being communicated to Elastic.
(This is using Security Onion, which has logstash in Docker)

I have a logstash filter that adds a field "network_name", this is visibile in Kibana but as it's not an idex I can't use it for filters.

My tests from within Docker...

Check the template file is binding correctly and being used by Docker

$ ls -lt /logstash-template.json
-rw-r--r-- 1 root root 77692 Jun  1 16:21 /logstash-template.json

Check this file has my config:

bash-4.2$ grep "network_name" /logstash-template.json -A 6
        "network_name":{
          "type":"text",
          "fields":{
            "keyword":{
              "type":"keyword"
            }
          }

Check the file is being loaded

bash-4.2$ grep "logstash-template" /var/log/logstash/logstash.log | tail -n 5
[2020-06-01T15:38:16,981][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}

Check the file has the right contents (positive, "network_name" exists near the end)

[2020-06-01T16:28:32,641][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}
[2020-06-01T16:28:32,646][INFO ][logstash.outputs.elasticsearch] e"=>"keywo  [..removed...]  pe"=>"ip"}, "network_name"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword"}}}, "year"=>{"type"=>"long"}}}}}}

"z" is a defaultfield, as a second test I removed it

bash-4.2$ grep "\"z\"" /logstash-template.json | wc -l
0

However it does not appear this file that is loaded is applied. In Kibana when I:
management -> index patterns -> reindex
....z is still listed, network_name is not

Can anyone add advice please.
Is it possible/how do I test deeper if the updated template file is being applied
Are any changes needed pushing to Elastic to honour this change?

Many thanks
Andy

[Badger]

Use the mapping API to ask elasticsearch what the mapping for the index is.

as it's not an idex I can't use it for filters

It is unclear what you mean by this.

[AndyS (Andrew Shepherd)]

Thanks @Badger ...

In Kibana, one log entry where the tags and field is applied has these tags

 tags	   	syslogng, bro, dns, external_destination, internal_source, home_network

The only output specified for "bro in tags"

output {
  if "bro" in [tags] and "test_data" not in [tags] and "import" not in [tags] {
#    stdout { codec => rubydebug }
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-bro-%{+YYYY.MM.dd}"
      template_name => "logstash"
      template => "/logstash-template.json"
      template_overwrite => true
    }
  }
}

Then looking at the mapping in Elastic

GET /*logstash*/_mapping

...z exists, "network_name" does not

"z" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword"
              }
            }
          }

From looking at docker/logstash/elastic config I don't see how mappings/logstash definitions are provided to elastic. Ignoring this setup, which part of the config in Elastic defines this?

"It is unclear what you mean by this."

Sorry, I mean this

"network_name" exists, but no mapping. I followed the instructions to refresh on index but it never adds "network_name" or removes "z"

[Badger]

I cannot answer the kibana question.

In your template, what is the value of index_patterns? Are you sure the template actually applies to the index you are writing to?

[AndyS (Andrew Shepherd)]

I know what you mean it feels like "logstash-template.json" isn't get imported, but from the logs I can see that it is being read

[2020-06-01T15:38:16,981][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}

This config applies to "logstash-bro-*"

# grep "index_patterns" logstash-template.json
  "index_patterns": ["logstash-ids-*", "logstash-firewall-*", "logstash-syslog-*", "logstash-bro-*", "logstash-import-*", "logstash-beats-*"],

The output for "bro" inputs

index => "logstash-bro-%{+YYYY.MM.dd}"

Here is the full entry from Elastic. It has the tags (bro) and it has the right index, but not "network_name" as a field

{
  "took" : 30,
  "timed_out" : false,
  "_shards" : {
    "total" : 120,
    "successful" : 120,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 2.0,
    "hits" : [
      {
        "_index" : "logstash-bro-2020.06.01",
        "_type" : "doc",
        "_id" : "3aTPcXIBGLNnyZGQ0wPh",
        "_score" : 2.0,
        "_source" : {
          "answers" : [
            "87.194.89.8",
            "87.194.8.8",
            "87.194.9.8"
          ],
          "destination_ips" : "8.8.4.4",
          "source_ip" : "192.168.1.3",
          "protocol" : "udp",
          "destination_ip" : "8.8.4.4",
          "event_type" : "bro_dns",
          "parent_domain_length" : 11,
          "syslog-facility" : "user",
          "host" : "gateway",
          "query_class" : 1,
          "aa" : false,
          "transaction_id" : 5364,
          "syslog-priority" : "notice",
          "query" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org",
          "network_name" : "home_network",
          "rcode" : 0,
          "query_type" : 1,
          "creation_date" : "2002-05-15T13:35:38.000Z",
          "ips" : [
            "192.168.1.3",
            "8.8.4.4"
          ],
          "subdomain_frequency_score" : 2.16,
          "syslog-host" : "seconion-NU691",
          "ra" : true,
          "tags" : [
            "syslogng",
            "bro",
            "dns",
            "external_destination",
            "internal_source",
            "home_network"
          ],
          "ttls" : [
            95.0,
            95.0,
            95.0
          ],
          "rd" : true,
          "port" : 55732,
          "frequency_scores" : [
            "3.9616",
            "3.8966",
            "2.16"
          ],
          "subdomain" : "epdg.epc.mnc010.mcc234.pub",
          "parent_domain_frequency_score" : 3.8966,
          "syslog-tags" : ".source.s_bro_dns",
          "syslog-host_from" : "seconion-nu691",
          "parent_domain" : "3gppnetwork",
          "syslog-sourceip" : "127.0.0.1",
          "query_class_name" : "C_INTERNET",
          "highest_registered_domain" : "3gppnetwork.org",
          "destination_port" : 53,
          "rejected" : false,
          "top_level_domain" : "org",
          "source_ips" : "192.168.1.3",
          "uid" : "CPd9873rVfIU2iH4Bf",
          "highest_registered_domain_frequency_score" : 3.9616,
          "destination_geo" : {
            "ip" : "8.8.4.4",
            "latitude" : 37.751,
            "country_name" : "United States",
            "country_code2" : "US",
            "continent_code" : "NA",
            "country_code3" : "US",
            "location" : {
              "lon" : -97.822,
              "lat" : 37.751
            },
            "longitude" : -97.822
          },
          "source_port" : 61982,
          "syslog-file_name" : "/nsm/bro/logs/current/dns.log",
          "@version" : "1",
          "timestamp" : "2020-06-01T21:35:44.996Z",
          "logstash_time" : 0.022396087646484375,
          "message" : """{"ts":"2020-06-01T21:35:43.991106Z","uid":"CPd9873rVfIU2iH4Bf","id.orig_h":"192.168.1.3","id.orig_p":61982,"id.resp_h":"8.8.4.4","id.resp_p":53,"proto":"udp","trans_id":5364,"rtt":0.008337974548339844,"query":"epdg.epc.mnc010.mcc234.pub.3gppnetwork.org","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["87.194.89.8","87.194.8.8","87.194.9.8"],"TTLs":[95.0,95.0,95.0],"rejected":false}""",
          "tld" : {
            "subdomain" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org"
          },
          "subdomain_length" : 26,
          "rcode_name" : "NOERROR",
          "tc" : "false",
          "site" : "0",
          "query_length" : 42,
          "@timestamp" : "2020-06-01T21:35:43.991Z",
          "rtt" : 0.008337974548339844,
          "query_type_name" : "A",
          "z" : 0
        }
      }
    ]
  }
}

[Badger]

@AndyS

"network_name" : "home_network",

In what sense does that elasticsearch record not have a network_name field?

[AndyS (Andrew Shepherd)]

Yes, that's the frustration lol

(sorry, back to Kibana lol)

In the image the field name exists, but the "!" (and the greyed out +/- signs) means it's not cached, so I can't filter on it.

Do you think this is an Elastic/Kibana question then? (i.e. separate forum)

[Badger]

I would try asking in the kibana forum.

[AndyS (Andrew Shepherd)]

Will do, thanks!