Logstash loading template, but not applying it?

I'm having a problem where logstash is loading a template file, but either it's not applying it, or this isn't being communicated to Elastic.
(This is using Security Onion, which has logstash in Docker)

I have a logstash filter that adds a field "network_name", this is visibile in Kibana but as it's not an idex I can't use it for filters.

My tests from within Docker...

Check the template file is binding correctly and being used by Docker

$ ls -lt /logstash-template.json
-rw-r--r-- 1 root root 77692 Jun  1 16:21 /logstash-template.json

Check this file has my config:

bash-4.2$ grep "network_name" /logstash-template.json -A 6
        "network_name":{
          "type":"text",
          "fields":{
            "keyword":{
              "type":"keyword"
            }
          }

Check the file is being loaded

bash-4.2$ grep "logstash-template" /var/log/logstash/logstash.log | tail -n 5
[2020-06-01T15:38:16,981][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}

Check the file has the right contents (positive, "network_name" exists near the end)

[2020-06-01T16:28:32,641][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}
[2020-06-01T16:28:32,646][INFO ][logstash.outputs.elasticsearch] e"=>"keywo  [..removed...]  pe"=>"ip"}, "network_name"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword"}}}, "year"=>{"type"=>"long"}}}}}}

"z" is a defaultfield, as a second test I removed it

bash-4.2$ grep "\"z\"" /logstash-template.json | wc -l
0

However it does not appear this file that is loaded is applied. In Kibana when I:
management -> index patterns -> reindex
....z is still listed, network_name is not

Can anyone add advice please.
Is it possible/how do I test deeper if the updated template file is being applied
Are any changes needed pushing to Elastic to honour this change?

Many thanks
Andy

Use the mapping API to ask elasticsearch what the mapping for the index is.

as it's not an idex I can't use it for filters

It is unclear what you mean by this.

Thanks @Badger ...

In Kibana, one log entry where the tags and field is applied has these tags

 tags	   	syslogng, bro, dns, external_destination, internal_source, home_network

The only output specified for "bro in tags"

output {
  if "bro" in [tags] and "test_data" not in [tags] and "import" not in [tags] {
#    stdout { codec => rubydebug }
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-bro-%{+YYYY.MM.dd}"
      template_name => "logstash"
      template => "/logstash-template.json"
      template_overwrite => true
    }
  }
}

Then looking at the mapping in Elastic

GET /*logstash*/_mapping

...z exists, "network_name" does not

"z" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword"
              }
            }
          }

From looking at docker/logstash/elastic config I don't see how mappings/logstash definitions are provided to elastic. Ignoring this setup, which part of the config in Elastic defines this?

"It is unclear what you mean by this."

Sorry, I mean this


"network_name" exists, but no mapping. I followed the instructions to refresh on index but it never adds "network_name" or removes "z"

I cannot answer the kibana question.

In your template, what is the value of index_patterns? Are you sure the template actually applies to the index you are writing to?

I know what you mean it feels like "logstash-template.json" isn't get imported, but from the logs I can see that it is being read

[2020-06-01T15:38:16,981][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"}

This config applies to "logstash-bro-*"

# grep "index_patterns" logstash-template.json
  "index_patterns": ["logstash-ids-*", "logstash-firewall-*", "logstash-syslog-*", "logstash-bro-*", "logstash-import-*", "logstash-beats-*"],

The output for "bro" inputs

index => "logstash-bro-%{+YYYY.MM.dd}"

Here is the full entry from Elastic. It has the tags (bro) and it has the right index, but not "network_name" as a field

{
  "took" : 30,
  "timed_out" : false,
  "_shards" : {
    "total" : 120,
    "successful" : 120,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 2.0,
    "hits" : [
      {
        "_index" : "logstash-bro-2020.06.01",
        "_type" : "doc",
        "_id" : "3aTPcXIBGLNnyZGQ0wPh",
        "_score" : 2.0,
        "_source" : {
          "answers" : [
            "87.194.89.8",
            "87.194.8.8",
            "87.194.9.8"
          ],
          "destination_ips" : "8.8.4.4",
          "source_ip" : "192.168.1.3",
          "protocol" : "udp",
          "destination_ip" : "8.8.4.4",
          "event_type" : "bro_dns",
          "parent_domain_length" : 11,
          "syslog-facility" : "user",
          "host" : "gateway",
          "query_class" : 1,
          "aa" : false,
          "transaction_id" : 5364,
          "syslog-priority" : "notice",
          "query" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org",
          "network_name" : "home_network",
          "rcode" : 0,
          "query_type" : 1,
          "creation_date" : "2002-05-15T13:35:38.000Z",
          "ips" : [
            "192.168.1.3",
            "8.8.4.4"
          ],
          "subdomain_frequency_score" : 2.16,
          "syslog-host" : "seconion-NU691",
          "ra" : true,
          "tags" : [
            "syslogng",
            "bro",
            "dns",
            "external_destination",
            "internal_source",
            "home_network"
          ],
          "ttls" : [
            95.0,
            95.0,
            95.0
          ],
          "rd" : true,
          "port" : 55732,
          "frequency_scores" : [
            "3.9616",
            "3.8966",
            "2.16"
          ],
          "subdomain" : "epdg.epc.mnc010.mcc234.pub",
          "parent_domain_frequency_score" : 3.8966,
          "syslog-tags" : ".source.s_bro_dns",
          "syslog-host_from" : "seconion-nu691",
          "parent_domain" : "3gppnetwork",
          "syslog-sourceip" : "127.0.0.1",
          "query_class_name" : "C_INTERNET",
          "highest_registered_domain" : "3gppnetwork.org",
          "destination_port" : 53,
          "rejected" : false,
          "top_level_domain" : "org",
          "source_ips" : "192.168.1.3",
          "uid" : "CPd9873rVfIU2iH4Bf",
          "highest_registered_domain_frequency_score" : 3.9616,
          "destination_geo" : {
            "ip" : "8.8.4.4",
            "latitude" : 37.751,
            "country_name" : "United States",
            "country_code2" : "US",
            "continent_code" : "NA",
            "country_code3" : "US",
            "location" : {
              "lon" : -97.822,
              "lat" : 37.751
            },
            "longitude" : -97.822
          },
          "source_port" : 61982,
          "syslog-file_name" : "/nsm/bro/logs/current/dns.log",
          "@version" : "1",
          "timestamp" : "2020-06-01T21:35:44.996Z",
          "logstash_time" : 0.022396087646484375,
          "message" : """{"ts":"2020-06-01T21:35:43.991106Z","uid":"CPd9873rVfIU2iH4Bf","id.orig_h":"192.168.1.3","id.orig_p":61982,"id.resp_h":"8.8.4.4","id.resp_p":53,"proto":"udp","trans_id":5364,"rtt":0.008337974548339844,"query":"epdg.epc.mnc010.mcc234.pub.3gppnetwork.org","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["87.194.89.8","87.194.8.8","87.194.9.8"],"TTLs":[95.0,95.0,95.0],"rejected":false}""",
          "tld" : {
            "subdomain" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org"
          },
          "subdomain_length" : 26,
          "rcode_name" : "NOERROR",
          "tc" : "false",
          "site" : "0",
          "query_length" : 42,
          "@timestamp" : "2020-06-01T21:35:43.991Z",
          "rtt" : 0.008337974548339844,
          "query_type_name" : "A",
          "z" : 0
        }
      }
    ]
  }
}

In what sense does that Elasticsearch record not have a network_name field? :smiley:

Yes, that's the frustration lol :slight_smile:

(sorry, back to Kibana lol)

In the image the field name exists, but the "!" (and the greyed out +/- signs) means it's not cached, so I can't filter on it.

Do you think this is an Elastic/Kibana question then? (i.e. separate forum)

I would try asking in the kibana forum.

1 Like

Will do, thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.