.keyword empty


(Jacob M) #1

I'm having an issue with something related to this but, they get off that topic. I'm using the logstash default template and have the .keyword field but, it's empty and I'd like to visualize on the data. The documentation tells me that via dynamic field mapping a text field with a keyword subfield will be created. That keyword field does not seem to be automatically populated with data like I thought it might

Is this by design? Need I create a template with a map to get the main fields aggregatable instead because, my data doesn't have a keyword in the source? Logstash seems to be parsing everything correctly based on the pattern/conf

Index snip

{
  "logstash-2017.09.05" : {
    "aliases" : { },
    "mappings" : {
      "system_logs" : {
        "_all" : {
          "enabled" : true,
          "norms" : false
        },
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "match_mapping_type" : "string",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              }
            }
          },
          {
            "string_fields" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                },
                "norms" : false,
                "type" : "text"
              }
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "include_in_all" : false
          },
          "@version" : {
            "type" : "keyword",
            "include_in_all" : false
          },
          "action_id" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "client_ip" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },

Logstash Config

input {
        tcp {
        port => 5000
        type => "syslog"
      }
}

filter {
  if [type] == "syslog" {
    grok {
        patterns_dir => ["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.1/patterns"]
      match => { "message" => "%{BSF_SCAN}" }
      match => { "message" => "%{BSF_SEND}" }
      match => { "message" => "%{BSF_SEND_NO_DESTINATION}" }
      match => { "message" => "%{BSF_RECV_SCAN}" }
      match => { "message" => "%{BSF_RECV_SCAN_2}" }
      match => { "message" => "%{BSF_RECV}" }
      match => { "message" => "%{BSF_WEB_SYSLOG}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    document_type => "system_logs"
  }
}

(Mark Walkom) #2

What's the data and some of your custom grok patterns look like?


(Jacob M) #3

This is the pattern file

Data - I changed some data to hide information like IP and domain names. I really wasn't sure the best way to send this to you so I looked around the forum and this looked like a decent way

{
  "took" : 93,
  "timed_out" : false,
  "_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
  },
  "hits" : {
"total" : 175302,
"max_score" : 1.0,
"hits" : [
  {
    "_index" : "logstash-2017.09.05",
    "_type" : "system_logs",
    "_id" : "AV5PVw2T5aGozuriYs6Y",
    "_score" : 1.0,
    "_source" : {
      "process" : "scan",
      "subject" : "=?utf-8?B?VGhlIDJuZCBSZWFzb24gV2Ug4p2k77iPIMKgTGFib3IgRGF5?=",
      "message_id" : "1504569654-071d1e44c6e8dc70001-bSw4EI",
      "message" : "<23> scan: vmta-k-120.lstrk.net[142.0.83.120] 1504569654-071d1e44c6e8dc70001-bSw4EI 1504569654 1504569656 SCAN - KBOOBO3JB3O4R48R5D56B6FBN0I52FI3E63R0B0730F@bounce.enews.bostonproper.com zieglerd@domain.org 1.263 0 0 - SZ:28495 SUBJ:=?utf-8?B?VGhlIDJuZCBSZWFzb24gV2Ug4p2k77iPIMKgTGFib3IgRGF5?=",
      "type" : "syslog",
      "reason_extra" : "-",
      "message_size" : "28495",
      "reason_id" : "0",
      "unix_start_time" : "1504569654",
      "score" : "1.263",
      "@timestamp" : "2017-09-05T00:00:58.765Z",
      "encryption" : "-",
      "port" : 57015,
      "sender" : "KBOOBO3JB3O4R48R5D56B6FBN0I52FI3E63R0B0730F@bounce.enews.bostonproper.com",
      "action_id" : "0",
      "service" : "SCAN",
      "@version" : "1",
      "host" : "167.217.4.148",
      "unix_end_time" : "1504569656",
      "recipient" : "zieglerd@domain.org",
      "client_ip" : "142.0.83.120",
      "client_name" : "vmta-k-120.lstrk.net"
    }
  },


  {
    "_index" : "logstash-2017.09.05",
    "_type" : "system_logs",
    "_id" : "AV5PVxUO5aGozuriYs_s",
    "_score" : 1.0,
    "_source" : {
      "process" : "inbound/pass1",
      "message_id" : "1504569658-071d1e44c5e8dca0001-OhuqVO",
      "message" : "<23> inbound/pass1: unknown[internal ip] 1504569658-071d1e44c5e8dca0001-OhuqVO 1504569658 1504569658 RECV client@domain.org client@domain.com 2 7 -",
      "type" : "syslog",
      "reason_extra" : "-",
      "reason_id" : "7",
      "unix_start_time" : "1504569658",
      "@timestamp" : "2017-09-05T00:01:00.678Z",
      "port" : 57015,
      "sender" : "Client@domain.org",
      "action_id" : "2",
      "service" : "RECV",
      "@version" : "1",
      "host" : mail server IP",
      "unix_end_time" : "1504569658",
      "recipient" : "client@domain.com",
      "client_ip" : "inside ip address",
      "client_name" : "unknown"
    }

{
    "_index" : "logstash-2017.09.05",
    "_type" : "system_logs",
    "_id" : "AV5PVxYU5aGozuriYtAW",
    "_score" : 1.0,
    "_source" : {
      "process" : "outbound/smtp",
      "delivery_detail" : "250 2.6.0 <0.0.126.22D.1D325DA0B02259E.6908@vmta-k-120.lstrk.net> [InternalId=142365280960547, Hostname=BY2PR08MB475.namprd08.prod.outlook.com] 36559 bytes in 0.435, 81.957 KB/sec Queued mail for delivery",
      "destination" : "myips-org.mail.protection.outlook.com[216.32.181.106]:25",
      "message_id" : "1504569654-071d1e44c6e8dc70001-bSw4EI",
      "message" : "<23> outbound/smtp: 127.0.0.1 1504569654-071d1e44c6e8dc70001-bSw4EI 0 0 SEND ENC 1 6E7F617968AB 250 2.6.0 <0.0.126.22D.1D325DA0B02259E.6908@vmta-k-120.lstrk.net> [InternalId=142365280960547, Hostname=BY2PR08MB475.namprd08.prod.outlook.com] 36559 bytes in 0.435, 81.957 KB/sec Queued mail for delivery #to#myips-org.mail.protection.outlook.com[216.32.181.106]:25",
      "type" : "syslog",
      "unix_start_time" : "0",
      "@timestamp" : "2017-09-05T00:01:00.941Z",
      "encryption" : "ENC",
      "port" : 57015,
      "action_id" : "1",
      "service" : "SEND",
      "@version" : "1",
      "host" : "mail server ip",
      "unix_end_time" : "0",
      "client_ip" : "127.0.0.1",
      "queue_id" : "6E7F617968AB"
    }

Let me know if there's anything else I can grab

Thank you!


(Jacob M) #4

Mark

Wondering if you have any insight for me. If it doesn't work as is plan is to create a template but, I'd like to avoid that if possible because this will be getting data from a lot of different places and if it works using the default that would be great.

Let me know please

Thank you!


(Jacob M) #5

Ran a search on the "empty" fields and everything was there

Found this by looking at the available fields and unchecking the "hide missing fields". Then all the .keyword fields were listed and I could click visualize like expected

Hope it helps some other newbie out there


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.