Missing Available Fields after Update

[bfrd (bradford)]

Hello,

Recently upgraded elastic stack and filebeat on clients from around 6.5.1 to 6.5.3. We use the system filebeat module. In Kibana > Discover I can still search using terms like "system.auth.user: root" but system.auth.user is no longer showing up as an available field. When I unhide missing fields it still exists and I can add it but the field data comes up as "-" (no data).

If I expand the document, from the table tab, system has a "?" to the left which I think indicates unknown type. The content of system is json.

{
  "auth": {
    "ssh": {
      "event": "Accepted",
      "method": "publickey",
      "signature": "<OMITTED>",
      "geoip": {},
      "port": "38508",
      "ip": "<OMITTED>"
    },
    "hostname": "<OMITTED>",
    "pid": "27374",
    "user": "<OMITTED>",
    "timestamp": "Dec 31 11:12:01"
  }
}

Also under available fields, if I select "? system", it expands and below reads "Analysis is not available for object fields" Under "t tags" it shows the top 5 values per usual.

Forgive me for my crude explanation and ignorance. This may be posted to the wrong section, I only noticed the problem after upgrading the shippers but it may be a logstash, elasticsearch, or kibana issue.

Thanks for your help!

[bfrd (bradford)]

I checked my filters just in case, I am using the one from the 6.5 Reference found here.

I added tagging, they are coming through with success tag.

filter {
  if [fileset][module] == "system" {
    if [fileset][name] == "auth" {
      grok {
        match => { 
          "message" => [
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
            "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"
        ]
      }
      pattern_definitions => {
        "GREEDYMULTILINE" => "(.|\n)*"
      }
      remove_field        => "message"
      add_tag             => [ "_grok_system_auth_success" ]
      tag_on_failure      => [ "_grok_system_auth_nomatch" ]
    }
    date {
      match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    geoip {
      source => "[system][auth][ssh][ip]"
      target => "[system][auth][ssh][geoip]"
    }
  }
  else if [fileset][name] == "syslog" {
    grok {
      match => {
        "message" => [
          "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"
        ]
      }
      pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
      remove_field        => "message"
      add_tag             => [ "_grok_system_syslog_success" ]
      tag_on_failure      => [ "_grok_system_syslog_nomatch" ]
    }
    date {
      match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }
  }
}

Also, in case it would help I am running up-to-date Debian 9 Stretch with elastic-6.x stable repo.

Installed package versions here:

elasticsearch   6.5.2
kibana-oss      6.3.0
logstash        1:6.5.2-1

[bfrd (bradford)]

I've downloaded and installed *-oss-6.5.4 packages

elasticsearch-oss   6.5.4
kibana-oss          6.5.4
logstash            1:6.5.4-1

Was hoping the old Kibana was the problem but the problem persists.

[bfrd (bradford)]

After upgrading everything and waiting a while it seems as though the problem as been solved.

Now fields are populated in Kibana.