# Fields are not visible in the Discovery panel

**URL:** <https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915>\
**Category:** Kibana\
**Created:** [May 22, 2020, 2:27pm UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915 "2020-05-22T14:27:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\
**Post date:** [May 22, 2020, 2:27pm UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915/1 "2020-05-22T14:27:07Z")

</div>

Hello,

I am encountering a "weird" problem with kibana. I am ingesting some few event from an ESXi Vmware through logstash to Elasticsearch.

At first I can see the fields that I added with logstash on the discovery panel like so :  
 ![MicrosoftTeams-image (3)](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e96c07c6e1138e88aa789246dbc253d0225c9d6.png)

But after a while I don't see them. I figured that it is normal that the discovery won't show you the fields if they were any of them indexed during the specified time range but the weird thing is that when I expand my time range to go back to the previous ones i don't see them :  
Like in this screenshot where the field show above named event\_name don't show up during a bigger time range :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57ca770dca676c17a52d0f9e7afbff4355b35593.png)  
the same for others like vm\_name, event\_code, ... etc

_Context :_  
**Cluster of 3 nodes** : one master-dedicated node and two data nodes HOT/WARM plus one logstash-only node

I am able to query the data with API call and i can see the fields are indexed :

```
GET vmware/_search?q=event_name:Machine Network VLAN Connected

```

Gave me this :

```auto
{
          "took" : 385,
          "timed_out" : false,
          "_shards" : {
            "total" : 1,
            "successful" : 1,
            "skipped" : 0,
            "failed" : 0
          },
          "hits" : {
            "total" : {
              "value" : 14,
              "relation" : "eq"
            },
            "max_score" : 15.027323,
            "hits" : [
              {
                "_index" : "vmware",
                "_type" : "_doc",
                "_id" : "qNI5PHIBaJTCGwydIlFk",
                "_score" : 15.027323,
                "_source" : {
                  "priority" : 182,
                  "timestamp8601" : "2020-05-22T11:50:39.744Z",
                  "severity" : 6,
                  "message" : """cpu24:3174471)Net: 2456: connected windows-elk eth0 to ELK VLAN, portID 0x3000030
        """,
                  "program" : "vmkernel",
                  "facility" : 22,
                  "timestamp" : "2020-05-22T11:50:39.744Z",
                  "severity_label" : "Informational",
                  "data1" : "cpu24:3174471)",
                  "data2" : """portID 0x3000030
        """,
                  "host" : "@hidden",
                  "vm_name" : "windows-elk",
                  "event_name" : "Machine Network VLAN Connected",
                  "@timestamp" : "2020-05-22T11:50:39.744Z",
                  "@version" : "1",
                  "id" : "2456",
                  "facility_label" : "local6",
                  "vlan_name" : "@hidden",
                  "logsource" : "@hidden",
                  "interface_name" : "eth0"
                }
              },

```

I can also visualize my fields :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a45d40cf63321f6648570757d9f654334f82098.png)

The fields are persistent in the index pattern :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f67f63c2c014b2875c519a2b7e5b65dd104a7161.png)

My logstash configuration :

```auto
input {
  syslog {
    port => 514
  }
}

filter {
        if [message] =~ /verbose/ {
                drop { }
        }
        if "Debug" in [severity_label] {
                drop { }
        }

############################# Hostd ####################################

        else if "Hostd" in [program] and [message] =~ /Event 705/ {
                grok {
                        match => { "message" => "%{GREEDYDATA:data1} Event 705 : User %{DATA:username}@%{IPV4:source_ip} logged in as %{GREEDYDATA:loggedAs}" }
                }
                mutate {
                        add_field => { "event_code" => "705" }
                        add_field => { "event_name" => "Successful Authentication" }
                }
        }
        else if "Hostd" in [program] and [message] =~ /powered off/ {
                grok {
                        match => { "message" => "%{GREEDYDATA:data1} Event %{NUMBER:iddd} : %{DATA:vm_name} on %{GREEDYDATA:logmsg}" }
                }
                mutate {
                       # add_field => { "event_code" => "701" }
                        add_field => { "event_name" => "Machine Powered OFF" }
                }
        }
        else if "Hostd" in [program] and [message] =~ /powered on/ {
                grok {
                        match => { "message" => "%{GREEDYDATA:data1} Event %{NUMBER:iddd} : %{DATA:vm_name} on %{GREEDYDATA:logmsg}" }
                }
                mutate {
                        #add_field => { "event_code" => "711" }
                        add_field => { "event_name" => "Machine Powered ON" }
                }
        }

############################### vmkernel ##############################

        else if "vmkernel" in [program] and [message] =~ /Net: / {
                grok {
                        match => { "message" => "%{GREEDYDATA:data1}Net: %{DATA:id}: connected %{DATA:vm_name} %{DATA:interface_name} %{DATA:vlan_name}, %{GREEDYDATA:data2}"}
                }
                mutate {
                        add_field => { "event_name" => "Machine Network VLAN Connected" }
                }
        }

############################## sshd ##################################

        else if "sshd" in [program] and [message] =~ /Connection from/ {
                grok {
                        match => { "message" => "Connection from %{IPV4:source_ip} port %{NUMBER:source_port}"}
                }
                mutate {
                        add_field => { "event_name" => "Connection Attempt" }
                }
        }
        else if "sshd" in [program] and [message] =~ /Accepted keyboard-interactive/ {
                grok {
                        match => { "message" => "Accepted keyboard-interactive/pam for %{USERNAME:username} from %{IPV4:source_ip} port %{NUMBER:source_port} %{GREEDYDATA:protocol}"}
                }
                mutate {
                        add_field => { "event_name" => "SSH Authentication Success" }
                }
        }
        else if "sshd" in [program] and [message] =~ /Failed keyboard-interactive/ {
                grok {
                        match => { "message" => "Failed keyboard-interactive/pam for invalid user %{USERNAME:username} from %{IPV4:source_ip} port %{NUMBER:source_port} %{GREEDYDATA:protocol}"}
                }
                mutate {
                        add_field => { "event_name" => "SSH Authentication Failure" }
                }
        }
        else if "sshd" in [program] and [message] =~ /pam_unix\(sshd\:auth\)/ {
                grok {
                        match => { "message" => "pam_unix\(sshd\:auth\)\: authentication failure\; logname=%{DATA:logname} uid=%{DATA:uid} euid=%{DATA:euid} tty=%{DATA:tty} ruser=%{DATA:username} rhost=%{IPV4:source_ip}"}
                }
                mutate {
                        add_field => { "event_name" => "SSH Authentication Failure" }
                }
        }
        else if "sshd" in [program] and [message] =~ /pam_unix\(sshd\:session\)/ {
                grok {
                        match => { "message" => "pam_unix\(sshd\:session\)\: session opened for user %{DATA:username} %{GREEDYDATA:data1}"}
                }
                mutate {
                        add_field => { "event_name" => "SSH Session Opened" }
                }
        }
        else if "sshd" in [program] and [message] =~ /Invalid user/ {
                grok {
                        match => { "message" => "Invalid user %{DATA:username} from %{IPV4:source_ip} port %{NUMBER:source_port}"}
                }
                mutate {
                        add_field => { "event_name" => "Invalid Username" }
                }
        }

}

output {
        elasticsearch {
        hosts => ["https://hiddenIP:9200"]
        cacert => "/etc/logstash/ca.pem"
        user => "hidden"
        password => "hidden"
        index => "vmware"
        ssl => true
        ssl_certificate_verification => false
}
}

```

Even though some events doesn't match my grok I still should get the previous parsed field in Discovery but I don't.

Help please

---

<div class="post-metadata">

**Author:** ![poff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poff/32/81795_2.png) [@poff](https://discuss.elastic.co/u/poff)\
**Post date:** [May 22, 2020, 9:27pm UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915/2 "2020-05-22T21:27:01Z")

</div>

Hey there!

When was the last time you refreshed your index pattern? Is it possible that the event\_name field isn't mapped?

---

<div class="post-metadata">

**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\
**Post date:** [May 24, 2020, 11:08am UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915/3 "2020-05-24T11:08:03Z")

</div>

Hello @poff,

No every field is mapped in the index pattern. Every field is populated with the corresponding values. My index date reference is @timestamp.  
To me everythings looks good the only problem is that the field i added with logstash are not persistent in Discovery Panel, I only see them when the event occurs withing a recent (like 15min or 1 hour) timeline.

Thank you for your time

---

<div class="post-metadata">

**Author:** ![poff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poff/32/81795_2.png) [@poff](https://discuss.elastic.co/u/poff)\
**Post date:** [June 2, 2020, 7:59pm UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915/4 "2020-06-02T19:59:12Z")

</div>

I'm still not sure exactly why those fields aren't appearing when you expect but if you click on the `Filter by type` option in Discover, there should be an option to disable the "Hide missing fields" option (which is on by default)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2020, 7:59pm UTC](https://discuss.elastic.co/t/fields-are-not-visible-in-the-discovery-panel/233915/5 "2020-06-30T19:59:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.