Fields are not shown in KIbana

Elastic Stack Kibana
1

Hello dear friends of the community. I wanted to know if you could give me some clues as to why I can't view the data loaded in Kibana in an index that I created. I created an index with the mapping to load all the data from a logstash, the log comes from a weblogic server. If I check the data locally in the logstash, it is generating the fields correctly and based on those fields I create the index mapping. The Index Management shows me the index and shows me that it is receiving documents. I was even able to add the index to the Index Pattern, but it doesn't show me the data when I want to see it in Discovery. Looking at another topic I made an "update" of the index:

GET /weblogic-log/_search
{
      "query": {
       "match_all": {}
   }
}

And it shows me several fields:

{
  "took" : 229,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2247,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "zQYSL5EBNkEbxp-bbw_W",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-000000a9",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:00:22,918 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "Diagnostics",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071622918",
          "message" : "####<Aug 7, 2024 7:00:22,918 PM CLT> <Info> <Diagnostics> <f8cloud5032> <producerTotCorpQa01> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-000000a9> <1723071622918> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320145> <Size based data retirement operation completed on archive EventsDataArchive. Retired 0 records in 0 ms.> ",
          "log_message" : "Size based data retirement operation completed on archive EventsDataArchive. Retired 0 records in 0 ms.",
          "log_number" : "BEA-320145",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "8QYSL5EBNkEbxp-bbw_b",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-000000a9",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:00:22,918 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "Diagnostics",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071622918",
          "message" : "####<Aug 7, 2024 7:00:22,918 PM CLT> <Info> <Diagnostics> <f8cloud5032> <producerTotCorpQa01> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-000000a9> <1723071622918> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320144> <Size based data retirement operation started on archive EventsDataArchive.> ",
          "log_message" : "Size based data retirement operation started on archive EventsDataArchive.",
          "log_number" : "BEA-320144",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "RwYSL5EBNkEbxp-byR7V",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "weblogic.GCMonitor",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-00000006",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:00:45,281 PM CLT",
          "kernel" : "anonymous",
          "thread" : "Health",
          "misc" : "[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071645281",
          "message" : "####<Aug 7, 2024 7:00:45,281 PM CLT> <Info> <Health> <f8cloud5032> <producerTotCorpQa01> <weblogic.GCMonitor> <<anonymous>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-00000006> <1723071645281> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <68% of the total memory in the server is free.> ",
          "log_message" : "68% of the total memory in the server is free.",
          "log_number" : "BEA-310002",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "OQYSL5EBNkEbxp-b-CTJ",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:00:57,149 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071657149",
          "message" : "####<Aug 7, 2024 7:00:57,149 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723071657149> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "xAYUL5EBNkEbxp-bymc9",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:02:57,156 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071777156",
          "message" : "####<Aug 7, 2024 7:02:57,156 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723071777156> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "UAYWL5EBNkEbxp-bn6qY",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:04:57,160 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723071897160",
          "message" : "####<Aug 7, 2024 7:04:57,160 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723071897160> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "sgYYL5EBNkEbxp-bdOj6",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:06:57,169 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723072017169",
          "message" : "####<Aug 7, 2024 7:06:57,169 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723072017169> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "3AcaL5EBNkEbxp-bSi5X",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:08:57,174 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723072137174",
          "message" : "####<Aug 7, 2024 7:08:57,174 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723072137174> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "agccL5EBNkEbxp-bH4S-",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "Timer-2",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:10:57,184 PM CLT",
          "kernel" : "WLS Kernel",
          "thread" : "WorkManager",
          "misc" : "[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723072257184",
          "message" : "####<Aug 7, 2024 7:10:57,184 PM CLT> <Info> <WorkManager> <f8cloud5032> <producerTotCorpQa01> <Timer-2> <<WLS Kernel>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-0000000b> <1723072257184> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads> ",
          "log_message" : "Self-tuning thread pool contains 1 running threads, 1 idle threads, and 31 standby threads",
          "log_number" : "BEA-002959",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      },
      {
        "_index" : "weblogic-log",
        "_type" : "_doc",
        "_id" : "ngcdL5EBNkEbxp-bxr1I",
        "_score" : 1.0,
        "_source" : {
          "hostname" : "f8cloud5032",
          "timer" : "weblogic.GCMonitor",
          "uuid" : "7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-00000006",
          "path" : "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log",
          "log_timestamp" : "Aug 7, 2024 7:12:45,283 PM CLT",
          "kernel" : "anonymous",
          "thread" : "Health",
          "misc" : "[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] ",
          "timestamp" : "1723072365283",
          "message" : "####<Aug 7, 2024 7:12:45,283 PM CLT> <Info> <Health> <f8cloud5032> <producerTotCorpQa01> <weblogic.GCMonitor> <<anonymous>> <> <7bcb9779-4af8-4254-bfc4-2bc6ac2dfbaa-00000006> <1723072365283> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <95% of the total memory in the server is free.> ",
          "log_message" : "95% of the total memory in the server is free.",
          "log_number" : "BEA-310002",
          "servername" : "producerTotCorpQa01",
          "log_level" : "Info"
        }
      }
    ]
  }
}

After that I did an index update in the Index Management and in the Index Patterns, but it didn't work, it doesn't show the data..

image
image2557×1375 173 KB

Can anyone have an idea that of what else I can do?. Please, I would really appreciate if you could help me with this.

2

What is the time field that you are using in the weblogic-logs-* data view?

Go into Stack Management > Kibana > Data Views, select the weblogic-logs-* data view, take a screenshot and share it.

3

it may be due to timestamp field you are using .

Try changing the date year from above date filter to 1970. if data appears then follow the steps given below

try convert the log's timestamp and override the @timestamp field value with that and use it in the @timestamp in data view by default it will pick that up if any other is selected then reselect the @timestamp.

date {
    match => ["log_timestamp", "MMM d, yyyy h:mm:ss,SSS a Z"]
    timezone => "Chile/Continental"
    target => "[@timestamp]"
  }
4

Dear Leandro, I´m new in the elastic, i don´t find de "Stack Management > Kibana > Data Views" option, sorry about that, I have Kibana 7.5.1 version, may be is tha reason, I don´t know. However, I show you the next screenshot, may be is use for you.

image
image2527×1272 212 KB

5

Thanks Dear Kishor for your answer.
Question, when you say to convert the log_timestamp field and override the @timestamp field value with that value. Do I have to do that in the Kibana mapping, or add that portion of code in the logstash.conf that I use to send the data?, I meant, some thing like that:

input {
  file {
    path => "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {

   grok {
   match=>{"message"=>["<%{DATA:log_timestamp}> <%{WORD:log_level}> <%{WORD:thread}> <%{HOSTNAME:hostname}> <%{HOSTNAME:servername}> <%{DATA:timer}> <<%{DATA:kernel}>> <> <%{DATA:uuid}> <%{NUMBER:timestamp}> <%{DATA:misc}> <%{DATA:log_number}> <%{DATA:log_message}>"]
   remove_field => ["message"]
   }
date {
    match => ["log_timestamp", "MMM d, yyyy h:mm:ss,SSS a Z"]
    timezone => "Chile/Continental"
    target => "[@timestamp]"
  }
}

  mutate {
        remove_field => ["offset", "prospector","@version","source","host","[beat][hostname]","[beat][name]","[beat][version]","@timestamp","input","beat","log"]
      }

}

output {
      elasticsearch {
        hosts => ["phineas.falabella.cl:9200"]
        index => "weblogic-log"
        user => "user"
        password => "passwd"
  }
  stdout { codec => rubydebug }
}
6

Yes, 7.5.1 is pretty old, you would not have Data Views, these were called Index Patterns in this version.

As you can see in this screen it says that the Time Filter field name for this Index pattner is the @timestamp field.

But in your Logstash you are removing the @timestamp field, so your documents in Elasticsearch will not show up on Kibana because they do not have this field.

Change your Logstash pipeline and remove the @timestamp field from the remove_field and your documents should appear.

Also, any reason to use such an old and unsupported version?

7

It's installed at my work, I can't do anything about it :pensive: . I can only work on this version.

8

Yeah, not a big problem, but keep in mind that this version is pretty old and there is no more support for it.

9

As you are using the logstash so its better to handle there.

i can see you were removing the field @timestamp in remove_field. so it will not give @timestamp field in output as you want to keep that in output and use the default field of @timestamp .

But one thing to keep in mind that you must identify your timestamp field which is your's event/logs timestamp and you that in data fitler to match accordingly and add that to the @timestamp

 date {
          match => ["log_timestamp", "MMM d, yyyy h:mm:ss,SSS a Z"]
          timezone => "Chile/Continental"
          target => "@timestamp"
   }

Also update the

mutate {
        remove_field => ["offset", "prospector","@version","source","host","[beat][hostname]","[beat][name]","[beat][version]","input","beat","log"]
      }
10

Good morning. I want to thank you guys for your great and valuable support, I really appreciate it.
Finally, you were absolutely right, the @timestamp field was not being displayed, after removing part of the code (which by the way had nothing to do with the rest), the data finally appeared in Kibana.

input {
  file {
    path => "/u01/home/app/mdw/producer/domains/producerTotCorpQa/servers/producerTotCorpQa01/logs/producerTotCorpQa01.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {

   grok {
   match=>{"message"=>["<%{DATA:log_timestamp}> <%{WORD:log_level}> <%{WORD:thread}> <%{HOSTNAME:hostname}> <%{HOSTNAME:servername}> <%{DATA:timer}> <<%{DATA:kernel}>> <> <%{DATA:uuid}> <%{NUMBER:timestamp}> <%{DATA:misc}> <%{DATA:log_number}> <%{DATA:log_message}>"]
   }
}

#  mutate {
#        remove_field => ["offset", "prospector","@version","source","host","[beat][hostname]","[beat][name]","[beat][version]","@timestamp","input","beat","log"]
#      }

}

output {
      elasticsearch {
        hosts => ["phineas.falabella.cl:9200"]
        index => "weblogic-log"
        user => "user"
        password => "passwd"
  }
  stdout { codec => rubydebug }
}

image
image2498×1341 497 KB

The other thing I wanted to tell you is that I also realized that the server time, where the logstash is, is wrong, and it is something else that I have to correct.

Server time with Logstash:

$ date
Mon Aug 12 15:58:17 UTC 2024

Server time with Elastic:

$ date
Mon Aug 12 11:59:58 HSP 2024
$

But the main thing is already corrected and it is thanks to you. Have a great day and thank you again, you are really a great team!!