Cannot search for _source.fields in Kibana

[Vladpov]

Hello everyone,

I have a problem with searching for fields that I specified in Logstash filter. In particular I cannot search for syslog_timestamp that I created in filter.conf file in Logstash.

I defined the syslog_timestamp as follows:

 %{SYSLOGTIMESTAMP:syslog_timestamp}

I also have defined duration field in the same .conf file which I can search for in Kibana by typing _source.duration somevalue and it works good. Maybe I have wrong syntax for searching for syslog_timestamp data type...

In console I can make a query:

GET /my_index/_search 
{"query": {
"match_phrase": {
  
  "syslog_timestamp": "Jul  4 13:56:15"
}
  }
}

And appropriate result is found!

Has anyone had any similar problem recently?

I'm thinking about change the grok parsing from the beginning and don't use the SYSLOGTIMESTAMP pattern for parsing the date. Do you have any ideas how else to parse date in this form : Jul 4 13:56:15

Every message in the file is in this form:

Jul  4 13:56:17 vMMR mmr-core[29839]:  GtsAwegAOMTbez_1562241377271986.mt npdbProfiling-end: pid[29839] table[npdbcz] operation[SELECT] duration[6.27 ms] error [] sql[SELECT carrier,validity,now()>validity as now_valid FROM npdbcz WHERE `range` IN ('606339842','60633984','6063398','606339','60633','6063') ORDER BY now_valid DESC,validity DESC]

My current solution:

%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:type} %{SPACE} %{DATA:file_id} %{DATA:file_name} %{DATA:syslog_pid} .*table\s*\[%{WORD:table}\] .*operation\s*\[%{WORD:operation}\] .*duration\s*\[%{NUMBER:duration:float} ms\] %{GREEDYDATA:rest}

Any advice would be appreciated!!!

[Badger]

In Kibana, does syslog_timestamp show as a text field, or as a timestamp field?

[Vladpov]

@Badger Thank you for your reply!

How can I find out if it is text field or timestamp field ?

This is how I see it in Kibana:

I'd like to search in search panel by typing for example Jul 4 13:56:15 and see logs only from this particular date.

When I type _source.duration 11 it works...

[Badger]

In the list of fields on the left, you can see a symbol to the left of @timestamp that indicates it is a timestamp, whilst to the left of @version is a t for text.

You will need to refresh the index-pattern in the Kibana management pane and then see which symbol shows up next to syslog_timestamp. I would expect the ? to change to either t or the clock.

[Vladpov]

Thank you for your pieces of advice again!

I refreshed the index-pattern and it works but symbol of syslogt_timestamp hasn't changed (there is sit question mark). But I can searching for the date now.