Creating fields from grok

[bobbytan]

I have used grok to pull fields from logs as shown below:

filter {
grok {
match => {"message"=> "%{DATE_EU:logdate:date} %{TIME:logtime:date} %{GREEDYDATA}" }
}
}

However the logdate and logtime fields are shown in the stdout { codec => rubydebug } output, but do not appear as fields when creating kibana visualizations.

Please advice thanks

[magnusbaeck (Magnus Bäck)]

Did you refresh the field list in Kibana?

match => {"message"=> "%{DATE_EU:logdate:date} %{TIME:logtime:date} %{GREEDYDATA}" }

"date" is not a valid conversion for a field. As the grok filter documentation states only "int" and "float" are valid.

You'll want to combine the date and time into a single field (either in the grok filter or by merging the fields afterwards) and feed the resulting string to a date filter.

[bobbytan]

Did you refresh the field list in Kibana?---yes

i did not combine date and time as i would like to create visuals based on the date field alone.

ok changed the conversion to integer type however same issue still persist where fields are shown in stdout but not shown when creating kibana visualizations, image is shown below

[magnusbaeck (Magnus Bäck)]

i did not combine date and time as i would like to create visuals based on the date field alone.

I'm not sure that makes any sense.

ok changed the conversion to integer type

Why? The dates and times matched by DATE_EU and TIME are not integers.

however same issue still persist where fields are shown in stdout but not shown when creating kibana visualizations, image is shown below

If you really have refreshed the field list I'm not sure what's up.