Missing tags and extra fields

Hi guys,

Once again I have an issue trying out the ELK. In general, in Kibana I cannot see any tags I set and also my extra fields (e.g. number/date) displayed which I parsed. The message itself is seen in Kibana, line by line as a message but "tags" stay empty and also the parsed fields are not displayed.
And the index itself is just displayed as "[@metadata][index]".

Can you please advise? I am running version 6.2.4 if it is needed to know.

The log file looks like this:
22.01.2020 ! 08.39.55 ! 78 ! This is a text ! 0 #
22.01.2020 ! 08.39.55 ! 78 ! This as well ! 0 #
22.01.2020 ! 08.39.55 ! 78 ! And finally this ! 0 #

My part for filebeat.yml looks like this:
filebeat_type: abc-def
filebeat_base_path: "/path/to/our/log/"

filebeat_suffix_path: "with/suffix"
ignore_older: 24h
close_inactivhe: 5m
close_removed: true
close_eof: false
clean_removed: true
clean_inactive: 36h

And also, my logstash looks like this:
filter {

        if [@metadata][logtype] == "abc-def"{
        # see: https://www.elastic.co/blog/logstash-metadata or https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html
        mutate {
            add_field => {
        "[@metadata][index]" => "my-test}"

                grok {
                        match => {
                                "message" => "%{DATE_EU:date} ! %{DATA:time} ! %{INT:number} ! %{DATA:category} ! %{INT:occurence} #\n"}

      tags => ["my_tag"]

                mutate {
                        add_field => {"date" => "%{date}"}
                        add_field => {"time" => "%{time}"}
                        add_field => {"plant" => "%{number}"}
                        add_field => {"category" => "%{category}"}
                        add_field => {"occurence" => "%{occurence}"}

I do not see where you are setting [@metadata][logtype]. Are you sure it is equal to that value? Also, remove the \n from the grok pattern. The event will not include a newline.

Hi Badger,

I thought through this here in Filebeat:
filebeat_type: abc-def

I set that parameter? Should I just leave it out or how do I set it?

Hi Badger,
I found my mistake, a nice little bracket was missing and therefore none of my parsing or setting the tag etc. was recognized. Thank you for your help :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.