# Filtered Log messages show up as empty fields in Kibana

**URL:** <https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399>\
**Category:** Logstash\
**Created:** [September 28, 2022, 7:04pm UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399 "2022-09-28T19:04:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chma](https://avatars.discourse-cdn.com/v4/letter/c/ce73a5/32.png) [@Chma](https://discuss.elastic.co/u/Chma)\
**Post date:** [September 28, 2022, 7:04pm UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399/1 "2022-09-28T19:04:38Z")

</div>

I have filtered my log message using grok. But when I check Kibana, I find the new fields on the left side of the page, but they are empty. I am also getting the` _grokparsefailure` tag.

Here's an example of my log message:

```auto
[2022-09-28 18:11:25,144] {processor.py:641} INFO - Processing file /opt/airflow/dags/dag_filtered.py for tasks to queue

```

Here's my logstash config file:

```auto
input {
  beats {
    port => 5044
    codec => "line"
  }
}
filter{
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}]%{DATA:class} %{SPACE}%{LOGLEVEL:loglevel} -%{GREEDYDATA:logMessage}" }
    overwrite => ["message"]
  }
  date {
    match => ["timestamp", "MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss", "ISO8601"]
    target => "@timestamp"
  }
}
output {
  elasticsearch {
    hosts => ["${IP}:9200"]
    index =>"logss-%{+YYYY.MM.dd}"
  }
}

```

And here's my filebeat configuration:

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /home/ubuntu/logs/**/*.log
filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["${ip}:5044"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_fields:
      fields: ["agent", "cloud", "ecs", "host", "input", "tags", "log.offset"]
      ignore_missing: true

```

When I test my log message and the grok pattern I have on Grok Debugger, it works fine. So what am I missing?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [October 1, 2022, 2:02am UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399/2 "2022-10-01T02:02:28Z")

</div>

I don't think this sample has issues. I have tested your line, it's produced:

```auto
{
         "class" => " {processor.py:641}",
     "timestamp" => "2022-09-28 18:11:25,144",
      "loglevel" => "INFO",
    "logMessage" => " Processing file /opt/airflow/dags/dag_filtered.py for tasks to queue",
    "@timestamp" => 2022-09-28T16:11:25.144Z,
       "message" => "[2022-09-28 18:11:25,144] {processor.py:641} INFO - Processing file /opt/airflow/dags/dag_filtered.py for tasks to queue",
        
}

```

Most likely another line cause a problem. This is a little bit improved grok(without braces and space) with error handling:

```auto
filter{
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}]\s*{%{DATA:class}}\s*%{LOGLEVEL:loglevel}\s*-\s*%{GREEDYDATA:logMessage}" }
  }
  
  date { match => ["timestamp", "ISO8601"] }

}

output {
 if ("_grokparsefailure" in [tags]) {
  elasticsearch {
    hosts => ["${IP}:9200"]
    index =>"grokfailure-%{+YYYY.MM.dd}"
  }
  # optionally
  file { path => "/path/grok_failure_%{+YYYY-MM-dd}.txt" }
 }
 else {
   elasticsearch {
    hosts => ["${IP}:9200"]
    index =>"logss-%{+YYYY.MM.dd}"
  }
 }

}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 29, 2022, 2:03am UTC](https://discuss.elastic.co/t/filtered-log-messages-show-up-as-empty-fields-in-kibana/315399/3 "2022-10-29T02:03:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.