# Logstash dropping a lot of logs

**URL:** <https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429>\
**Category:** Logstash\
**Created:** [September 29, 2021, 5:11am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429 "2021-09-29T05:11:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Parthib\_Dutta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthib_dutta/32/78648_2.png) [@Parthib\_Dutta](https://discuss.elastic.co/u/Parthib_Dutta)\
**Post date:** [September 29, 2021, 5:11am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429/1 "2021-09-29T05:11:15Z")

</div>

I have this current setup where Filebeat is reading Lumen logs and forwarding them to Logstash . And Logstash is parsing one of those field in log lines as JSON . I am only seeing partial outputs in my Kibana dashboard and it seems it's dropping a lot of logs . I am confused as in the console it shows no errors . Here are my configs:

```auto
filebeat.inputs:
- enabled: true
  paths:
    - /home/parthib/recruitcrm/Albatross/storage/logs/lumen-*.log
  type: log
  fields: {log_type: lumen_logs}
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

```

```auto
input {
  tcp {
    port => 10514
    type => syslog
  }
  beats {
    port => "5044"
    type => beats
  }
}

filter {

  if [type] == 'beats' and [fields][log_type] == "lumen_logs" {
    grok {
       match => { "message" => ["\[%{TIMESTAMP_ISO8601:php_timestamp}\] %{DATA:origin_hostname}\.%{DATA:log_level}: (%{URI:request_url} )?%{GREEDYDATA:json_message}"]}
       add_field => ["received_at", "%{@timestamp}"]
       add_field => ["received_from", "%{host}"]
    }
    if [json_message] {
      json {
        source => "json_message"
        target => "parsed_json_message"
        }
    }
  }
  
}

output {
  elasticsearch{
    hosts => ["https://xxxx.io:9243"] 
    user => "elastic"
    password => "xxxx"
    index => "filebeat-logs"
  }
  stdout { 
    codec => "rubydebug"
   }
  file {
    path => "/tmp/logstash-output.txt"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Parthib\_Dutta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthib_dutta/32/78648_2.png) [@Parthib\_Dutta](https://discuss.elastic.co/u/Parthib_Dutta)\
**Post date:** [September 29, 2021, 10:46am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429/2 "2021-09-29T10:46:48Z")

</div>

Update: Found out that the JSON filter is dropping messages containing the same @timestamp. Any solution to this ?

---

<div class="post-metadata">

**Author:** ![Parthib\_Dutta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthib_dutta/32/78648_2.png) [@Parthib\_Dutta](https://discuss.elastic.co/u/Parthib_Dutta)\
**Post date:** [September 29, 2021, 11:12am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429/3 "2021-09-29T11:12:44Z")

</div>

Related: [Setting "target" and "source" to "message" silently drops events · Issue #34 · logstash-plugins/logstash-filter-json · GitHub](https://github.com/logstash-plugins/logstash-filter-json/issues/34)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2021, 11:12am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429/4 "2021-10-27T11:12:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.