# Not able to get see logs until I delete the index

**URL:** <https://discuss.elastic.co/t/not-able-to-get-see-logs-until-i-delete-the-index/290728>\
**Category:** Logstash\
**Created:** [December 2, 2021, 4:26am UTC](https://discuss.elastic.co/t/not-able-to-get-see-logs-until-i-delete-the-index/290728 "2021-12-02T04:26:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![manavtidhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manavtidhan/32/97951_2.png) [@manavtidhan](https://discuss.elastic.co/u/manavtidhan)\
**Post date:** [December 2, 2021, 4:26am UTC](https://discuss.elastic.co/t/not-able-to-get-see-logs-until-i-delete-the-index/290728/1 "2021-12-02T04:26:49Z")

</div>

Hi ,

I am not able to see logs next day , until I delete the index manually from Kibana  
Below is the status of my logstash and I am not able to see any error logs even I put the debug mode in logstash.yml

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8dc249485d87041fe366a87fcefb68331eb87541.png)

Below is the pipelines.yaml for problematic index.

```auto
- pipeline.id: a-b-c-log-processing
  pipeline.batch.size: 5000
  config.string: |
    input { pipeline { address => a_b_c } }
    filter {
      dissect { mapping => { "message" => "[%{LOGLEVEL}] %{request}" } }
      mutate { remove_field => ["message"] }
      dissect { mapping => {
          "request" => "%{LOGDATETIME};;%{APPLICATION};;%{APINAME};;%{STATUS};;%{TRANSACTIONID};;%{REGION};;%{SERVICENAME};;%{SERVERNAME};;%{ERRORCODE};;%{LOGMSG};;%{LOGDETAIL};;%{ORDERSEQNO};;%{PAYLOAD};;%{OWNERID};;%{REQUESTID};;%{REFERENCE};;%{FUNCTION};;%{IPADDRESS};;%{PORT};;%{ERRORDETAIL};;%{INTERNALID};;%{EXTERNALID};;%{ERRORDUMP};;%{CODE};;%{SUBCODE};;"
          
        }
      }
      mutate { remove_field => ["request","ORDERSEQNO","agent","log","host","input","ecs","tags"] }
      date { match => ['LOGDATETIME', 'YYYY-MM-dd HH:mm:ss.SSS'] }
    }
    output {
      elasticsearch {
        hosts => ["http://xxxx:9200","http://xxxx:9200","http://xxxx:9200"]
        index => "a-b-c-%{+YYYY.MM.dd}"
        user => "xxxxx"
        password => "xxxxxxxxxxxxxx"

      }
    }

```

The background is that we have 3 application which genearte the logs in similar format , we used filebeat to send those logs to the logstash , where we apply the filter . In a,b and c applicition , one applicton "c" which is producer of 90% of logs. I have no problem to see logs of a, b appliciton but C appliciton logs I cannot see in kibana until I delete the index. but if I remove the filter in pipelines.yaml then i have no problem . I am not even getting any error as well in logstash and elastisearch . I am not sure how to trobleshoot this further. Please help it out , I have been stuck this issue with last 3 weeks and still I donot have any clue , I also tried to increase the batch size from 50 to 500 to 5000 but it the problem still persists

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 7, 2021, 3:08pm UTC](https://discuss.elastic.co/t/not-able-to-get-see-logs-until-i-delete-the-index/290728/2 "2021-12-07T15:08:43Z")

</div>

Can you create another output just for the c producer and manually send it to stdout/logfile/another index?  
Just to help you troubleshoot what's going on.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2022, 3:09pm UTC](https://discuss.elastic.co/t/not-able-to-get-see-logs-until-i-delete-the-index/290728/3 "2022-01-04T15:09:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.