# Not getting data in Elastic & Kibna from logstash

**URL:** <https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154>\
**Category:** Elasticsearch\
**Created:** [November 3, 2020, 12:30pm UTC](https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154 "2020-11-03T12:30:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\
**Post date:** [November 3, 2020, 12:30pm UTC](https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154/1 "2020-11-03T12:30:52Z")

</div>

I am trying to upgrade our 5.6 cluster to 7.6 and can certainly use some expertise. I have configured the filebeat with below config

```
filebeat:

```

config:  
inputs:  
path: /etc/filebeat/filebeat.yml  
filebeat:  
inputs:  
- input\_type: log  
document\_type: inf\_os\_logs\_AT  
paths:  
- /var/log/messages  
- /var/log/secure  
exclude\_files: ['.gz$']  
ignore\_older: 48h  
clean\_inactive: 72h  
- input\_type: log  
document\_type: inf\_cmd\_audit  
paths:  
- /var/log/.logging\_history/commands.log  
exclude\_files: ['.gz$']  
ignore\_older: 48h  
clean\_inactive: 72h  
output:  
logstash:  
hosts: ["XXXXXXX:5044","XXXXXXX:5044"]  
loadbalance: true  
compression\_level: 3  
worker: 6  
logging:  
to\_files: true  
files:  
path: /var/log/mybeat  
name: beat.log  
keepfiles: 7  
rotateeverybytes: 104857600 # = 100MB  
level: info

and do see logstatsh getting the payload and processing it

```
[DEBUG] 2020-11-03 07:11:20.810 [defaultEventExecutorGroup-7-2] BeatsHandler - [local: xxxxx:5044, remote: xxxxx:40085] Received a new payload

```

[DEBUG] 2020-11-03 07:11:20.811 [defaultEventExecutorGroup-7-2] BeatsHandler - [local: xxxxx:5044, remote: xxxxx:40085] Sending a new message for the listener, sequence: 1  
[DEBUG] 2020-11-03 07:11:20.814 [defaultEventExecutorGroup-7-2] BeatsHandler - [local: xxxxx:5044, remote: xxxxx:40085] Sending a new message for the listener, sequence: 2  
[DEBUG] 2020-11-03 07:11:20.818 [defaultEventExecutorGroup-7-2] BeatsHandler - 91c281a4: batches pending: false  
[DEBUG] 2020-11-03 07:11:20.920 [[main]\>worker0] grok - Running grok filter {:event=\>#LogStash::Event:0x5e012eb2}  
[DEBUG] 2020-11-03 07:11:20.921 [[main]\>worker0] grok - Event now: {:event=\>#LogStash::Event:0x5e012eb2}  
[DEBUG] 2020-11-03 07:11:20.922 [[main]\>worker0] grok - Running grok filter {:event=\>#LogStash::Event:0x5b2306d4}  
[DEBUG] 2020-11-03 07:11:20.923 [[main]\>worker0] grok - Event now: {:event=\>#LogStash::Event:0x5b2306d4}  
[DEBUG] 2020-11-03 07:11:21.953 [pool-4-thread-1] jvm - collector name {:name=\>"ParNew"}

Logstash filter:

```
filter {

```

if [type] =~ "inf\_os\_logs" or [type] =~ "XXX" or [type] =~ "XXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXXX" or [type] =~ "XXX" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp}" }  
}  
}  
}

I am unable to determine if the index name i expect from filebeat is being passed because we filter logs based on the index name and also i am unable to see any activity in elasticsearch / Kibana which i think may be due to logs getting dropped due to index name.

I have exhausted all possible troubleshooting steps and can use a expert's suggestion.

---

<div class="post-metadata">

**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\
**Post date:** [November 4, 2020, 9:04am UTC](https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154/2 "2020-11-04T09:04:40Z")

</div>

Checking if anyone has any ideas ?

---

<div class="post-metadata">

**Author:** ![Atul\_Chadha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atul_chadha/32/75469_2.png) [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)\
**Post date:** [November 5, 2020, 12:10pm UTC](https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154/3 "2020-11-05T12:10:08Z")

</div>

I figured it out , the index name was sent as filebeat-\* and logstash was not processing it because it was expecting a different index name.

Added below along with customer index name in filebeat and got data in kibana

> setup.ilm.enabled: false  
> setup.template.enabled: true  
> setup.template.name: "inf\_os\_logs\_at-%{[agent.version]}"  
> setup.template.pattern: "inf\_os\_logs\_at-%{[agent.version]}-\*"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 3, 2020, 12:10pm UTC](https://discuss.elastic.co/t/not-getting-data-in-elastic-kibna-from-logstash/254154/4 "2020-12-03T12:10:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.