# We are not receiving the log from kafka to logstash

**URL:** <https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472>\
**Category:** Logstash\
**Created:** [July 16, 2020, 1:52pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472 "2020-07-16T13:52:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![saravananveera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravananveera/32/59765_2.png) [@saravananveera](https://discuss.elastic.co/u/saravananveera)\
**Post date:** [July 16, 2020, 1:52pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472/1 "2020-07-16T13:52:13Z")

</div>

Hi  
I have received the logs from beat to Kafaka, but i can't able to receive logs from kafka to logstash .

Kafka log:

[root@ip kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sscplblogs

{"@timestamp":"2020-07-16T13:47:30.090Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.2","topic":"sscplblogs"},"offset":336188123,"source":"/var/log/nginx/test-live.com/ssl-access.log","tags":["PLAYER LB"],"input":{"type":"log"},"fields":{"log\_topics":"sscplblogs"},"beat":{"name":"PL-LB-14-36","hostname":"[lb-p1l1.com](http://lb-p1l1.com)","version":"6.3.2"},"host":{"name":"PL-LB-14-36"},"message":"[16/Jul/2020:13:47:29 +0000] 219.68.172.199 - [test-live.com](http://test-live.com) : 178.20.18.48:8443 : POST /member/Config.jsp HTTP/1.0 - upstream\_response\_time : 1.040 - msec: 1594907249.291 - request\_time : 1.206 - upstream\_connect\_time : 0.002 - upstream\_status: 200 - user\_agent: Mozilla/5.0 (Linux; Android 9; Redmi Note 7 Pro Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/83.0.4103.83 Mobile Safari/537.36--clientip:106.217.32.109","prospector":{"type":"log"}}

These log i can't able to receive from logstash.

Logstash conf:

input {  
kafka {  
bootstrap\_servers =\>["localhost:9092"]  
topics =\> ["sscplblogs"]  
codec =\> "json"  
add\_field =\> { "ssclogz" =\> "sscplblogz" }  
}  
}

output {

if [ssclogz] == "sscplblogz" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "sscplblogs-%{+YYYY.MM.dd}"  
}  
file {  
path =\> "/var/log/logstash/sscplblogs.log"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [July 16, 2020, 3:14pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472/2 "2020-07-16T15:14:35Z")

</div>

I'd try this first. If you get an output then you are getting data.

```auto
output {
   stdout { }
}

```

If the above works try to change your conditional statement to below.

```auto
if [kafka][topic] = "sscplblogs" {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "sscplblogs-%{+YYYY.MM.dd}"
   }
   file {
      path => "/var/log/logstash/sscplblogs.log"
   }
}

```

---

<div class="post-metadata">

**Author:** ![saravananveera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravananveera/32/59765_2.png) [@saravananveera](https://discuss.elastic.co/u/saravananveera)\
**Post date:** [July 17, 2020, 2:07pm UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472/3 "2020-07-17T14:07:25Z")

</div>

Hi @aaron-nimocks

Thanks for the reply, I have fixed this issues, Now i have face the issues with index creating in kibana.  
Provided the index got created in Elastic search.  
Kibana lists down the index from elastick search. But when creating the index-pattern hitting below error.

"Could not locate that index-pattern (id: false), click here to re-create it"

 ![Selection_529](https://us1.discourse-cdn.com/elastic/original/3X/9/7/979cb4f4323e20d682dcfd06d737567a275ac659.png)

 ![Selection_531](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1fa0bdb9f95d74952eb8ccb5bac663ec97bcd0d7.png)

---

<div class="post-metadata">

**Author:** ![saravananveera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravananveera/32/59765_2.png) [@saravananveera](https://discuss.elastic.co/u/saravananveera)\
**Post date:** [July 18, 2020, 5:08am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472/4 "2020-07-18T05:08:28Z")

</div>

Hi team,

Please fine the below logs

api/saved\_objects/index-pattern 403 (Forbidden)

![Selection_534](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e91545ca6511b1f37babbe849c476e62137afeb.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2020, 5:08am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472/5 "2020-08-15T05:08:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.