We are not receiving the log from kafka to logstash

[saravananveera]

Hi
I have received the logs from beat to Kafaka, but i can't able to receive logs from kafka to logstash .

Kafka log:

[root@ip kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sscplblogs

{"@timestamp":"2020-07-16T13:47:30.090Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.2","topic":"sscplblogs"},"offset":336188123,"source":"/var/log/nginx/test-live.com/ssl-access.log","tags":["PLAYER LB"],"input":{"type":"log"},"fields":{"log_topics":"sscplblogs"},"beat":{"name":"PL-LB-14-36","hostname":"lb-p1l1.com","version":"6.3.2"},"host":{"name":"PL-LB-14-36"},"message":"[16/Jul/2020:13:47:29 +0000] 219.68.172.199 - test-live.com : 178.20.18.48:8443 : POST /member/Config.jsp HTTP/1.0 - upstream_response_time : 1.040 - msec: 1594907249.291 - request_time : 1.206 - upstream_connect_time : 0.002 - upstream_status: 200 - user_agent: Mozilla/5.0 (Linux; Android 9; Redmi Note 7 Pro Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/83.0.4103.83 Mobile Safari/537.36--clientip:106.217.32.109","prospector":{"type":"log"}}

These log i can't able to receive from logstash.

Logstash conf:

input {
kafka {
bootstrap_servers =>["localhost:9092"]
topics => ["sscplblogs"]
codec => "json"
add_field => { "ssclogz" => "sscplblogz" }
}
}

output {

if [ssclogz] == "sscplblogz" {
elasticsearch {
hosts => ["localhost:9200"]
index => "sscplblogs-%{+YYYY.MM.dd}"
}
file {
path => "/var/log/logstash/sscplblogs.log"
}
}
}

[aaron_nimocks]

I'd try this first. If you get an output then you are getting data.

output {
   stdout { }
}

If the above works try to change your conditional statement to below.

if [kafka][topic] = "sscplblogs" {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "sscplblogs-%{+YYYY.MM.dd}"
   }
   file {
      path => "/var/log/logstash/sscplblogs.log"
   }
}

[saravananveera]

Hi @aaron_nimocks

Thanks for the reply, I have fixed this issues, Now i have face the issues with index creating in kibana.
Provided the index got created in Elastic search.
Kibana lists down the index from elastick search. But when creating the index-pattern hitting below error.

"Could not locate that index-pattern (id: false), click here to re-create it"

[saravananveera]

Hi team,

Please fine the below logs

api/saved_objects/index-pattern 403 (Forbidden)