Filebeat -> kafka - > logstash no output

Elastic Stack Logstash
1

Hi,

I am trying to read apache-access log file via filebeat and send it to kafka topic.
The kafka topic is used as input to logstash as follows.

input {
        kafka {
                bootstrap_servers => "localhost:9092"
                topics => ["apache-access"]
                group_id => "apache-access"
                type => "apache-access"
                tags => "apache-access"
                decorate_events => true
                consumer_threads => "3"
                auto_offset_reset => "earliest"
        }
}

filter {
}

output {
        if [type] == "apache-access" {
                elasticsearch
                { hosts => "http://localhost:9200"
                  action => "index"
                  index => "apache-access-%{+YYYY.MM.dd}"
                }
                stdout { codec => rubydebug}
        }
}

I see all output message as follows:

{
  "_index": "apache-access-2017.06.22",
  "_type": "apache-access",
  "_id": "AVzRT3B43opijfMgJUKH",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2017-06-22T19:37:43.272Z",
    "kafka": {
      "consumer_group": "apache-access",
      "partition": 0,
      "offset": 32,
      "topic": "apache-access",
      "key": null
    },
    "@version": "1",
    "message": "{\"@timestamp\":\"2017-06-22T19:37:34.817Z\",\"beat\":{\"hostname\":\"localhost\",\"name\":\"localhost\",\"version\":\"5.4.0\"},\"input_type\":\"log\",\"message\":\"10.104.85.253 - - [22/Jun/2017:19:37:34 +0000] \\\"GET /tc/content\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36\\\" [10.104.21.75, 10.104.85.253, 10.104.84.110] [26836] [-]\",\"offset\":5732787,\"path\":\"/wworks/rgs/apache/dev09/logs/\",\"source\":\"/wworks/rgs/apache/dev09/logs/apache_access.log\",\"type\":\"apache_access\"}",
    "type": "apache-access",
    "tags": [
      "apache-access"
    ]
  },
  "fields": {
    "@timestamp": [
      1498160263272
    ]
  },
  "sort": [
    1498160263272
  ]
}

Do you see how message from apache-access log is coming inside filebeat message, which is inside kafka message.
I want to extract message out of it. I tried using codec="json", but the conf file doesnt give any out put to elastic search.
Unfortunately, I dont see any error in logstash-stderr.log. The log.level:debug
Any help is appreciated.
Thanks.

2

I want to extract message out of it. I tried using codec="json", but the conf file doesnt give any out put to Elasticsearch.

Are you getting anything to stdout then? Use stdout { codec => rubydebug} to verify that Logstash is processing the events correctly, then focus on getting the elasticsearch output working.

Unfortunately, I dont see any error in logstash-stderr.log.

That's not where you should be looking. IIRC Logstash logs to logstash-main.log which you should find alongside logstash-stderr.log.

3

Thanks for your response.

I am able to resolve the issue. codec="json" is able to extract the message out as an attribute. I am able to see the output in logstash-stdout.log. The problem was the type parameter. It is set in filebeat, and once it is set in filebeat, it cannot be override if the codec="json". Since I have if condition on type parameter, the output is not going to elasticsearch.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.