Issue with conditionals in logstash with fields from Kafka ----> FileBeat prospectors

Hello experts

I have the following scenario:

FileBeat ----> Kafka -----> Logstash -----> Elastic ----> Kibana

In Filebeat I have 2 prospectors the in YML file,,. and I've added some fields to identify in logstash the log data. But, the issue is: in Logstash I haven't be able to validate this fields.

The configuration files are:

1. filebeat.yml

- input_type: log
    - /opt/jboss/server.log*
  tags: ["log_server"]
    environment: integracion
    log_type: log_server

  document_type: log_server
  fields_under_root: true

- input_type: log
    - /var/todo1_apps/ebanTX.log*
  tags: ["log_eban"]
    environment: integracion
    log_type: log_ebanking

  document_type: log_ebanking
  fields_under_root: true

    enabled: true
    hosts: [""]
    topic: "sve_logs"
    timeout: 30s

2. logstash.conf

input {
  kafka {
    bootstrap_servers => ""
    group_id => "sve_banistmo"
    topics => ["sve_logs"]
    decorate_events => true
    codec => "plain"

filter {
if [type] == "log_ebanking" {
    grok {
       patterns_dir => ["patterns/patterns"]
        match => { "message" => "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:thread}]%{SPACE}-%{SPACE}%{GREEDYDATA:message_log}" }

output {
if [type] == "log_ebanking" {
      elasticsearch {
        hosts => [""]
        index => "sve-banistmo-ebanking-%{+YYYY.MM.dd}"
        stdout { codec => json}

The problem is in the conditional filter and output section. I've tried with


With both the type and log_type variable nothing works !! If I don't put conditionals, the data flow without problem. I mean, is not a conection issue, this is an example of message in Logstash without conditionals:

{"@timestamp":"2016-11-25T18:57:48.569Z","beat":{"hostname":‌​"inrhas14","name":"i‌​nrhas14","version":"‌​5.0.0"},"environment‌​":"integracion","inp‌​ut_type":"log",**"log_‌​type":"log_ebanking"‌**​,"message":"2016-11-‌​25 13:36:53,508 ERROR [http-] - USER_ID: NONE","offset":216302,"source":"/var/todo1_apps/SVE-BSMO/log‌​s/ebankingTX.log","t‌​ags":["log_ebanking"‌​],**"type":"log_ebanki‌​ng"**

Please help me. I've reviewed all the information related, but in my case the conditional doesn't work.

Thanks in advance

Dario R

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.