Syslog input plugin is not working correctly

Elastic Stack Logstash
1

hi,
im trying to recieve syslog messages with logstash,
i have configured the following conf file:

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { 
   hosts => "dude-elastic:9200"
   index => "syslog_test" 
  }
  stdout { codec => rubydebug }
}

the pipeline is starting correctly, but no input seems to arrive.
when i terminate the batch with ctrl+z, suddenly all of the massages are starting to flow, and i can see them in stdout, and in the LS index, before terminating the index is not there..

this is how the logstash log looks like:

[2018-03-20T14:40:50,618][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Program Files/Elastic/Logstash/6.2.2/modules/fb_apache/configuration"}
[2018-03-20T14:40:50,665][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Program Files/Elastic/Logstash/6.2.2/modules/netflow/configuration"}
[2018-03-20T14:40:51,227][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-20T14:40:52,368][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.2"}
[2018-03-20T14:40:53,551][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-20T14:41:06,547][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-20T14:41:07,536][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://dude-elastic:9200/]}}
[2018-03-20T14:41:07,552][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://dude-elastic:9200/, :path=>"/"}
[2018-03-20T14:41:07,958][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://dude-elastic:9200/"}
[2018-03-20T14:41:08,083][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-03-20T14:41:08,099][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-03-20T14:41:08,130][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-03-20T14:41:08,161][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-03-20T14:41:08,255][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//dude-elastic:9200"]}
[2018-03-20T14:41:08,916][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:514", :ssl_enable=>"false"}
[2018-03-20T14:41:09,478][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x665ad90c run>"}
[2018-03-20T14:41:09,619][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2018-03-20T14:41:09,666][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:514", :receive_buffer_bytes=>"65536", :queue_size=>"2000"}
[2018-03-20T14:41:09,791][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-03-20T15:06:08,177][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2018-03-20T15:06:13,219][WARN ][logstash.runner          ] Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss.
[2018-03-20T15:06:13,498][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x665ad90c run>"}

that's really weird,
any ideas?
thanks.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.