Logstash is not processing logs

Hi,
Logstash was installed and was receiving logs from filebeat and syslog file stream. Currently logstash starts up successfully but no logs are being processed.

[2021-04-02T08:48:29,269][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2021-04-02T08:48:33,245][INFO ][org.reflections.Reflections] Reflections took 52 ms to scan 1 urls, producing 23 keys and 47 values
[2021-04-02T08:48:34,791][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2021-04-02T08:48:35,049][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2021-04-02T08:48:35,107][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2021-04-02T08:48:35,115][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-04-02T08:48:35,155][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2021-04-02T08:48:35,267][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-04-02T08:48:35,371][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/01-beats-syslog.conf"], :thread=>"#<Thread:0x1bc9257d run>"}
[2021-04-02T08:48:35,399][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
[2021-04-02T08:48:35,404][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-04-02T08:48:36,611][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.21}
[2021-04-02T08:48:36,828][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-04-02T08:48:36,855][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-04-02T08:48:36,873][INFO ][logstash.inputs.tcp      ][main][901cf63d15e66de7011a172dc10fc21abf93fadfbe8111ec347bf41b38abe23e] Starting tcp input listener {:address=>"0.0.0.0:5140", :ssl_enable=>"false"}
[2021-04-02T08:48:36,922][INFO ][org.logstash.beats.Server][main][bd384a02481e4c17d87f2586f58cacb65814ab0c031227b68541fbe9bf7b9ebd] Starting server on port: 5044
[2021-04-02T08:48:37,096][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-04-02T08:48:37,342][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I have confirmed the input port 5044 for filebeat and 5140 for syslog are open from one of the servers sending the logs as shown below:

    [root@filebeat-host ~]# telnet 192.168.6.112 5044
    Trying 192.168.6.112...
    Connected to 192.168.6.112.
    Escape character is '^]'.
    Connection closed by foreign host.
[root@filebeat-host ~]# telnet 192.168.6.112 5140
Trying 192.168.6.112...
Connected to 192.168.6.112.
Escape character is '^]'.

The logstash config file is as below:

root@logserver:~# cat /etc/logstash/conf.d/01-beats-syslog.conf
#tcp syslog stream via 5140
input {
  tcp {
    type => "syslog"
    port => 5140
  }
}
#udp syslogs stream via 5140
#input {
#  udp {
#    type => "syslog"
#    port => 5140
#  }
#}


input {
 beats {
  port => 5044
  ssl => false
  # consider adding type => "misc"
  }
}


#filter {
#if [type] == "syslog" {
#    grok {
#      match => { "message" => "%{SYSLOGLINE}" }
#    }
#
#    date {
#match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
#}
#  }
#
#}


output {
 elasticsearch {
  hosts => localhost
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
       }
stdout {    codec => rubydebug      }

There is no error reported when the config file is started with with test_and_exit option

root@logserver:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash --config.test_and_exit -f 01-beats-syslog.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-04-02 09:01:27.229 [main] runner - Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[WARN ] 2021-04-02 09:01:27.949 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-04-02 09:01:30.052 [LogStash::Runner] Reflections - Reflections took 51 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2021-04-02 09:01:30.774 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2021-04-02 09:01:30.981 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Logstash is meant to send data to elastic search and that appears not to have space issues:

root@logserver:/etc/logstash/conf.d# curl localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 188,
  "active_shards" : 188,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 187,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.13333333333333
}

Please assist.

Issue resolved. There was no issue with logstash only sending of data from servers with filebeat

  1. Filebeat didn't restart automatically on one server
  2. There was an error starting filebeat on another
  3. Logging verbosity was reduced on one server thus fewer logs were generated

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.