Hi,
Logstash was installed and was receiving logs from filebeat and syslog file stream. Currently logstash starts up successfully but no logs are being processed.
[2021-04-02T08:48:29,269][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2021-04-02T08:48:33,245][INFO ][org.reflections.Reflections] Reflections took 52 ms to scan 1 urls, producing 23 keys and 47 values
[2021-04-02T08:48:34,791][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2021-04-02T08:48:35,049][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2021-04-02T08:48:35,107][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2021-04-02T08:48:35,115][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-04-02T08:48:35,155][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2021-04-02T08:48:35,267][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-04-02T08:48:35,371][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/01-beats-syslog.conf"], :thread=>"#<Thread:0x1bc9257d run>"}
[2021-04-02T08:48:35,399][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
[2021-04-02T08:48:35,404][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-04-02T08:48:36,611][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.21}
[2021-04-02T08:48:36,828][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-04-02T08:48:36,855][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-04-02T08:48:36,873][INFO ][logstash.inputs.tcp ][main][901cf63d15e66de7011a172dc10fc21abf93fadfbe8111ec347bf41b38abe23e] Starting tcp input listener {:address=>"0.0.0.0:5140", :ssl_enable=>"false"}
[2021-04-02T08:48:36,922][INFO ][org.logstash.beats.Server][main][bd384a02481e4c17d87f2586f58cacb65814ab0c031227b68541fbe9bf7b9ebd] Starting server on port: 5044
[2021-04-02T08:48:37,096][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-04-02T08:48:37,342][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
I have confirmed the input port 5044 for filebeat and 5140 for syslog are open from one of the servers sending the logs as shown below:
[root@filebeat-host ~]# telnet 192.168.6.112 5044
Trying 192.168.6.112...
Connected to 192.168.6.112.
Escape character is '^]'.
Connection closed by foreign host.
[root@filebeat-host ~]# telnet 192.168.6.112 5140
Trying 192.168.6.112...
Connected to 192.168.6.112.
Escape character is '^]'.
The logstash config file is as below:
root@logserver:~# cat /etc/logstash/conf.d/01-beats-syslog.conf
#tcp syslog stream via 5140
input {
tcp {
type => "syslog"
port => 5140
}
}
#udp syslogs stream via 5140
#input {
# udp {
# type => "syslog"
# port => 5140
# }
#}
input {
beats {
port => 5044
ssl => false
# consider adding type => "misc"
}
}
#filter {
#if [type] == "syslog" {
# grok {
# match => { "message" => "%{SYSLOGLINE}" }
# }
#
# date {
#match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
#}
# }
#
#}
output {
elasticsearch {
hosts => localhost
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
There is no error reported when the config file is started with with test_and_exit option
root@logserver:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash --config.test_and_exit -f 01-beats-syslog.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-04-02 09:01:27.229 [main] runner - Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[WARN ] 2021-04-02 09:01:27.949 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-04-02 09:01:30.052 [LogStash::Runner] Reflections - Reflections took 51 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2021-04-02 09:01:30.774 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2021-04-02 09:01:30.981 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
Logstash is meant to send data to elastic search and that appears not to have space issues:
root@logserver:/etc/logstash/conf.d# curl localhost:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 188,
"active_shards" : 188,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 187,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.13333333333333
}
Please assist.