I am trying to fetch data using beats input from a log file whose path I have given in filebeat.yml. But the log file is not read by the logstash while running.
filebeat.inputs:
- type: log
enabled: true
paths:
- /etc/logstash/filedata.log
output.elasticsearch:
hosts: ["localhost:9200"]
This is my log filedata.log for input
this is file beat data
This is my logstash configuration file
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}}
The output I am getting after running the logstash configuration file is this
/usr/share/logstash/bin/logstash -f /etc/logstash/databeats.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-03 15:36:07.706 [main] runner - Starting Logstash {"logstash.version"=>"7.13.0", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-03 15:36:07.995 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-06-03 15:36:09.593 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-06-03 15:36:10.155 [Converge PipelineAction::Create<main>] Reflections - Reflections took 33 ms to scan 1 urls, producing 24 keys and 48 values
[WARN ] 2021-06-03 15:36:10.823 [Converge PipelineAction::Create<main>] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-03 15:36:10.910 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-06-03 15:36:11.016 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2021-06-03 15:36:11.408 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2021-06-03 15:36:11.628 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2021-06-03 15:36:11.777 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.13.0) {:es_version=>7}
[WARN ] 2021-06-03 15:36:11.780 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[WARN ] 2021-06-03 15:36:11.906 [[main]-pipeline-manager] elasticsearch - Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[WARN ] 2021-06-03 15:36:11.960 [Ruby-0-Thread-10: :1] elasticsearch - Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[INFO ] 2021-06-03 15:36:11.982 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/databeats.conf"], :thread=>"#<Thread:0xb786d77 run>"}
[INFO ] 2021-06-03 15:36:12.075 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2021-06-03 15:36:12.847 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.86}
[INFO ] 2021-06-03 15:36:12.882 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2021-06-03 15:36:12.917 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-06-03 15:36:12.952 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-06-03 15:36:13.134 [[main]<beats] Server - Starting server on port: 5044