Filebeat not reading the input log file

I am trying to fetch data using beats input from a log file whose path I have given in filebeat.yml. But the log file is not read by the logstash while running.

 filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /etc/logstash/filedata.log
output.elasticsearch:
   hosts: ["localhost:9200"]

This is my log filedata.log for input

  this is file beat data

This is my logstash configuration file

input {
  beats {
    port => 5044
  }
}

output {
 elasticsearch {
    hosts => ["http://localhost:9200"]
}}

The output I am getting after running the logstash configuration file is this

/usr/share/logstash/bin/logstash -f /etc/logstash/databeats.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-06-03 15:36:07.706 [main] runner - Starting Logstash {"logstash.version"=>"7.13.0", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-06-03 15:36:07.995 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-06-03 15:36:09.593 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-06-03 15:36:10.155 [Converge PipelineAction::Create<main>] Reflections - Reflections took 33 ms to scan 1 urls, producing 24 keys and 48 values
[WARN ] 2021-06-03 15:36:10.823 [Converge PipelineAction::Create<main>] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-06-03 15:36:10.910 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-06-03 15:36:11.016 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2021-06-03 15:36:11.408 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2021-06-03 15:36:11.628 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2021-06-03 15:36:11.777 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.13.0) {:es_version=>7}
[WARN ] 2021-06-03 15:36:11.780 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[WARN ] 2021-06-03 15:36:11.906 [[main]-pipeline-manager] elasticsearch - Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[WARN ] 2021-06-03 15:36:11.960 [Ruby-0-Thread-10: :1] elasticsearch - Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[INFO ] 2021-06-03 15:36:11.982 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/databeats.conf"], :thread=>"#<Thread:0xb786d77 run>"}
[INFO ] 2021-06-03 15:36:12.075 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2021-06-03 15:36:12.847 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.86}
[INFO ] 2021-06-03 15:36:12.882 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2021-06-03 15:36:12.917 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-06-03 15:36:12.952 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-06-03 15:36:13.134 [[main]<beats] Server - Starting server on port: 5044

@Ashwani_Shukla , I see you are using the output to elasticsearch option in beats instead of Logstash. More on the output to Logstash here Configure the Logstash output | Filebeat Reference [7.13] | Elastic.
Can you follow the steps there and let us know if things are working?

That means in filebeat.yml file I should use logstash output.
And in logstash configuration file I should write like this

output {
 file{ 
        path => "/etc/logstash/outputdata.log"
    }
}

I did like this nothing has happened.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.