Logstash not able to pull multiple beats data (ELK 8.6)

Elastic Stack Logstash
1

Hello all:

I am trying to get my filebeat and metricbeat data from another server into logstash . I am using below command line at the logstash bin folder. And I got separate conf file for filebeat and metricbeat on my logstash CONF folder.

logstash.bat -f .\config\filebeatLogstash.conf -f .\config\metricbeatLogstash.conf

When I execute it only find the metricbeatlogstash and parse the data into kibana, not the filebeat.

But when I execute one by one ie, filebeat and metricbeat separately - it recognizes and fetches the data from source and shows up on kibana.

Could someone tell me what is the reason and how to solve it ?

logstash.yml given below

node.name: node01
path.data: C:\elastic\logstash-8.6.2-windows-x86_64\logstash-8.6.2\data
pipeline.id: main
3

And this is the logstash-plain.log when I execute the logstash.bat command. As you can see it recognizes only metricbeat conf file.

[2023-05-03T22:49:35,800][INFO ][logstash.runner          ] Log4j configuration path used is: C:\elastic\logstash-8.6.2-windows-x86_64\logstash-8.6.2\config\log4j2.properties
[2023-05-03T22:49:35,800][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-05-03T22:49:35,816][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-mswin32]"}
[2023-05-03T22:49:35,816][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-05-03T22:49:35,863][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-05-03T22:49:36,991][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-05-03T22:49:37,574][INFO ][org.reflections.Reflections] Reflections took 114 ms to scan 1 urls, producing 127 keys and 444 values
[2023-05-03T22:49:38,166][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-05-03T22:49:38,166][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://30.131.16.3:9200"]}
[2023-05-03T22:49:38,244][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@30.131.16.3:9200/]}}
[2023-05-03T22:49:38,336][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@30.131.16.3:9200/"}
[2023-05-03T22:49:38,336][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.6.2) {:es_version=>8}
[2023-05-03T22:49:38,336][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-05-03T22:49:38,352][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}"}
[2023-05-03T22:49:38,352][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-05-03T22:49:38,352][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-05-03T22:49:38,367][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/config/metricbeatLogstash.conf"], :thread=>"#<Thread:0x300d7255@C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-03T22:49:39,125][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.76}
[2023-05-03T22:49:39,157][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"10.111.206.5:5045"}
[2023-05-03T22:49:39,157][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-05-03T22:49:39,250][INFO ][org.logstash.beats.Server][main][5b557c9f54726c5dda08d9791dd98e05bb6dd51520eca5c33752b8f92043aef7] Starting server on port: 5045
[2023-05-03T22:49:39,260][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
4

If you want to use multiple configuration files then put them into a directory that does not contain anything else and point -f at the directory rather than the file. logstash will combine them into a single configuration, so data from both inputs will be sent to all of the outputs in both files.

If you want them to be run separately then use pipelines.yml to define two pipelines.

5

As suggested I moved the filebeat and winlogbeat conf to a separate folder with CONFIG folder of logstash and executed as suggested by point -f at the directory. look like that helped to pick both conf files.

but now I can see only filebeat indices, but nothing from winlogbeat on kibana side.

I can see winlogbeat.json files getting created at winlogbeat source side and ports are established. there are no errors on logstash plain log as shown below.

23-05-04T07:05:04,115][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-05-04T07:05:04,130][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/config/beats_conf/filebeatlogstash.conf", "C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/config/beats_conf/winlogbeatLogstash.conf"], :thread=>"#<Thread:0x4f6cc439@C:/elastic/logstash-8.6.2-windows-x86_64/logstash-8.6.2/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-04T07:05:05,876][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.74}
[2023-05-04T07:05:05,876][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"10.111.206.5:5046"}
[2023-05-04T07:05:05,891][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"10.111.206.5:5045"}
[2023-05-04T07:05:05,952][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-05-04T07:05:05,971][INFO ][org.logstash.beats.Server][main][ca3048e0bd12e112ab94f70325a2683db54238889fb66e21edec83f4f8a42432] Starting server on port: 5046
[2023-05-04T07:05:06,002][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-05-04T07:05:06,018][INFO ][org.logstash.beats.Server][main][4971e405c01f6c6214f3a7332c0fabe1901321a5274cc0e31980a2732f126fac] Starting server on port: 5045
6

Solved. Output tag in winlogbeat conf file was mentioned incorrectly.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.