# No incoming data to Logstash Output from Elastic Agents - Only Elasticsearch ouptut works **URL:** https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350 **Category:** Elasticsearch **Tags:** fleet **Created:** [April 30, 2023, 12:38pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350 "2023-04-30T12:38:09Z") **Posts on this page:** 7 **Page:** 1
### Author: ![mehdi-lamrani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-lamrani/32/108993_2.png) [@mehdi-lamrani](https://discuss.elastic.co/u/mehdi-lamrani) #### Post date: [April 30, 2023, 12:38pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/1 "2023-04-30T12:38:09Z")
8.7 stack here After I setup a logstash output in Fleet, and set a policy to use that logstash ouptut for integrations, no data comes to it basically. When I switch the output for integrations to Elasticsearch instead of logstash, I get data. I meticulously followed the steps here : > **[Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide...](https://www.elastic.co/guide/en/fleet/current/secure-logstash-connections.html)** _elastic-agent-pipeline-secure.conf_ ```auto input { elastic_agent { port => 5044 ssl => true ssl_certificate_authorities => ["/etc/logstash/certs/ca/ca.crt"] ssl_certificate => "/etc/logstash/certs/logstash.crt" ssl_key => "/etc/logstash/certs/logstash.pkcs8.key" ssl_verify_mode => "force_peer" } } output { elasticsearch { hosts => "https://172.0.0.1:9200" api_key => "xxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxxxxxx" data_stream => true ssl => true cacert => "/etc/logstash/certs/http_ca.crt" } } ``` made sure to reference this pipeline in pipelines.yml, and that it is indeed running without errors by checking on the logs ```auto systemd[1]: Started logstash. logstash[46688]: Using bundled JDK: /usr/share/logstash/jdk logstash[46688]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties [INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.7.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-linux]"} Apr 30 11:36:31 ip-172.0.0.1 logstash[46688]: [2023-04-30T11:36:31,693][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO][org.reflections.Reflections] Reflections took 77 ms to scan 1 urls, producing 132 keys and 462 values [INFO][logstash.javapipeline] Pipeline `elastic-agent-pipeline` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [INFO][logstash.outputs.elasticsearch][elastic-agent-pipeline] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.0.0.1:9200"]} [INFO][logstash.outputs.elasticsearch][elastic-agent-pipeline] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://1172.0.0.1:9200/]}} [WARN][logstash.outputs.elasticsearch][elastic-agent-pipeline] Restored connection to ES instance {:url=>"https://172.0.0.1:9200/"} [INFO][logstash.outputs.elasticsearch][elastic-agent-pipeline] Elasticsearch version determined (8.7.0) {:es_version=>8} [WARN][logstash.outputs.elasticsearch][elastic-agent-pipeline] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8} [WARN][logstash.outputs.elasticsearch][elastic-agent-pipeline] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning. [INFO][logstash.javapipeline][elastic-agent-pipeline] Starting pipeline {:pipeline_id=>"elastic-agent-pipeline", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/elastic-agent-pipeline.conf"], :thread=>"#"} [INFO][logstash.javapipeline][elastic-agent-pipeline] Pipeline Java execution initialization time {"seconds"=>0.54} [INFO][logstash.inputs.beats][elastic-agent-pipeline] Starting input listener {:address=>"0.0.0.0:5044"} [INFO][logstash.javapipeline][elastic-agent-pipeline] Pipeline started {"pipeline.id"=>"elastic-agent-pipeline"} [logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:"elastic-agent-pipeline"], :non_running_pipelines=>[]} [INFO][org.logstash.beats.Server][elastic-agent-pipeline][cacd334d179fceaa1a9b9ebb723484a4a8781322e61ccf95432df67590c08c91] Starting server on port: 5044 ``` I made sure the Elasticsearch / Fleet Server / Logstash machine is accessible from Elastic Agent Host and that the port is effectively open ```auto ubuntu@elastic-agent-host:/var/log$ telnet 172.0.0.1 5044 Trying 172.0.0.1 ... Connected to 172.0.0.2 . Escape character is '^]'. ``` ![Screenshot 2023-04-30 at 13.52.55](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40bacf0b3c7c67c3d80a19836400122efa8e131e.png) ![Screenshot 2023-04-30 at 13.54.40](https://us1.discourse-cdn.com/elastic/original/3X/5/0/504c195e57a17fc9503e3cacf27cf320de243b7b.png) ![Screenshot 2023-04-30 at 14.09.29](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b259e0cc63d05164d677d3789c15664a725c2101.png) ![Screenshot 2023-04-30 at 14.14.00](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66d10e1602c08d130f9a0ab3e52bfa3c28626a58.png) . Now I can wait for hours nothing happens. No Data Stream, No input, Nothing. As you can see below. So this basically tell us the Agent is not sending anything to logstash, or at least logstash is not seeinganything received on its end. ```auto root@mainhost# curl -XGET localhost:9600/_node/stats/events?pretty { "host" : "ip-172-0-0-1", "version" : "8.7.0", "http_address" : "127.0.0.1:9600", "id" : "eac3fd1c-a24a-4e51-8b52-e41aa73b4628", "name" : "ip-172-30-2-238", "ephemeral_id" : "e458c193-b795-43d9-b0b8-5cfa90288070", "status" : "green", "snapshot" : false, "pipeline" : { "workers" : 8, "batch_size" : 125, "batch_delay" : 50 }, "events" : { "in" : 0, "filtered" : 0, "out" : 0, "duration_in_millis" : 92, "queue_push_duration_in_millis" : 0 } ``` . But as soon as I switch the output for integrations to Elasticsearch default, data streams pops out. This is depressing . ![Screenshot 2023-04-30 at 14.17.22](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d4a2e47bf96d20fcd7711919d9550b2dd61d7b6.png) ![Screenshot 2023-04-30 at 14.19.04 1](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1ca20edd282c7f919a70fbd17b83a59fecf1e5c7.png) . I am really out of ideas here. Any help is welcome ---
### Author: ![mehdi-lamrani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-lamrani/32/108993_2.png) [@mehdi-lamrani](https://discuss.elastic.co/u/mehdi-lamrani) #### Post date: [April 30, 2023, 12:56pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/2 "2023-04-30T12:56:34Z")
A couple of things worth noting (or not) : I skipped a couple of fields during the cert process (I dont have an FQDN) ```auto ./bin/elasticsearch-certutil cert \ --name logstash \ --ca-cert /path/to/ca/ca.crt \ --ca-key /path/to/ca/ca.key \ --dns your.host.name.here \ << ### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [April 30, 2023, 4:42pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/3 "2023-04-30T16:42:33Z") Hi @mehdi-lamrani Typo? 5044 vs 5440 Logstash input ```auto input { elastic_agent { port => 5044 ``` Vs agent output in your screenshot `5440` [![](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40bacf0b3c7c67c3d80a19836400122efa8e131e.png) ](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40bacf0b3c7c67c3d80a19836400122efa8e131e.png) Just for reference `5044` is the normal beats port The agent logs / filebeat logs should be showing the bad connection ---
### Author: ![mehdi-lamrani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-lamrani/32/108993_2.png) [@mehdi-lamrani](https://discuss.elastic.co/u/mehdi-lamrani) #### Post date: [April 30, 2023, 6:25pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/4 "2023-04-30T18:25:04Z")
> [@stephenb](#): > > Hi @mehdi-lamrani > Typo? 5044 vs 5440 Are you implying that I spent hours going through each and every corner of the documentation and carefully redoing each and every step, while overlooking a simple port type ? Because if that's what you are implying you would be dead right. I suppose that's why weekends are made to rest 😌 ![Screenshot 2023-04-30 at 20.13.42](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75ecbbc0eaa35e22018ed4db2cd061e822e11130.png) Just to make sure I was braindead while configuring this, I added an integration with a new dataset name and I started getting data right away. I apologize to the community and for wasting your precious time. I need to get some time away from the computer now lol. Thanks a bunch. ---
### Author: ![mehdi-lamrani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-lamrani/32/108993_2.png) [@mehdi-lamrani](https://discuss.elastic.co/u/mehdi-lamrani) #### Post date: [April 30, 2023, 6:28pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/5 "2023-04-30T18:28:32Z")
PS : > [@stephenb](#): > > The agent logs / filebeat logs should be showing the bad connection I beg you pardon ? I tried to scan for this like everywhere. problem is I was being "blind" and not seeing logs that could point out to this direction. Where do I find those agent logs / filebeat logs exactly ? (sorry if the question sounds stupid) ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [April 30, 2023, 9:36pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/6 "2023-04-30T21:36:01Z")
No worries. We call it pair programming 🙂 You are not the first nor the last! The agent logs are a bit buried.... I'm assuming nothing showed up in the normal agent logs? So then you have to drill down under the filebeat portion under the agent and find the logs there. You also may have been able to run the agent diagnostics. Did you look at any of that? > **[Troubleshoot common problems | Fleet and Elastic Agent Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/fleet/current/fleet-troubleshooting.html)** I will sympathize as I still find the agent a bit hard to debug and dev test cycle as well. ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [May 28, 2023, 9:36pm UTC](https://discuss.elastic.co/t/no-incoming-data-to-logstash-output-from-elastic-agents-only-elasticsearch-ouptut-works/331350/7 "2023-05-28T21:36:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.