Help receiving logs in Logstash deployed in Heroku

Elastic Stack Logstash
1

Hi! I just deployed my Logstash project in Heroku. My plan is use it to process and send the logs to my Elastic Cloud deploy (Kibana and Elasticsearch).

Logstash is running successfully. These are the Logstash logs that appear in the Heroku Application Logs section:

2023-03-01T16:32:54.000000+00:00 app[api]: Build succeeded
2023-03-01T16:33:07.240662+00:00 heroku[worker.1]: Starting process with command `bin/logstash -f logstash.conf`
2023-03-01T16:33:07.837277+00:00 heroku[worker.1]: State changed from starting to up
2023-03-01T16:33:08.217431+00:00 app[worker.1]: Using system java: /app/.jdk/bin/java
2023-03-01T16:33:08.217455+00:00 app[worker.1]: warning: ignoring JAVA_TOOL_OPTIONS=-XX:+UseContainerSupport -Xmx671m -XX:CICompilerCount=2 -Dfile.encoding=UTF-8
2023-03-01T16:33:08.217516+00:00 app[worker.1]: warning: ignoring JAVA_OPTS=-XX:+UseContainerSupport -Xmx671m -XX:CICompilerCount=2 -Dfile.encoding=UTF-8 ; pass JVM parameters via LS_JAVA_OPTS
2023-03-01T16:33:08.535286+00:00 app[worker.1]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2023-03-01T16:33:24.716067+00:00 app[worker.1]: Sending Logstash logs to /app/logs which is now configured via log4j2.properties
2023-03-01T16:33:24.817037+00:00 app[worker.1]: [2023-03-01T16:33:24,814][INFO ][logstash.runner          ] Log4j configuration path used is: /app/config/log4j2.properties
2023-03-01T16:33:24.828384+00:00 app[worker.1]: [2023-03-01T16:33:24,828][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
2023-03-01T16:33:24.829670+00:00 app[worker.1]: [2023-03-01T16:33:24,829][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 11.0.18+10-LTS on 11.0.18+10-LTS +indy +jit [x86_64-linux]"}
2023-03-01T16:33:24.831877+00:00 app[worker.1]: [2023-03-01T16:33:24,831][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
2023-03-01T16:33:24.850558+00:00 app[worker.1]: [2023-03-01T16:33:24,849][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/app/data/queue"}
2023-03-01T16:33:24.851629+00:00 app[worker.1]: [2023-03-01T16:33:24,851][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/app/data/dead_letter_queue"}
2023-03-01T16:33:25.094899+00:00 app[worker.1]: [2023-03-01T16:33:25,094][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2023-03-01T16:33:25.613673+00:00 app[worker.1]: [2023-03-01T16:33:25,607][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-03-01T16:33:25.898432+00:00 app[worker.1]: [2023-03-01T16:33:25,898][INFO ][org.reflections.Reflections] Reflections took 149 ms to scan 1 urls, producing 127 keys and 444 values
2023-03-01T16:33:26.304215+00:00 app[worker.1]: [2023-03-01T16:33:26,304][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
2023-03-01T16:33:26.334852+00:00 app[worker.1]: [2023-03-01T16:33:26,334][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/app/logstash.conf"], :thread=>"#<Thread:0x9261f3f@/app/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
2023-03-01T16:33:26.921830+00:00 app[worker.1]: [2023-03-01T16:33:26,921][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.59}
2023-03-01T16:33:26.965656+00:00 app[worker.1]: [2023-03-01T16:33:26,965][INFO ][logstash.codecs.json     ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
2023-03-01T16:33:27.266185+00:00 app[worker.1]: [2023-03-01T16:33:27,265][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
2023-03-01T16:33:27.270862+00:00 app[worker.1]: [2023-03-01T16:33:27,270][INFO ][logstash.inputs.http     ][main][af92ecabe36b7ae9bc324a41a5eb6754618416463ac56cc537b12d00af175ed5] Starting http input listener {:address=>"0.0.0.0:46110", :ssl=>"false"}
2023-03-01T16:33:27.282084+00:00 app[worker.1]: [2023-03-01T16:33:27,281][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

But, when I try to send a log via http, nothing happens.

This is my conf file.

input {
  http {
      host => "0.0.0.0"
      port => "${PORT}"
  }
}
output {
  stdout {}
}

I would greatly appreciate a hand. Thank you so much.

2

From where you tried? I'm not sure how heroku works, but do you have any firewall where you need to allow traffic?

Try to send something via http directly in the Logstash server, if it works, then Logstash is working fine and you may have a network/infrastructure issue.

3

I'm using Postman. Actually, it works to send logs to my local project. And it doesn't have a firewall.

Im sending the logs using a Post to https://mydomain.herokuapp.com:PORT

4

You are sending using https, but your input is not configured to use https, try to use http instead.

5

Same result. Do you think I need to change something in logstash.yml?

6

What is the result you got in Postman? Do you got a timeout?

If you make a request and Logstash is not receiving you will get a timeout, if you are not getting a timeout, then you may have other issues. Do you have anything else in the logs?

There is nothing to change in logstash.yml, this file has settings for the logstash service only.

7

Hi! I have an update.

I resolved the issue:

Heroku needs a Procfile to know how to start the app. I configured it in the following way:
worker: bin/logstash -f logstash.conf

However, it was not the correct process. It had to start with the type "web", instead of "worker". It already allows me to receive the logs and use them.

I hope someone will find this information useful.

Thank you so much!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.