How to integrate the datadog output in the logstash

Elastic Stack Logstash
1

Greetings, I have deployed only Logstash using helm chart in the Anthos Bare metal cluster.
I want to integrate the Logstash with Datadog platform, for which plugin is installed using offline plugin management. Since our cluster is in air gapped environment.
Datadog output plugin is successfully installed using below command.
bin/logstash-plugin install file:///path/to/logstash-offline-plugins-8.13.4.zip

Issue is after installing the plugin, when I try to configure the output to send logs to the Datadog platform in the helm chart values.yaml of the Logstash, Container is going into crashloopbackoff error.
Please suggest how to configure the output or am I using right deployment method.

Chart used: helm repo add elastic https://helm.elastic.co
Install it: helm install logstash elastic/logstash

output Configuration:

output {
logstash-output-datadog {
api_key => "<DATADOG_API_KEY>"
}
}

Plugin Info:
logstash@logstash-logstash-0:~$ bin/logstash-plugin list | grep -i datadog
logstash-output-datadog

2

You need to share the logstash error log.

3

When output is configured in the values.yaml of the logstash helm chart, Pods is in crashloopbackoff. Hence output configuration is removed and copied the conf file to the below directory and ran the command.

logstash@logstash-logstash-0:~/config$ ls
jvm.options log4j2.properties logstash-datadog.conf logstash-sample.conf logstash.yml pipelines.yml startup.options
logstash@logstash-logstash-0:~/config$ ls -ltr
total 28
-rw-r--r-- 1 logstash root 286 Nov 9 2022 pipelines.yml
-rw-r--r-- 1 logstash root 91 Nov 9 2022 logstash.yml
-rw-r--r-- 1 logstash root 605 Nov 9 2022 log4j2.properties
-rw-rw-r-- 1 logstash root 1696 Nov 9 2022 startup.options
-rw-rw-r-- 1 logstash root 342 Nov 9 2022 logstash-sample.conf
-rw-r--r-- 1 root root 146 May 30 12:22 logstash-datadog.conf
-rw-rw-r-- 1 logstash root 22 Jun 4 10:27 jvm.options
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$ cat logstash-datadog.conf
output {
datadog_logs {
api_key => "api-key"
host => "url
"
}
}logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$ pwd
/usr/share/logstash/config
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$
logstash@logstash-logstash-0:~/config$ logstash -f logstash-datadog.conf
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2024-06-04T13:15:27,537][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2024-06-04T13:15:27,549][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.5.1", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]"}
[2024-06-04T13:15:27,552][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Xmx1g, -Xms1g, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-04T13:15:28,448][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-06-04T13:15:28,456][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2024-06-04T13:15:28,459][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

4

Below is the Pods logs after removing the output configuration

[2024-06-04T10:05:33,046][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2024-06-04T10:06:03,046][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2024-06-04T10:06:03,062][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}