I’m working on a logging solution that is really nothing more than a forwarder. rsyslog on several servers are sending their logs to Logstash, which then send them out to HEC splunk endpoint.
In Logstash, we are using the TCP endpoint without persistent queues. We are not doing any filtering at all.
This is an AWS m5.xlarge instance (4 vCPU, 16 GiB Ram, iot type disk). There is next to no load on the box.
The problem it that some items arriving into the Splunk endpoint are being ingested up to 10 hours after they were being created. This only happens on certain “source types”.
My understanding of this use case logstash is that, logstash does not care about ‘source-types’ it takes logs in and sends them out. Is there any reason for any type of message as being delayed?
jvm.options
-server
-Xms8g
-Xmx8g
-XX:+UseShenandoahGC
-XX:+AlwaysPreTouch
-XX:+UseNUMA
-XX:-UseBiasedLocking
-Duser.language=en
-Duser.country=US
#-Djava.io.tmpdir=/opt/logstash/tmp
-Dfile.encoding=UTF-8
-Djruby.compile.invokedynamic=true
-Djruby.jit.threshold=0
-Djruby.regexp.interruptible=true
-XX:+HeapDumpOnOutOfMemoryError
-Djava.security.egd=file:/dev/urandom
-Dlog4j2.isThreadContextMapInheritable=true
11-:--add-opens=java.base/java.security=ALL-UNNAMED
11-:--add-opens=java.base/java.io=ALL-UNNAMED
11-:--add-opens=java.base/java.nio.channels=ALL-UNNAMED
11-:--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
11-:--add-opens=java.managemen
logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
pipeline.ordered: false
#pipeline.workers: 12
pipeline.workers: 40
pipeline.batch.size: 1536
log.level: info
pipeline.ecs_compatibility: disabled
dead_letter_queue.enable: true
conf.d/pipeline.conf
input {
tcp {
port => 1514
}
}
filter {
}
output {
http {
http_method => “post”
url => “https://mysecret_hec_endpoint/services/collector/event/1.0”
headers => [“Authorization”, “Splunk my-super-secret-hec-toke”]
mapping => {
event => “%{message}”
# logstash_time => “%{@timestamp}”
}
}
}