good day!
I am trying to extract data from redis using logstash, the data comes from an apm version 8.10.3 but I receive a warining that does not allow me to see the data in kibana.
The log I receive is the following:
[2023-12-14T08:13:27,086][WARN ][logstash.outputs.elasticsearch][main][3f84191c12101efd5465b694189618aac7065894dd23dded84c55dc6a110b194] Badly formatted index, after interpolation still contains placeholder: [apm-8.10.3-%{[processor][event]}-2023.12.13]; event: `{"observer"=>{"hostname"=>"ithrtc3aen1elk1", "type"=>"apm-server", "version"=>"8.10.3"}, "process"=>{"pid"=>26872, "title"=>"/home/elk/jdk1.8.0_161/jre/bin/java"}, "agent"=>{"activation_method"=>"javaagent-flag", "name"=>"java", "ephemeral_id"=>"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e", "version"=>"1.44.0"}, "@timestamp"=>2023-12-13T22:17:51.407Z, "data_stream"=>{"namespace"=>"default", "type"=>"metrics", "dataset"=>"apm.internal"}, "service"=>{"node"=>{"name"=>"ithrtc3aen1elk5"}, "name"=>"PruebaPAM", "runtime"=>{"name"=>"Java", "version"=>"1.8.0_161"}, "language"=>{"name"=>"Java", "version"=>"1.8.0_161"}}, "@metadata"=>{"version"=>"8.10.3", "type"=>"_doc", "beat"=>"apm-server", "indice"=>"apm-8.10.3-%{[processor][event]}-2023.12.13"}, "@version"=>"1", "host"=>{"hostname"=>"ithrtc3aen1elk5", "os"=>{"platform"=>"Linux"}, "ip"=>["10.191.147.52"], "name"=>"ithrtc3aen1elk5", "architecture"=>"amd64"}, "metricset"=>{"name"=>"app", "samples"=>[{"name"=>"jvm.memory.non_heap.pool.committed", "value"=>45350912.0}, {"name"=>"jvm.memory.non_heap.pool.used", "value"=>43672384.0}]}, "event"=>{"original"=>"{\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"@metadata\":{\"beat\":\"apm-server\",\"type\":\"_doc\",\"version\":\"8.10.3\"},\"data_stream\":{\"dataset\":\"apm.internal\",\"namespace\":\"default\",\"type\":\"metrics\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"ithrtc3aen1elk5\",\"ip\":[\"10.191.147.52\"],\"name\":\"ithrtc3aen1elk5\",\"os\":{\"platform\":\"Linux\"}},\"labels\":{\"name\":\"Metaspace\"},\"process\":{\"pid\":26872,\"title\":\"/home/elk/jdk1.8.0_161/jre/bin/java\"},\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"agent\":{\"name\":\"java\",\"version\":\"1.44.0\",\"activation_method\":\"javaagent-flag\",\"ephemeral_id\":\"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e\"},\"metricset\":{\"name\":\"app\",\"samples\":[{\"name\":\"jvm.memory.non_heap.pool.committed\",\"value\":4.5350912e+07},{\"name\":\"jvm.memory.non_heap.pool.used\",\"value\":4.3672384e+07}]},\"observer\":{\"type\":\"apm-server\",\"version\":\"8.10.3\",\"hostname\":\"ithrtc3aen1elk1\"},\"service\":{\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"language\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"name\":\"PruebaPAM\",\"node\":{\"name\":\"ithrtc3aen1elk5\"}}}"}, "labels"=>{"name"=>"Metaspace"}}`
I want to form the index with the events, for example:
*apm-8.10.3-span-yyyy.MM.dd
*apm-8.10.3-error-yyyy.MM.dd
*apm-8.10.3-transaction-yyyy.MM.dd
my logstash file configuration is:
input {
redis {
host => ["10.191.147.52"]
db => 0
key => "logstash_apm"
password => "password"
port => 6379
data_type => "list"
threads => 12
}
redis {
host => ["10.191.147.47"]
db => 0
key => "logstash_apm"
password => "password"
port => 6379
data_type => "list"
threads => 12
}
}
filter {
mutate {
add_field => { "[@metadata][indice]" => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}"}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["10.119.131.8:9201", "10.119.131.10:9260"]
index => "%{[@metadata][indice]}"
user => "user"
password => "password"
}
}
i need a help