Logstash 8.10.3 ERROR Badly formatted index, after interpolation still contains placeholder

Elastic Stack Logstash
1

good day!
I am trying to extract data from redis using logstash, the data comes from an apm version 8.10.3 but I receive a warining that does not allow me to see the data in kibana.
The log I receive is the following:

[2023-12-14T08:13:27,086][WARN ][logstash.outputs.elasticsearch][main][3f84191c12101efd5465b694189618aac7065894dd23dded84c55dc6a110b194] Badly formatted index, after interpolation still contains placeholder: [apm-8.10.3-%{[processor][event]}-2023.12.13]; event: `{"observer"=>{"hostname"=>"ithrtc3aen1elk1", "type"=>"apm-server", "version"=>"8.10.3"}, "process"=>{"pid"=>26872, "title"=>"/home/elk/jdk1.8.0_161/jre/bin/java"}, "agent"=>{"activation_method"=>"javaagent-flag", "name"=>"java", "ephemeral_id"=>"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e", "version"=>"1.44.0"}, "@timestamp"=>2023-12-13T22:17:51.407Z, "data_stream"=>{"namespace"=>"default", "type"=>"metrics", "dataset"=>"apm.internal"}, "service"=>{"node"=>{"name"=>"ithrtc3aen1elk5"}, "name"=>"PruebaPAM", "runtime"=>{"name"=>"Java", "version"=>"1.8.0_161"}, "language"=>{"name"=>"Java", "version"=>"1.8.0_161"}}, "@metadata"=>{"version"=>"8.10.3", "type"=>"_doc", "beat"=>"apm-server", "indice"=>"apm-8.10.3-%{[processor][event]}-2023.12.13"}, "@version"=>"1", "host"=>{"hostname"=>"ithrtc3aen1elk5", "os"=>{"platform"=>"Linux"}, "ip"=>["10.191.147.52"], "name"=>"ithrtc3aen1elk5", "architecture"=>"amd64"}, "metricset"=>{"name"=>"app", "samples"=>[{"name"=>"jvm.memory.non_heap.pool.committed", "value"=>45350912.0}, {"name"=>"jvm.memory.non_heap.pool.used", "value"=>43672384.0}]}, "event"=>{"original"=>"{\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"@metadata\":{\"beat\":\"apm-server\",\"type\":\"_doc\",\"version\":\"8.10.3\"},\"data_stream\":{\"dataset\":\"apm.internal\",\"namespace\":\"default\",\"type\":\"metrics\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"ithrtc3aen1elk5\",\"ip\":[\"10.191.147.52\"],\"name\":\"ithrtc3aen1elk5\",\"os\":{\"platform\":\"Linux\"}},\"labels\":{\"name\":\"Metaspace\"},\"process\":{\"pid\":26872,\"title\":\"/home/elk/jdk1.8.0_161/jre/bin/java\"},\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"agent\":{\"name\":\"java\",\"version\":\"1.44.0\",\"activation_method\":\"javaagent-flag\",\"ephemeral_id\":\"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e\"},\"metricset\":{\"name\":\"app\",\"samples\":[{\"name\":\"jvm.memory.non_heap.pool.committed\",\"value\":4.5350912e+07},{\"name\":\"jvm.memory.non_heap.pool.used\",\"value\":4.3672384e+07}]},\"observer\":{\"type\":\"apm-server\",\"version\":\"8.10.3\",\"hostname\":\"ithrtc3aen1elk1\"},\"service\":{\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"language\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"name\":\"PruebaPAM\",\"node\":{\"name\":\"ithrtc3aen1elk5\"}}}"}, "labels"=>{"name"=>"Metaspace"}}`

I want to form the index with the events, for example:
*apm-8.10.3-span-yyyy.MM.dd
*apm-8.10.3-error-yyyy.MM.dd
*apm-8.10.3-transaction-yyyy.MM.dd

my logstash file configuration is:

input {
        redis {
               host => ["10.191.147.52"]
               db => 0
               key => "logstash_apm"
               password => "password"
               port => 6379
               data_type => "list"
               threads => 12
        }
        redis {
              host => ["10.191.147.47"]
              db => 0
              key => "logstash_apm"
              password => "password"
              port => 6379
              data_type => "list"
              threads => 12
        }
}
filter {
        mutate {
                       add_field => { "[@metadata][indice]" => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}"}
                }
}
output {
        stdout {
               codec => rubydebug
        }
        elasticsearch {
                hosts => ["10.119.131.8:9201", "10.119.131.10:9260"]
                index => "%{[@metadata][indice]}"
                user => "user"
                password => "password"
        }
}

i need a help

2

In your mutate+add_field to create [indice] you have three interpolations, but for the event in question only two are interpolated, because it does not have a [processor] field. The result is an invalid index name.

3

So the correct thing would be to see it like this?

add_field => { "[@metadata][index]" => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}"}

4

The main issue here is that you are using %{[processor][event} in the name of your index, but this field does not exist in your document.

If you want to have index named like this:

*apm-8.10.3-span-yyyy.MM.dd
*apm-8.10.3-error-yyyy.MM.dd
*apm-8.10.3-transaction-yyyy.MM.dd

You need to have the span, error and transaction information somewhere, which the document you shared does not have it.

5

I have seen the logstash output of the data it extracts from Redis, I have executed some methods of the app that I monitored and whether the "transaction" and "span" events exist.
I show logstash log:

{
          "agent" => {
             "ephemeral_id" => "7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e",
                  "version" => "1.44.0",
                     "name" => "java",
        "activation_method" => "javaagent-flag"
    },
       "observer" => {
        "hostname" => "ithrtc3aen1elk1",
         "version" => "8.10.3",
            "type" => "apm-server"
    },
    "transaction" => {
                  "span_count" => {
            "started" => 0,
            "dropped" => 0
        },
                          "id" => "ffbb621d445f2ba0",
                     "sampled" => true,
                        "name" => "HelloWorldExample#doGet",
                        "type" => "request",
        "representative_count" => 1,
                      "result" => "HTTP 2xx"
    },
           "host" => {
            "hostname" => "ithrtc3aen1elk5",
                  "os" => {
            "platform" => "Linux"
        },
        "architecture" => "amd64",
                  "ip" => [
            [0] "x.x.x.x"
        ],
                "name" => "ithrtc3aen1elk5"
    },
        "process" => {
        "title" => "/home/elk/jdk1.8.0_161/jre/bin/java",
          "pid" => 26872
    },
         "source" => {
        "ip" => "x.x.x.x"
    },
        "service" => {
         "language" => {
            "version" => "1.8.0_161",
               "name" => "Java"
        },
        "framework" => {
            "name" => "Servlet API"
        },
             "node" => {
            "name" => "ithrtc3aen1elk5"
        },
             "name" => "PruebaPAM",
          "runtime" => {
            "version" => "1.8.0_161",
               "name" => "Java"
        }
    },
           "span" => {
        "id" => "ffbb621d445f2ba0"
    },
         "client" => {
        "ip" => "x.x.x.x"
    },
       "@version" => "1",
           "tags" => [
        [0] "_mutate_error"
    ],
     "user_agent" => {
        "original" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    },
    "data_stream" => {
        "namespace" => "default",
          "dataset" => "apm",
             "type" => "traces"
    },
     "@timestamp" => 2023-12-14T21:57:48.382Z,
          "event" => {
        "duration" => 650000,
         "outcome" => "success",
        "original" => "{\"@timestamp\":\"2023-12-14T21:57:48.382Z\",\"@metadata\":{\"beat\":\"apm-server\",\"type\":\"_doc\",\"version\":\"8.10.3\"},\"http\":{\"request\":{\"headers\":{\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\"],\"Referer\":[\"http://x.x.x.x:8082/examples/servlets/\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"es-ES,es;q=0.9\"],\"Host\":[\"x.x.x.x:8082\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"]},\"method\":\"GET\"},\"response\":{\"headers\":{\"X-Frame-Options\":[\"DENY\"],\"X-Content-Type-Options\":[\"nosniff\"],\"X-Xss-Protection\":[\"1; mode=block\"]},\"headers_sent\":false,\"status_code\":200,\"finished\":true},\"version\":\"1.1\"},\"timestamp\":{\"us\":1.702591068382017e+15},\"url\":{\"domain\":\"x.x.x.x\",\"full\":\"http://x.x.x.x:8082/examples/servlets/servlet/HelloWorldExample\",\"path\":\"/examples/servlets/servlet/HelloWorldExample\",\"port\":8082,\"scheme\":\"http\"},\"user_agent\":{\"original\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\"},\"client\":{\"ip\":\"x.x.x.x\"},\"data_stream\":{\"dataset\":\"apm\",\"namespace\":\"default\",\"type\":\"traces\"},\"event\":{\"duration\":650000,\"outcome\":\"success\"},\"service\":{\"language\":{\"version\":\"1.8.0_161\",\"name\":\"Java\"},\"name\":\"PruebaPAM\",\"node\":{\"name\":\"ithrtc3aen1elk5\"},\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"framework\":{\"name\":\"Servlet API\"}},\"trace\":{\"id\":\"23659953d2223df735feb333ae64e08a\"},\"@timestamp\":\"2023-12-14T21:57:48.382Z\",\"agent\":{\"activation_method\":\"javaagent-flag\",\"ephemeral_id\":\"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e\",\"name\":\"java\",\"version\":\"1.44.0\"},\"observer\":{\"hostname\":\"ithrtc3aen1elk1\",\"type\":\"apm-server\",\"version\":\"8.10.3\"},\"transaction\":{\"sampled\":true,\"span_count\":{\"dropped\":0,\"started\":0},\"type\":\"request\",\"id\":\"ffbb621d445f2ba0\",\"name\":\"HelloWorldExample#doGet\",\"representative_count\":1,\"result\":\"HTTP 2xx\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"ithrtc3aen1elk5\",\"ip\":[\"10.191.147.52\"],\"name\":\"ithrtc3aen1elk5\",\"os\":{\"platform\":\"Linux\"}},\"process\":{\"pid\":26872,\"title\":\"/home/elk/jdk1.8.0_161/jre/bin/java\"},\"source\":{\"ip\":\"x.x.x.x\"},\"span\":{\"id\":\"ffbb621d445f2ba0\"}}"
    },
6

In your output you are using the field [processor][event], this field does not exist in the log you shared, that's why you got that error.