Logstash 8.10.3 ERROR Badly formatted index, after interpolation still contains placeholder

good day!
I am trying to extract data from redis using logstash, the data comes from an apm version 8.10.3 but I receive a warining that does not allow me to see the data in kibana.
The log I receive is the following:

[2023-12-14T08:13:27,086][WARN ][logstash.outputs.elasticsearch][main][3f84191c12101efd5465b694189618aac7065894dd23dded84c55dc6a110b194] Badly formatted index, after interpolation still contains placeholder: [apm-8.10.3-%{[processor][event]}-2023.12.13]; event: `{"observer"=>{"hostname"=>"ithrtc3aen1elk1", "type"=>"apm-server", "version"=>"8.10.3"}, "process"=>{"pid"=>26872, "title"=>"/home/elk/jdk1.8.0_161/jre/bin/java"}, "agent"=>{"activation_method"=>"javaagent-flag", "name"=>"java", "ephemeral_id"=>"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e", "version"=>"1.44.0"}, "@timestamp"=>2023-12-13T22:17:51.407Z, "data_stream"=>{"namespace"=>"default", "type"=>"metrics", "dataset"=>"apm.internal"}, "service"=>{"node"=>{"name"=>"ithrtc3aen1elk5"}, "name"=>"PruebaPAM", "runtime"=>{"name"=>"Java", "version"=>"1.8.0_161"}, "language"=>{"name"=>"Java", "version"=>"1.8.0_161"}}, "@metadata"=>{"version"=>"8.10.3", "type"=>"_doc", "beat"=>"apm-server", "indice"=>"apm-8.10.3-%{[processor][event]}-2023.12.13"}, "@version"=>"1", "host"=>{"hostname"=>"ithrtc3aen1elk5", "os"=>{"platform"=>"Linux"}, "ip"=>["10.191.147.52"], "name"=>"ithrtc3aen1elk5", "architecture"=>"amd64"}, "metricset"=>{"name"=>"app", "samples"=>[{"name"=>"jvm.memory.non_heap.pool.committed", "value"=>45350912.0}, {"name"=>"jvm.memory.non_heap.pool.used", "value"=>43672384.0}]}, "event"=>{"original"=>"{\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"@metadata\":{\"beat\":\"apm-server\",\"type\":\"_doc\",\"version\":\"8.10.3\"},\"data_stream\":{\"dataset\":\"apm.internal\",\"namespace\":\"default\",\"type\":\"metrics\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"ithrtc3aen1elk5\",\"ip\":[\"10.191.147.52\"],\"name\":\"ithrtc3aen1elk5\",\"os\":{\"platform\":\"Linux\"}},\"labels\":{\"name\":\"Metaspace\"},\"process\":{\"pid\":26872,\"title\":\"/home/elk/jdk1.8.0_161/jre/bin/java\"},\"@timestamp\":\"2023-12-13T22:17:51.407Z\",\"agent\":{\"name\":\"java\",\"version\":\"1.44.0\",\"activation_method\":\"javaagent-flag\",\"ephemeral_id\":\"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e\"},\"metricset\":{\"name\":\"app\",\"samples\":[{\"name\":\"jvm.memory.non_heap.pool.committed\",\"value\":4.5350912e+07},{\"name\":\"jvm.memory.non_heap.pool.used\",\"value\":4.3672384e+07}]},\"observer\":{\"type\":\"apm-server\",\"version\":\"8.10.3\",\"hostname\":\"ithrtc3aen1elk1\"},\"service\":{\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"language\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"name\":\"PruebaPAM\",\"node\":{\"name\":\"ithrtc3aen1elk5\"}}}"}, "labels"=>{"name"=>"Metaspace"}}`

I want to form the index with the events, for example:
*apm-8.10.3-span-yyyy.MM.dd
*apm-8.10.3-error-yyyy.MM.dd
*apm-8.10.3-transaction-yyyy.MM.dd

my logstash file configuration is:

input {
        redis {
               host => ["10.191.147.52"]
               db => 0
               key => "logstash_apm"
               password => "password"
               port => 6379
               data_type => "list"
               threads => 12
        }
        redis {
              host => ["10.191.147.47"]
              db => 0
              key => "logstash_apm"
              password => "password"
              port => 6379
              data_type => "list"
              threads => 12
        }
}
filter {
        mutate {
                       add_field => { "[@metadata][indice]" => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}"}
                }
}
output {
        stdout {
               codec => rubydebug
        }
        elasticsearch {
                hosts => ["10.119.131.8:9201", "10.119.131.10:9260"]
                index => "%{[@metadata][indice]}"
                user => "user"
                password => "password"
        }
}

i need a help

In your mutate+add_field to create [indice] you have three interpolations, but for the event in question only two are interpolated, because it does not have a [processor] field. The result is an invalid index name.

So the correct thing would be to see it like this?

add_field => { "[@metadata][index]" => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}"}

The main issue here is that you are using %{[processor][event} in the name of your index, but this field does not exist in your document.

If you want to have index named like this:

*apm-8.10.3-span-yyyy.MM.dd
*apm-8.10.3-error-yyyy.MM.dd
*apm-8.10.3-transaction-yyyy.MM.dd

You need to have the span, error and transaction information somewhere, which the document you shared does not have it.

I have seen the logstash output of the data it extracts from Redis, I have executed some methods of the app that I monitored and whether the "transaction" and "span" events exist.
I show logstash log:

{
          "agent" => {
             "ephemeral_id" => "7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e",
                  "version" => "1.44.0",
                     "name" => "java",
        "activation_method" => "javaagent-flag"
    },
       "observer" => {
        "hostname" => "ithrtc3aen1elk1",
         "version" => "8.10.3",
            "type" => "apm-server"
    },
    "transaction" => {
                  "span_count" => {
            "started" => 0,
            "dropped" => 0
        },
                          "id" => "ffbb621d445f2ba0",
                     "sampled" => true,
                        "name" => "HelloWorldExample#doGet",
                        "type" => "request",
        "representative_count" => 1,
                      "result" => "HTTP 2xx"
    },
           "host" => {
            "hostname" => "ithrtc3aen1elk5",
                  "os" => {
            "platform" => "Linux"
        },
        "architecture" => "amd64",
                  "ip" => [
            [0] "x.x.x.x"
        ],
                "name" => "ithrtc3aen1elk5"
    },
        "process" => {
        "title" => "/home/elk/jdk1.8.0_161/jre/bin/java",
          "pid" => 26872
    },
         "source" => {
        "ip" => "x.x.x.x"
    },
        "service" => {
         "language" => {
            "version" => "1.8.0_161",
               "name" => "Java"
        },
        "framework" => {
            "name" => "Servlet API"
        },
             "node" => {
            "name" => "ithrtc3aen1elk5"
        },
             "name" => "PruebaPAM",
          "runtime" => {
            "version" => "1.8.0_161",
               "name" => "Java"
        }
    },
           "span" => {
        "id" => "ffbb621d445f2ba0"
    },
         "client" => {
        "ip" => "x.x.x.x"
    },
       "@version" => "1",
           "tags" => [
        [0] "_mutate_error"
    ],
     "user_agent" => {
        "original" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    },
    "data_stream" => {
        "namespace" => "default",
          "dataset" => "apm",
             "type" => "traces"
    },
     "@timestamp" => 2023-12-14T21:57:48.382Z,
          "event" => {
        "duration" => 650000,
         "outcome" => "success",
        "original" => "{\"@timestamp\":\"2023-12-14T21:57:48.382Z\",\"@metadata\":{\"beat\":\"apm-server\",\"type\":\"_doc\",\"version\":\"8.10.3\"},\"http\":{\"request\":{\"headers\":{\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\"],\"Referer\":[\"http://x.x.x.x:8082/examples/servlets/\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"es-ES,es;q=0.9\"],\"Host\":[\"x.x.x.x:8082\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"]},\"method\":\"GET\"},\"response\":{\"headers\":{\"X-Frame-Options\":[\"DENY\"],\"X-Content-Type-Options\":[\"nosniff\"],\"X-Xss-Protection\":[\"1; mode=block\"]},\"headers_sent\":false,\"status_code\":200,\"finished\":true},\"version\":\"1.1\"},\"timestamp\":{\"us\":1.702591068382017e+15},\"url\":{\"domain\":\"x.x.x.x\",\"full\":\"http://x.x.x.x:8082/examples/servlets/servlet/HelloWorldExample\",\"path\":\"/examples/servlets/servlet/HelloWorldExample\",\"port\":8082,\"scheme\":\"http\"},\"user_agent\":{\"original\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\"},\"client\":{\"ip\":\"x.x.x.x\"},\"data_stream\":{\"dataset\":\"apm\",\"namespace\":\"default\",\"type\":\"traces\"},\"event\":{\"duration\":650000,\"outcome\":\"success\"},\"service\":{\"language\":{\"version\":\"1.8.0_161\",\"name\":\"Java\"},\"name\":\"PruebaPAM\",\"node\":{\"name\":\"ithrtc3aen1elk5\"},\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_161\"},\"framework\":{\"name\":\"Servlet API\"}},\"trace\":{\"id\":\"23659953d2223df735feb333ae64e08a\"},\"@timestamp\":\"2023-12-14T21:57:48.382Z\",\"agent\":{\"activation_method\":\"javaagent-flag\",\"ephemeral_id\":\"7d1e59a5-bddb-4664-b8a3-a1f67ffefe1e\",\"name\":\"java\",\"version\":\"1.44.0\"},\"observer\":{\"hostname\":\"ithrtc3aen1elk1\",\"type\":\"apm-server\",\"version\":\"8.10.3\"},\"transaction\":{\"sampled\":true,\"span_count\":{\"dropped\":0,\"started\":0},\"type\":\"request\",\"id\":\"ffbb621d445f2ba0\",\"name\":\"HelloWorldExample#doGet\",\"representative_count\":1,\"result\":\"HTTP 2xx\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"ithrtc3aen1elk5\",\"ip\":[\"10.191.147.52\"],\"name\":\"ithrtc3aen1elk5\",\"os\":{\"platform\":\"Linux\"}},\"process\":{\"pid\":26872,\"title\":\"/home/elk/jdk1.8.0_161/jre/bin/java\"},\"source\":{\"ip\":\"x.x.x.x\"},\"span\":{\"id\":\"ffbb621d445f2ba0\"}}"
    },

In your output you are using the field [processor][event], this field does not exist in the log you shared, that's why you got that error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.