Weird Bug, Field name is blocked, cant use the same name of field it in other pipelines

Hi, I have a pipeline that stores the fields memory_memused_per and memory_swapused_per in an index , in another pipeline that stores data in another index I have tried to use the same name but nothing is indexed. after hours of debugging I just realize if I change the name to mem_memused_per and mem_swapused_per it works without problems, this is the ruby code:

        ruby {
          code => '
            log = event.get("status")
            mem_memused_per = log.scan(/Physical Memory Used\s*%=\s*(-?[\d.]+)/).flatten.first
            mem_swapused_per = log.scan(/Memoria Swap\s*%=\s*(-?[\d.]+)/).flatten.first

            event.set("memory_memused_per", mem_memused_per)
            event.set("memory_swapused_per", mem_swapused_per)

          '
        }

        mutate {
            convert => { "memory_memused_per"  => "float" }
            convert => { "memory_swapused_per" => "float" }
      }

so if i just change the name of the field, and the mutate convert

            event.set("mem_memused_per", mem_memused_per)
            event.set("mem_swapused_per", mem_swapused_per)

the documents are indexed.

Any known bugs related to this?

Logstash 7.5

Have you looked at the mappings?

Perhaps your original names have an incorrect mapping so the fields can not be stored

And when you changed the name you got a correct mapping and thus the can be stored

Hi, thanks for you reply, so i create a new index with no mapping and add these lines to ruby so I can index fields that start with memory an the fields that start with mem, both from the same source:

           memory_memused_per = log.scan(/Physical Memory Used\s*%=\s*(-?[\d.]+)/).flatten.first
           memory_swapused_per = log.scan(/Memoria Swap\s*%=\s*(-?[\d.]+)/).flatten.first


            event.set("memory_memused_per", memory_memused_per)
            event.set("memory_swapused_per", memory_swapused_per)

            event.set("mem_memused_per", memory_memused_per)
            event.set("mem_swapused_per", memory_swapused_per)

the mapping response in devtools:

And then in Discover only the field mem_memused_per appears with data, both fields have the same source, and mapping

Look at the source json in discover....

Is elasticsearch 7.5 as well?... could be a bug... but seems awfully "weird" as you say
7.5 is pretty old... any chance run a quick new elasticsearch?

Oh...I feel very dumb right now, In the same pipeline was another ruby filter using the same field and parsing another type of log :man_facepalming:, thanks for your anwers and sorry to waste your time brother.

1 Like

No worries... We've been running into a lot of these things. It's good to just have someone help you find them.

No waste of time. You're a good member of the community!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.