Performing a conditional mutation to circumvent parsing issue

Elastic Stack Logstash
1

Hi,

I'm a newcomer to Logstash, so bear with me.

My logstash logs are reporting:

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mobileapps-prod-logs-current", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3c968aa8>], :response=>{"index"=>{"_index"=>"mobileapps-prod-logs-000049", "_type"=>"_doc", "_id"=>"VpKf_3cB32I4ebBj8VwA", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [fields.EventId] tried to parse field [EventId] as object, but found a concrete value"}}}}

An errant service is proving a scalar string value for the EventId instead of a json object which is what is expected.

What options do I have for renaming it to say, EventIdString - but only where the value is in fact a string and not an object. That is - I want it left alone if EventId is an object.

Example of where I'd want the filter to mutate/rename EventId to EventIdString:

    {
      "@version" => "1",
      "fields" => {
        "IPAddress" => "X.XXX.X.XX",
        "Environment" => "Production",
        "EventId" => "BCC3A1F0E14947FCE053052117ACC53D",
        "QueueWorkerId" => "f111dded-61f8-49f7-98c4-2eeb873938a5",
        "Application" => "XXXXXXXXXXXXXXXXXXXX",
        "HostName" => "xxxxxxxxxxx-worker-564c799ccd-t7stx",
        "BlobEventTime" => "03/05/2021 12:38:20",
        "BlobQueueMessageId" => "XXXXXXXXXXXXXXXXXXXX",
        "CorrelationId" => "XXXXXXXXXXXXXXXXXXXX",
        "SourceContext" => "XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX",
        "ActionId" => "01701330816920210305",
        "CardHash" => nil
      },
      "level" => "Information",
      "message" => "XXXXXXXXXXXXXXXX",
      "@timestamp" => 2021-03-05T04:38:27.221Z,
      "messageTemplate" => "XXXXXXXXXXXXXXXXXXXXXX"
    }

Example of where I'd want EventId left as is:

    {
      "@version" => "1",
      "fields" => {
        "options" => "None",
        "IPAddress" => "X.XXX.X.XX",
        "Environment" => "Production",
        "version" => "3.1.1",
        "EventId" => {
          "Name" => "XXXXXXXXXX",
          "Id" => 10403
        },
      "QueueWorkerId" => "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
      "Application" => "XXXXXXXXXXXXXXXXXXXXX",
      "HostName" => "xxxxxxxx-worker-564c799ccd-t7stx",
      "contextType" => "XXXXXXXXXXXXXXXXX",
      "provider" => "XXXXXXXXXXXXXXXX",
      "BlobEventTime" => "03/05/2021 12:38:24",
      "BlobQueueMessageId" => "XXXXXXXXXXXXXXXXXXXXX",
      "CorrelationId" => "XXXXXXXXXXXXXXXXXXXX",
      "SourceContext" => "XXXXXXXXXXXXXXXXXXXXXXX"
      },
      "level" => "Information",
      "message" => "XXXXXXXXXXXXXXXXXX",
      "@timestamp" => 2021-03-05T04:38:27.228Z,
      "messageTemplate" => "XXXXXXXXXXXXXXXXXXXXXXXXXXX"
    }
2

You might be able to use

if ![EventId][Id] { mutate { rename => { "[EventId]" => "[EventId][Name]" } } }

or you may need to use ruby

ruby {
    code => '
        if event.get("EventId").is_a? String
            event.set("[@metadata][renameEventId]", true
        end
    '
}
if [@metadata][renameEventId] { mutate { rename => { "[EventId]" => "[EventId][Name]" } } }
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.