Output: specify document id *or* allow elasticsearch to pick

Elastic Stack Logstash
1

Sometimes on injest we have an id we want to keep as the document id:

    if [trace_id] {
      mutate {
        copy => { "trace_id" => "[@metadata][document_id]" }
        remove_field => "trace_id"
      }
    }

so we specify as an output filter (for example):

  if [type] == "bbq" {
    elasticsearch {
      id                 => "bbq"
      hosts              => ["elasticsearch-1", "elasticsearch-2", "elasticsearch-3"]
      index              => "bbq-%{+YYYY.MM.dd}"
      document_id        => "%{[@metadata][document_id]}"
    }
  }

And this works great, but there are certain logs generated where we do not have a trace_id and so want elastic to generate its own document ID. But if [@metadata][document_id] was not specified previously, we get a document in elastic with the literal text [@metadata][document_id] as its document ID.

Possible solutions to this:

  • generate a document ID ourselves if not already specified (I don't like this as elastic should be handling this)
  • have two separate output filters (I do not like the duplication of doing so)

Is there a better way of handling this? What is the best pattern?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.