Send parts of a message to different Elasticsearch indices from a single Logstash instance

Elastic Stack Logstash
1

Suppose my message has fields A and B. I want to separately send each of the fields to the same ES instance, but to different indices. Is that possible?
E.g. if this is the message: {"A": some text, "B": some more text}, I want "some text" to end up in Index_1, and "some more text" in Index_2. Both indices are on the same host. I don't know if this is important, but the B field only gets added to the message in the filter section.

If only the elasticsearch output plugin had the ability to parse the message like other plugins do, I could just extract the desired field in each of the elasticsearch output blocks. However, that doesn't seem to be possible.

Of course, what I'm describing can be achieved using two Logstash instances, but can it be achieved using only one?

Note: I'm looking for a solution that can be put into a Logstash pipeline.conf file. :slight_smile:

2

You can use a clone filter to create two events that you can then format separately and send to different indices.

1 Like
3

Hello powerful_clouds

Have you tried _reindex with filter?

Something like

POST /_reindex
{
  "source": {
    "index": "SOURCE INDEX",
    "query": {
      "match": {
        "A": "some text"
      }
    }
  },
  "dest": {
    "index": "DESTINATION_INDEX"
  }
}


POST /_reindex
{
  "source": {
    "index": "SOURCE INDEX",
    "query": {
      "match": {
        "B": "some more text"
      }
    }
  },
  "dest": {
    "index": "DESTINATION_INDEX"
  }
}

Regrads
Krishna

1 Like
4

As @Christian_Dahlqvist already said, this can be done using the clone filter.

You would need something like this.

filter {
    clone {
        clones => ["index-field-b"]
    }
    if [type] == "index-field-b" {
        mutate {
            remove_field => ["A"]
        }
        mutate {
            ... logic that you use to add field B ...
        }
    }
}
output {
    if [type] == "index-field-b" {
        elasticsearch {
            ... output to the index  for the field B
        }
    } else {
        elasticsearch {
            ... output to the index for the field A
        }
    }
}

You just need to check if you have pipeline.ecs_compatibility enabled for your pipeline or not as this will change the behavior of the filter, you can check the documentation for more examples, but basically with pipeline.ecs_compatibility set to v1 or v8, you will need to use the tags field in the conditional.

Something like:

if "index-field-b" in [tags]
2 Likes
5

@Christian_Dahlqvist @leandrojmp Thank you very much for your help. :slight_smile: I've managed to get it working.
There's one thing I wanted to note: I noticed that the number of docs in each of the ES indices was double what I was expecting. I looked at the documents in the B index and saw that half of them had the desired field (e.g. a field of type string called desired_b_field), whereas half of them didn't. After some googling, I realized that the simplest solution is to add an and part to the if in the output section in order to check that the desired field is not null. E.g. for messages of type B:

if "index-field-b" in [tags] and ("" in [desired_b_field]) {
...
}

If you have any additional info on why that duplication happened, feel free to share it here. I'm sure people will find it useful. :slight_smile:

Also, @Krishna_Teja, thanks for your proposed solution. Although it's not what I was looking for, I'm sure others will find it useful.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.