Combine fields in Elasticsearch from logs with same ID

Elastic Stack Logstash
1

I am working on a log aggregation solution for my company's web requests, using Logstash and Elasticsearch. There are several logs produced for each web request containing slightly different information, and they would like all of the logs tracked. If every web request is identified with a unique ID upon the point of entry, is there a way to 'join' (yes I know there's no SQL equivalent join in Elasticsearch) all of the logs to do with one web request when forwarding to Elasticsearch? For example, if there was a log with id 123 and fields 'a' and 'b', and then another log with id 123 but fields 'a' and 'c' came in, I would want a single document with fields 'a', 'b' and 'c' stored in Elasticsearch.

2

Welcome to our community! :smiley:

Take a look at https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html

3

Thanks, this will work to show me the information I want, but not to get around the issue of storing duplicate logs.

4

Hi there,

will the values of the field 'a' be the same in both documents? In this case you can simply use the elasticsearch output with that ID field as document_id, update as action and the upsert enabled.

That way the document with that ID will be overwritten and updated with the new fields every time and you do not duplicate the documents.

Something like:

output {
  elasticsearch {
    hosts => ...
    index => ...
    document_id => "%{your_id_field}"
    action => 'update'
    doc_as_upsert => true
  }
}
5

Yes it will, but I was under the impression that update completely replaces the log? I also want to keep field 'b'.

6

Actually, having done some more research, this looks like it will work. I can't check now because the logs don't have unique IDs assigned yet, but thanks for the help!

7

No problem. Update doesn't overwrite the document, it updates it, adding those fields which are not already written, and overwriting only the common ones.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.