Logstash Dead Letter Queue, send raw message to ElasticSearch

Elastic Stack Logstash
1

Hello, I am running into an issue where I am sending incorrectly formatted JSON to my logstash such that logstash throws errors about being unable to parse it. I learned about Dead Letter Queue, and now I am hoping someone can help me understand how I could send the entire bad JSON data as-is to elasticsearch. I was thinking I could use a single "message" key with a value set to the raw string of bad JSON.

My goal is to ensure I see EVERYTHING sent to logstash in Kibana even if they are malformed. The idea is that I would simply follow up with my app to correct the malformed data and in the mean time, I would not be missing messages when I am in the discover tab. I understand DLQ has the power to mutate the data so it is still usable, but as a stop-gap, I am very interested in getting the malformed data from Logstash into Elasticsearch and I am looking at DLQ to do it. Thanks!

2

If logstash is failing to parse the JSON you can do exactly that. If elasticsearch is failing to map the parsed JSON you need to fix or remove the fields it is complaining about.

3

Thanks, Badger, this will be the second time you have provided timely assistance to a question I have asked here, much appreciated.

If logstash is failing to parse the JSON you can do exactly that.

Great to see this!! :slight_smile:

If elasticsearch is failing to map the parsed JSON you need to fix or remove the fields it is complaining about.

My message will surely have a crazy double quote and curly brace situation that I fear will prevent parsing in Elasticsearch. Is there no way to send it with a key of "message" and a value of "DONT PARSE THIS MESS"?

Sorry, I guess I am looking to be spoon fed an answer for how to configure it here. I found Elasticsearch output plugin | Logstash Reference [8.11] | Elastic but am not sure how to define that the raw message be sent to elasticsearch via DLQ.

Here is what I have so far:

input {
  dead_letter_queue {
    path => "/tmp/dead_letter_queue"
    commit_offsets => true
    pipeline_id => "main"
  }
}

output {
	elasticsearch {
		hosts => "https://my-redacted-elasticsearch-host"
	}
}

But I feel stuck about how to achieve my stated goal in the actual configuration. Thanks!!

4

If events have been written to the DLQ then there is no point in sending them to elasticsearch until they have been modified, because it will fail to map them again. For example, if you have date field that cannot be parsed as a date then either delete it or rename it. If you have a text field that should be an object then make it an object.

5

So there is no way to send raw string data to elasticsearch after it fails to be parsed as valid JSON by logstash?

I can't use the DLQ to modify it so that it can be sent as a single key value of "message":"raw string that failed to parse"?

I don't want them to map, I want the raw malformed data to simply show up in my Discover tab.

6

Not sure if you are clear on the distinction between a parsing error and a mapping error.

When I say a parsing error I mean the json codec or filter fails to parse invalid JSON. In this case you can index message -- the event will not go to the DLQ.

When I says mapping error I mean the json is parsed into fields on the event, but elasticsearch objects to the contents of one of those fields. For example, if it thinks a field should be a date and it contains the text "the day after tomorrow" then elasticsearch will fail to index the event because it does not have a date parser that understands that text. In that case it gets sent to the DLQ if you have it enabled. Until that text is removed from the event it is not going to succesfully index.

If you do not know which fields are causing the problem then just strip off everything that can cause a problem.

    ruby {
        code => '
            event.to_hash.each { |k, v|
                unless [ "@timestamp", "@version", "host", "message" ].include?(k)
                    event.remove(k)
                end
            }
        '
        add_tag => [ "comeFixMe" ]
    }
7

Thanks for the explanation. I think I see now that elasticsearch is the one rejecting things and that I will in fact have to do some processing to modify the field causing the issue as it will never get accepted otherwise (as I believe you said earlier)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.