How to Code a Try/Catch in Logstash Config File?

[redapplesonly (RAO)]

Hi Logstash Iron Chefs,

I am running Docker Container Logstash 7.4.0 (I know, I need to get the latest version) and my Logstash filter has a lot of code blocks like these:

  if [MyData][dataA][dataB] == 0 {
    translate {
      field => “[MyData][dataA][dataC]”
      destination => “[MyData][dataA][newData]”
      dictionary_path => “/usr/share/logstash/config/dict1.yaml”
      fallback => "default_0"
    }
  }
  else if [MyData][dataA][dataB] == 1 {
    translate {
      field => “[MyData][dataA][dataC]”
      destination => “[MyData][dataA][newData]”
      dictionary_path => "/usr/share/logstash/config/dict2.yaml"
      fallback => "default_1"
    }
  }

And when I run my traffic through Logstash, for the most part, this config works great.

…however…

What I’ve noticed is that on rare occasions, my Logstash container crashes. I can’t tell why, but I’ve noticed that the container always crashes whenever I misconfigure my LS config file and then run test data. So I’m guessing that I have a bug in my filter code, a bug which impacts only 0.0001% of data. That’s hard to pinpoint, for I have no idea what the problematic data might look like.

So – this is a wild guess – but I’m guessing that the problem data runs through the filter code, the code doesn’t apply to the data, and that’s why Logstash crashes. If so, no worries, I’m fine with a LG config that simply ignores the problem data.

The question is, how to do this?

If this were a Java program, I would wrap the above code block in a try/catch statement:

try{
    if [MyData][dataA][dataB] == 0 {
      ...entire code block from above...
    }
} catch exception >whatever< {
  // do nothing...
  // or maybe log this...
}

Another possibility is to add an “else” clause to what I have thus far:

  if [MyData][dataA][dataB] == 0 {
    ...same...
  }
  else if [MyData][dataA][dataB] == 1 {
    ...same...
  }
  else {
    // do nothing...
    // or maybe log this...
  }

The second option is clunky, but will work in a pinch, I guess. Does anyone have any suggestions? Thanks

[kares (Karol Bucek)]

Hello RAO,

this sounds weird, could you please share any details of the crash (logs) ?
The filter plugin in particular should be resilient against failures.

Your Java try - catch suggestion are pretty much how the plugin is implemented.
In case of an error you should see Something went wrong when attempting to translate from dictionary in your logs (with trace details about the exception).

[redapplesonly (RAO)]

Hi Karol,

Thanks for giving my situation a little thought. My crashes are highly infrequent, and extremely difficult to reproduce. I'll have to hope that there's an existing log file that has already captured a crash.

That said, I haven't explicitly set up a log file in my filter config. Would the system be logging these kind of errors in some default log? If so, where would I find it? If not, can you recommend how I could enable logging that might capture the dictionary translation failure?

Many, many thanks!

[redapplesonly (RAO)]

Hi Karol,

I'm sorry, I realized I should have specified... I am running Logstash 7.4.0 as a Docker container. When I say "the server crashed," what I mean is I see the container no longer running. I omitted that because sometimes when you post, its easy to include a lot of non-relevant information, and I try to keep my posts streamlined.

But thinking through your comment, I realized that using a Docker container might be relevant here. Not that Docker will be able to tell me why the dictionary lookup failed, but might retain some general crash information. Just thought I'd mention it...

Thanks!

[kares (Karol Bucek)]

Unless you setup a custom configuration (using log4j.properties), logs are redirected to std-out.
Which is kept for you by Docker while the container instance is around.

If you do not want to configure that much simply try extracting the container's stdout to a file as you run the container: docker logs -f <LS_CONTAINER_ID> &> /var/log/logstash-docker.log

[redapplesonly (RAO)]

Thanks Karol,

Hmm. I did some digging, and I couldn't resuscitate the old logs which might have captured my crash information. I think its prudent to close this thread (with my compliments!) and then do a little research on capturing a Logstash container's syslog from the host. Then, if the issue happens again, I can repost this question with a little more evidence.

But you've really helped me. I assumed there was a glaring problem with my code. Now that I know that my symptoms are unusual, I can be better prepared.

Many thanks!