Run script from Logstash then fill new field with result

Elastic Stack Logstash
1

Hello !

I'm completely new to the ELK stack, so sorry if my question looks quite stupid :slight_smile:

I'm trying to create a new field for Elasticsearch with Logstach by running a bash command.

Explanations :

I have, for exemple, a Logstash field like "host" (for exemple, one value could be Host1).
What i want to do, is to run an external command (ex : echo %{host}TEST) and add this result in Elasticsearch in order to put the command's result in a new field.
So it will be like that :

Host1 -> -> Host1TEST

And then print "Host1TEST" in a new field.

I absolutely don't know how to do it, i spend my whole day researching for that but didn't find any informations, even in this forum...

The only result i can provide is to write in logstash.stdout by putting this in my output.conf file :

exec {
command => "echo %{host}TEST"
}

And i have "Host1TEST" in the stdout log. I want the same result in a new field in Elasticsearch, but the output plugins doesn't accept exec ...

Can you help me ?

Thanks a lot !

2

Why do you need to run a shell script? In what way do you want to process the event? There might be a way of doing what you want without running a shell script so let's focus on the underlying problem.

3

Hi,

Actually i need to run a bash command with my "host" value, not necessary a script, it was only for the exemple.
So how can i run, for exemple, an "echo %{hostvalue}_TEST" and then add it in a new field ?

4

Please answer both questions I asked.

5

I need to run this command because i want an additionnal information that is not displayed in the source logs, in order to have it I have to run this external command with one information that, this one, is in the logs.

So the process is : I receive logs, parse it, run the command with the value of one field, then send the result's command in Elasticsearch in a new field. Final result should be displayed in Kibana with all source logs fields plus the command's result.

I already tried to put this information in the source logs, but that's not possible.

I hope i'm clear enough !

Thanks a lot for your help.

6

"How do I run a program" isn't a problem, it's one solution to a problem. Focus on the underlying problem, i.e. "how do I transform value x into y".

So. Why is running a program the only way to look up the information to produce the additional field? What kind of transform does the program perform?

7

Ok i'm parsing Nagios logs. My Nagios servers are linked to Centreon.
In Centreon there are hostgroups which represents support group.
This information is not available in Nagios, only in Centreon. In order to find the good support group of a host, i have to run a CLapi command (Centreon API) by mentioning the hostname.
So there are not other way to find the good support group of a host except by running a Clapi command with the hostname. I also can find it in a SQL request, but that's exactly the same problem, i have to run an external command for each entries.

8

If you can make the transform with a SQL query you're in luck; the new jdbc_streaming filter does just that. It's like the old translate filter but for JDBC. The translate filter could also be useful if you're able to periodically dump the whole table of mappings between hosts and support groups.

9

Tried to install jdbc_streaming, but always get an error while downloading / installing this plugin ...
I'm quite surprised that Logstash can't run any external command and then use the value, but anyway, thanks for your help.

10

Tried to install jdbc_streaming, but always get an error while downloading / installing this plugin ...

Without details we can't help out.

I'm quite surprised that Logstash can't run any external command and then use the value, but anyway, thanks for your help.

For many of the use cases Logstash was built for running a command for each event would be far too inefficient. That doesn't mean that it's not a valid use case but it seems people typically have found other means of reaching their goals.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.