Script params via Logstash, is it possible?

[crickes (Steve Crickett)]

Is it possible to send 'params' to a stored script in Elasticsearch from Logstash, or do I have to use a script defined inline or as a file in Logstash in order to define a 'params' block for the script?

[Luca_Belluccini]

Hello @crickes,

The Logstash Elasticsearch output offers an option named script_type:

• inline (default), meaning the script option must contain the script to be executed
• indexed, meaning the script has already been stored in Elasticsearch and you must specify its ID / name on the script option
• file, the name of the script stored in Elasticsearch configuration directory

Logstash will send the content of the event as event attribute inside the params.
This means the stored script can access the parameters using params.event.get('<name of the field>').

Those parameters work only when the action option is set to update!
Example:

elasticsearch {
  hosts => [ ... ]
  index => ...
  action => "update"
  script_type => "indexed"
  script => "script-id"
...
}

[crickes (Steve Crickett)]

The golden nugget of information here is:

Logstash will send the content of the event as event attribute inside the params .

I spent quite some time trying to find this in the documentation, and if it is in there, I couldn't find it.

Thanks for your help.

[Luca_Belluccini]

I've submitted https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/931 to clarify this point in the documentation.

[parosio]

Hello, a clarification could really help...

I've a similar situation, and couldn't understand how to pass the event/field from within logstash, to achive an effect like this:

  ## Upsert with update of array field
  POST test-logs-ces/doc/1242272-10.127.0.1/_update
  {
  "script" : {
          "source": """
          if (ctx._source.files != null) {
             if (! ctx._source.files.contains( params.filename )) {
               ctx._source.files.add( params.filename )
            }
          } else 
            ctx._source.files = [ params.filename ]""",
          "lang": "painless",
          "params" : {
              "filename" : "extract1.xlsx"
          }
      },
    "upsert" : {
      "MID" : "1242272",
      "field_1" : "Some value"
    }
  }

so, i tried to unroll the param within the script, but hit the [script] Too many dynamic script compilations within ... please use indexed, or scripts with parameters instead

So, if my logstash event has a field filename I suppose I could use something like:

output {
 if [filename] { 
  elasticsearch {   
    document_id => "%{doc_id}"
    action => "update"
    doc_as_upsert => true
    script_lang => "painless"
    script => " fn=params.event.get('filename'); if (ctx._source.filenames != null) {ctx._source.filenames.add( fn )} else {ctx._source.filenames = [ fn ]} "
    script_type => "inline"
     . . . 
  }
 }
...

to have the new filename appended to filenames array.

Do I have advantages with indexed or filescripts over an inline one?

[Luca_Belluccini]

@parosio

Do I have advantages with indexed or file scripts over an inline one?

I would avoid the file.

If you use inline, the script is being sent together with the request each time.
It will be cached after the first compilation, so there are no major performance improvements.
The script will be sent each time.

If you use indexed, the script must be setup before starting Logstash and there will be less data to be transferred over network.

@Alex_Marquardt has a good tutorial in his blog: Using Logstash and Elasticsearch scripted upserts to transform eCommerce purchasing data

If you enable doc_as_upsert you might miss the first filename.

I think the correct elasticsearch output should be:

 elasticsearch {
    index => "ecommerce_ls_transformed"
    document_id => "%{doc_id}"
    action => "update"
    scripted_upsert => true
    script_lang => "painless"
    script => "def fn = params.event.get('filename'); if (ctx._source.filenames != null) {ctx._source.filenames.add(fn)} else {ctx._source.filenames = [ fn ]} "
  }

Regarding the following error:

@parosio

[script] Too many dynamic script compilations within ... please use indexed, or scripts with parameters instead

If the painless script is valid, I wouldn't expect it to be recompiled several times (as it is cached at the first execution, if there is no dynamic content in it - meaning you access params.event and there is no templated text in the script).
Are you sure this didn't occur just because you did some attempts adjusting the script and the compilation failed too many times in a short time frame?

[parosio]

About the errors...

Are you sure this didn't occur just because you did some attempts adjusting the script and the compilation failed too many times in a short time frame?

They occurred before I finally found your example of how to use event values as parameters (and had the values "cabled" inside the script )

Hence my initial comment about a clarification in the documentation.
Without a working example it isn't really simple to come up with the solution; there are a few other conversation about this topic, here and on stackoverflow, but none with the (simple) answer.

About the scripted_upsert... I decided it would have been too difficult to put into the script all the other fields of my document.

[parosio]

And, about documentation again:

@Luca_Belluccini

If you use indexed , the script must be setup before starting Logstash and there will be less data to be transferred over network.

an example of an indexed script would be really useful an example within logstash documentation, (currently only:

with a reference to elastic one