Logstash and CSV - append to field while upsert

Elastic Stack Logstash
1

Hi there :grinning:

I'm using logstash. My input is a CSV file. I'm using one of the columns as the document_id. There are lines in the CSV file that have the same document_id.

In my output, I'm using upsert. When Logstash performs an update, I want it to append one of the fields instead of replacing the existing one.

This is my Logstash file:

input {
    	file {
    	    path => "/home/anthony/10.txt"
    	    start_position => "beginning"
    	    sincedb_path => "/dev/null"
    	}
    }
    filter {
    	csv {
    	    separator => "|"
    	    skip_header => "true"
    	    columns => ["type_identifiant_pp","identifiant_pp","identification_nationale_pp","code_civilite_exercice","libelle_civilite_exercice","libelle_commune"]

    	}
    	

    }


    output {
    	stdout {}
    	amazon_es {
    		hosts => ["XXXXXX.eu-west-3.es.amazonaws.com"]
    		region => "eu-west-3"
    		aws_access_key_id => 'XXXXX'
    		aws_secret_access_key => 'XXXXX'

    		index => "index-002"
    		document_id => "%{identification_nationale_pp}" 
    		doc_as_upsert => "true"
    	
    	
    		}
}

I tried to use a script like this one:

script => "ctx._source.libelle_commune += params.event.get('libelle_commune')"

And also like the ones described in these topics:

I've spent a lot of time on this. I read a lot of documentation, but I must admit I'm stuck here.

Could someone give me a hint? :bowing_man: Thanks!
Anthony

2

Hi,

I also tried a script like this one:

     script => 'if (ctx._source.libelle_commune == null) { ctx._source.libelle_commune = new ArrayList();} if(ctx._source.libelle_commune != null) {ctx._source.libelle_commune.add("%{[libelle_commune]}");}'

To no avail :unamused:

Has anyone an idea or something I could try?

3

It looks like my syntax is wrong and should be like that but this does not work:

    script_lang => "painless"
    script_type => "inline"
    script => 'if (!ctx._source.containsKey("libelle_commune")) ctx._source.libelle_commune = new ArrayList(); ctx._source.libelle_commune.add("test");'
4

I can upsert and use a script with PHP. This is some working code:

    	$params['body'] = [
                "script" => [
    				        'lang' => 'painless',
    				        'source' => 'if (!ctx._source.containsKey("libelle_commune")) ctx._source.libelle_commune = new ArrayList(); ctx._source.libelle_commune.add(params.libelle_commune);',
    				        'params' => [
    				            'libelle_commune' => $l
    							]
    						],
    	        "upsert" => [
                    'nom_exercice' => $row['nom_exercice'],
                    'prenom_exercice' => $row['prenom_exercice'],
                    'libelle_commune' => [$row['libelle_commune']]
    				]
    	];

But I can't get it working using logstash. Can anyone help me translate the query written in PHP into an output for logstash? :vulcan_salute:t2:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.