Logstash and CSV - append to field while upsert

[AnthonyA (Anthony)]

Hi there

I'm using logstash. My input is a CSV file. I'm using one of the columns as the document_id. There are lines in the CSV file that have the same document_id.

In my output, I'm using upsert. When Logstash performs an update, I want it to append one of the fields instead of replacing the existing one.

This is my Logstash file:

input {
    	file {
    	    path => "/home/anthony/10.txt"
    	    start_position => "beginning"
    	    sincedb_path => "/dev/null"
    	}
    }
    filter {
    	csv {
    	    separator => "|"
    	    skip_header => "true"
    	    columns => ["type_identifiant_pp","identifiant_pp","identification_nationale_pp","code_civilite_exercice","libelle_civilite_exercice","libelle_commune"]

    	}
    	

    }


    output {
    	stdout {}
    	amazon_es {
    		hosts => ["XXXXXX.eu-west-3.es.amazonaws.com"]
    		region => "eu-west-3"
    		aws_access_key_id => 'XXXXX'
    		aws_secret_access_key => 'XXXXX'

    		index => "index-002"
    		document_id => "%{identification_nationale_pp}" 
    		doc_as_upsert => "true"
    	
    	
    		}
}

I tried to use a script like this one:

script => "ctx._source.libelle_commune += params.event.get('libelle_commune')"

And also like the ones described in these topics:

Logstash Elasticsearch ouput: Append array while upserting

I am upserting documents: new fields should be added and existing fields should be replaced by the new documents field value. With one exception: the value of text field 'shodan.protocols' should always be appended instead of replaced. This field should contain multiple unique string values. My logstash output looks like this: elasticsearch { index => "test" document_id => "%{ip}" doc_as_upsert => true …

Logstash create doc if not exist else append array

Elasticsearch: 6.4.2 Logstash: 6.4.2 Goal: Index the data if it hasn't been indexed before, otherwise, if the document does exist, then append an array. Input data round 1: { "item": ["1"] } Input data round 2: { "item": ["2"] } Desired data in Elasticsearch: { "item": ["1","2"] } The below will index the first time the document is seen or append "item" if it has not. However, it will always append even if the value is in the array. e.g. if the below is ran three times in a row, the out…

Append to a property if exists

Hello, Input csv looks like : "","Code","Date","Time","Open","High","Low","Close","Volume" "1","3IINFOTECH",20150703,"09:16:00",5.55,4.55,4.55,4.55,835 "2","3IINFOTECH",20150703,"09:17:00",5.55,4.55,4.55,4.55,390 Logstash config is : input { file { path => "/Users/sbezgoan/Documents/Elastic/StockData/nse-company-stocks/three.csv" start_position => "beginning" sincedb_path => "/Users/sbezgoan/Documents/Tools/logstash-7.5.1/temp.log" } } filter { csv { separator => …

Add an array using Mutate's add_field

Hi Guys, I have a logstash pipeline where I am receiving a JSON file as HTTP input and forwarding it to output plugin. I want to introduce below structure to input JSON : "parentField": { "field0": "value0", "arrayName": [ { "field1": "value1", "field2": "value2" } ] } To achieve that I am trying to use below filter :: filter { mutate { add_field => { "[parentField][field0]" => "value0" } add_field => { "[parentField][arrayName][0][field1]" => "value1" } add_field => { "[parentF…

I've spent a lot of time on this. I read a lot of documentation, but I must admit I'm stuck here.

Could someone give me a hint? Thanks!
Anthony

[AnthonyA (Anthony)]

Hi,

I also tried a script like this one:

     script => 'if (ctx._source.libelle_commune == null) { ctx._source.libelle_commune = new ArrayList();} if(ctx._source.libelle_commune != null) {ctx._source.libelle_commune.add("%{[libelle_commune]}");}'

To no avail

Has anyone an idea or something I could try?

[AnthonyA (Anthony)]

It looks like my syntax is wrong and should be like that but this does not work:

    script_lang => "painless"
    script_type => "inline"
    script => 'if (!ctx._source.containsKey("libelle_commune")) ctx._source.libelle_commune = new ArrayList(); ctx._source.libelle_commune.add("test");'

[AnthonyA (Anthony)]

I can upsert and use a script with PHP. This is some working code:

    	$params['body'] = [
                "script" => [
    				        'lang' => 'painless',
    				        'source' => 'if (!ctx._source.containsKey("libelle_commune")) ctx._source.libelle_commune = new ArrayList(); ctx._source.libelle_commune.add(params.libelle_commune);',
    				        'params' => [
    				            'libelle_commune' => $l
    							]
    						],
    	        "upsert" => [
                    'nom_exercice' => $row['nom_exercice'],
                    'prenom_exercice' => $row['prenom_exercice'],
                    'libelle_commune' => [$row['libelle_commune']]
    				]
    	];

But I can't get it working using logstash. Can anyone help me translate the query written in PHP into an output for logstash?