# Aggregate problem
**URL:** https://discuss.elastic.co/t/aggregate-problem/142572
**Category:** Logstash
**Created:** [August 1, 2018, 12:44pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572 "2018-08-01T12:44:18Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 1, 2018, 12:44pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/1 "2018-08-01T12:44:19Z")
Hello !
My logstash config can deal with serveral log type. Two of them are "Appel" and "SVI". These two types have a similar field called "ID appelSVI". I want to add à field called "Passage par SVI" to my "Appel" event if there is a "SVI" event with the same "ID appelSVI". There is my code :
> if [Type]=="Appel" or [Type]=="SVI"
> {
> aggregate
> {
> task\_id =\> "%{ID appelSVI}"
> code =\> "
> if(event.get('Type')=='SVI')
> map['Passage par SVI']='true'
> event.cancel
> else
> if(map['Passage par SVI']=='true')
> event.set('Passage par SVI'='true')
> end
> end
> "
> push\_map\_as\_event\_on\_timeout =\> false
> timeout\_task\_id\_field =\> "ID appelSVI"
> timeout =\> 210
> timeout\_code =\> ""
> timeout\_tags =\> ['\_aggregatetimeout\_aggregationSVI']
> }
> }
It works if the first event that come in the aggregate part is a SVI type. If it is, map['Passage par SVI'] is set to 'true'. Then when the event of type Appel come it will check if map['Passage par SVI'] is set to 'true' and will add the field 'Passage par SVI'.
The problem is when the first event that come in the aggregate part is a Appel Type. I can't know if a SVI type with the same ID will come so I can't add the field 'Passage par SVI' yet and if a event of Type SVI come it will be to late for adding the field to the Appel type event. How can I do that ?
Thanks
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 1:17pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/2 "2018-08-01T13:17:15Z")
As you say, logstash cannot predict what events it will see in the future. Can you sort your input before feeding it to logstash?
Otherwise, it would depend on what your output is. For example, if you are writing to elasticsearch then in principle you could update the document in elasticsearch when you see an SVI.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 1, 2018, 1:31pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/3 "2018-08-01T13:31:37Z")
I can't sort my input before feeding it to Logstash. How can I update the document in Elasticsearch when I see an SVI ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 1:41pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/4 "2018-08-01T13:41:39Z")
You would use logstash to generate a file that could be POSTed to elasticsearch using the [bulk](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) and [update](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html) APIs.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 2, 2018, 9:40am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/5 "2018-08-02T09:40:42Z")
Ok if I understand, Logstash will not transmit data to Elasticsearch but will create a file. And then thanks to the bulk API Elasticsearch will read the file and index all document in this file. If I'm right what will do the update API ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 2, 2018, 1:35pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/6 "2018-08-02T13:35:04Z")
The update API determines the format of that file. You can merge in new fields using update, as shown in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#_updates_with_a_partial_document). Let me see if I can find an example...
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 2, 2018, 6:33pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/7 "2018-08-02T18:33:06Z")
OK, the use case where I did this was parsing SiteMinder trace logs, where every line has a correlation id and one piece of information about the request. I needed to gather all the information about one request into a single document. I did this by doing a bulk update using doc\_as\_upsert. One update for each input line.
So, provided that you can use the 'ID appelSVI' as the document id, what you could do is something like
```
output {
if [Type] == "SVI" {
file { path => "/some/path/out.txt" codec => plain { format => '{ "update" : {"_id" : "%{ID appelSVI}", "_type" : "doc", "_index" : "someindex"} }
{ "doc": "Passage par SVI": true, "doc_as_upsert" : true }
' } }
}
}
```
Then
```
curl -X POST 'localhost:9200/someindex/doc/_bulk' -H "Content-Type:application/json" --data-binary @/some/path/out.txt
```
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 3, 2018, 2:12pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/8 "2018-08-03T14:12:44Z")
Ok I understand. The first code block works well but I don't realy know where I should write the second.
There is the output in Logstash for ligne where the "Type" is not "SVI" :
> elasticsearch{
> hosts =\> "localhost"
> index=\> "cdr\_sbc"
> document\_type=\>"CDR\_SBC"
> }
My line should look like
> curl -X POST 'localhost:9200/cdr\_sbc/CDR\_SBC/\_bulk' -H "Content-Type:application/json" --data-binary @C:\Users\GAUTSCPI\Documents\Elasticsearch\sortieSVI.txt
But where should I write it ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 3, 2018, 2:14pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/9 "2018-08-03T14:14:41Z")
After logstash has executed the file output will have been created. Then you run the curl command at a shell prompt.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 3, 2018, 2:18pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/10 "2018-08-03T14:18:51Z")
Ok, the probleme is that I work on Windows and the curl command is unknow.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 3, 2018, 2:24pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/11 "2018-08-03T14:24:20Z")
OK, you could do it in PowerShell using Invoke-WebRequest.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 6, 2018, 8:47am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/12 "2018-08-06T08:47:35Z")
Thank you from your answer. I searched on the internet but I can't find how to make this instruction on the Invoke-WebRequest format. There is what I tried :
> Invoke-WebRequest -Method POST -URI 'localhost:9200/cdr\_sbc/CDR\_SBC/\_bulk' -body "Content-Type:application/json" --data-binary @C:\Users\GAUTSCPI\Documents\Elasticsearch\sortieSVI.txt
I obtain this error :
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 6, 2018, 11:29am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/13 "2018-08-06T11:29:24Z")
Try
```
get-content C:\Users\GAUTSCPI\Documents\Elasticsearch\sortieSVI.txt | Invoke-WebRequest -Method POST -URI 'localhost:9200/cdr_sbc/CDR_SBC/_bulk' -Content-Type "application/json"
```
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 6, 2018, 11:43am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/14 "2018-08-06T11:43:42Z")
I think it's better but powershell don't know the
> -Content-Type"application/json" parameter
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 6, 2018, 11:46am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/15 "2018-08-06T11:46:37Z")
Sorry, there is no hyphen in -ContentType
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 6, 2018, 11:52am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/16 "2018-08-06T11:52:37Z")
Yes it's better but now it is the same problem with the URI param, I tried to write it "Uri" like in the Invoke-Webrequest doc but it's the same. I tried to put back slashes instead of slashes in the path but the result is the same.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 6, 2018, 12:24pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/17 "2018-08-06T12:24:15Z")
I founded the solution I added "http://" at the beginning of the URI. Now I get a new error.
There is my json file :
I tried to write \n instead of return to line and the result is the same.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 6, 2018, 12:56pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/18 "2018-08-06T12:56:04Z")
This is really turning into a PowerShell question rather than a logstash question and I am not able to test it ☹ With curl, the reason you use -data-binary rather than -d is to tell it not to strip the newlines from the file. I do not know what the equivalent is in PowerShell.
The JSON looks OK, except you should not have the blank lines.
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 6, 2018, 1:11pm UTC](https://discuss.elastic.co/t/aggregate-problem/142572/19 "2018-08-06T13:11:46Z")
Ok I will continue to try to find a solution and will post it if I find, thank you !
---
### Author:  [@Pierre2](https://discuss.elastic.co/u/Pierre2)
#### Post date: [August 7, 2018, 7:25am UTC](https://discuss.elastic.co/t/aggregate-problem/142572/20 "2018-08-07T07:25:19Z")
Hello !
I founded a solution. The get-content "function" got a parameter that is called -Delimiter. By default this value is the return tu line char (\n). That means that get-content will return each line separately. I read in the get-content doc that if the delimiter that the user set does not exist in the file, get-content will return the entire file as a single undelimited object and that is what we want. I set the delimiter to "?!@" to be certain that nothing will match with this string and it works.
[Next page](https://discuss.elastic.co/t/aggregate-problem/142572.md?page=2)