# Multiple patterns regrouping in one line index
**URL:** https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073
**Category:** Logstash
**Created:** [May 8, 2018, 10:18pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073 "2018-05-08T22:18:04Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 8, 2018, 10:18pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/1 "2018-05-08T22:18:04Z")
\< 2018-02-07T10:42:02,831 [ExtractDwhData] [INFO] Solife :: Tools :: DWH :: ITK DWH - version : 2.6.0-SNAPSHOT - build #265 on 2018-01-04 08:22:32
2018-02-07T10:42:02,832 [ExtractDwhData] [INFO] Starting DWH Data Extraction with run timestamp : 2018-02-07 10:42:02
2018-02-07T12:24:45,167 [ExtractDwhData] [INFO] Solife DWH data EXTRACTION finished in 1 hours, 42 minutes, 42.368 seconds
hello everyone /\>
for those three lines i use this config file
\
input {
file {
type =\> "test1"
path =\> ["C:/Users/THINKPAD/Downloads/logstash-6.2.2/essai/."]
}
}
filter {
if [type] == "test1"{
grok {
match =\>["message", "%{TIMESTAMP\_ISO8601:timestamp}%{GREEDYDATA:message1}\s+Extraction\sbatch\sID\s:\s%{NUMBER:ID\_extraction\_globale}",
"message","%{TIMESTAMP\_ISO8601:start\_time\_extraction\_globale}%{GREEDYDATA:message2}\sStarting\sDWH\sData\sExtraction%{GREEDYDATA:message3}"
,"message","%{TIMESTAMP\_ISO8601:END\_TIME}%{GREEDYDATA:message4}\sSolife\sDWH\sdata\sEXTRACTION\sfinished\sin%{GREEDYDATA:temps\_totales}"]
}
mutate {
remove\_field =\> ["message1" ,"message2","message3","message4"]
}
if "\_grokparsefailure" in [tags] {
drop {}
}
}
}
output {
if [type] == "test1"{
elasticsearch { hosts =\> ["localhost:9200"]
index=\>"globalextraction"}
stdout {
codec =\> rubydebug
}
}
}
i want to get the result of those 3 patterns in elastic search in one line whish have
the informations
but the problem is i get each pattern in a single line and in the table of elasticearch i have 3 lines
someeone help me to add a command to regroup the result of parsing in one line 🙂
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 1:05am UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/2 "2018-05-09T01:05:52Z")
Please read what you posted, it is really hard to read. I do not understand why you have a "Extraction\sbatch\sID" when none of your messages match it.
It sounds an aggregate filter might be able to do this.
For the grok, personally I would prefer dissect, but that's not a big deal. Do a grok to split the timestamp off the message, then do a grok to ignore the "ExtractDwhData" and ignore the log level. Then you really do have unstructured data that grok might work for.
```auto
if "_grokparsefailure" in [tags] {
drop {}
}
```
This tends to be a bad idea. If your configuration fails to parse the data then you will usually be better off tagging it for review than dropping it.
The reason I prefer not to use grok is that a GREEDYDATA anywhere except at the end of the message, such as
```auto
{GREEDYDATA:message2}\sStarting\sDWH\sData\sExtraction%{GREEDYDATA:message3}
```
can get _really_ expensive. The regexp processor will have to step through the message one character at a time seeing if the rest of the message matches. This can lead to timeouts.
Also, removing temporary fields (message2 etc.) should be deferred until you know the patterns are working. And do not even try to index them into elasticsearch until you get good output when you run logstash on the command line with 'output { stdout { codec =\> rubydebug } }'.
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 7:44am UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/3 "2018-05-09T07:44:20Z")
2018-02-07T10:42:06,865 [ExtractDwhData] [INFO] Extraction batch ID : 28
2018-02-07T10:42:02,832 [ExtractDwhData] [INFO] Starting DWH Data Extraction with run timestamp : 2018-02-07 10:42:02
2018-02-07T12:24:45,167 [ExtractDwhData] [INFO] Solife DWH data EXTRACTION finished in 1 hours, 42 minutes, 42.368 seconds
$ those are the correct lines i made a mistake
thanks for the answer but i have a log in which there is 1600 lines
i put those three lines because
those where i have the informations
yes i want the result in one line in the table of the index
not threee lines in which i can had an empty colons in some lines
so how my code will be if i use the filter aggregate ? thanks a lot
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 2:06pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/4 "2018-05-09T14:06:38Z")
If there are ever multiple extractions occuring then this will not work, since there appears to be nothing in the log messages allowing you to correlate which is which.
This would allow you to combine those three lines.
```auto
filter {
dissect { mapping => ["message", '%{ts} [%{f1}] [%{loglevel}] %{text}' ] }
mutate { add_field => { "static" => "1" } }
if [text] =~ /^Extraction batch ID/ {
grok { match => ["text", "Extraction batch ID : %{NUMBER:ID_extraction_globale}"] }
aggregate {
task_id => "%{static}"
code => "map['id'] = event.get('ID_extraction_globale')"
}
drop {}
}
if [text] =~ /^Starting DWH Data Extraction with run timestamp/ {
grok { match => ["text", "Starting DWH Data Extraction with run timestamp : %{TIMESTAMP_ISO8601:runtimestamp}"] }
aggregate {
task_id => "%{static}"
code => "map['runtimestamp'] = event.get('runtimestamp')"
}
drop {}
}
if [text] =~ /^Solife DWH data EXTRACTION finished/ {
mutate { gsub => ["text", "Solife DWH data EXTRACTION finished in ", "", "text", " hours, ", ":", "text", " minutes, ", ":", "text", " seconds", ""] }
mutate { rename => { "text" => "duration" } }
aggregate {
task_id => "%{static}"
code => "event.set('ID_extraction_globale', map['id'])
event.set('runtimestamp', map['runtimestamp'])"
map_action => "update"
}
}
date { match => ["ts" , "YYYY-MM-dd'T'HH:mm:ss,SSS"] }
}
```
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 2:42pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/5 "2018-05-09T14:42:58Z")
i had this error ☹
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 2:44pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/6 "2018-05-09T14:44:04Z")
this is the config file
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 2:53pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/7 "2018-05-09T14:53:13Z")
> [@Badger](#):
>
> code =\> "event.set('ID\_extraction\_globale', map['id'])
> event.set('runtimestamp', map['runtimestamp'])"
It appears to be joining the two lines. Add a semi-colon at the end of the first line.
```auto
code => "event.set('ID_extraction_globale', map['id']);
event.set('runtimestamp', map['runtimestamp'])"
```
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 3:01pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/8 "2018-05-09T15:01:21Z")
sorry for your time 😢
but the same error i don't know where is the error
and why there is a semicolon and "with" with pink color
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 3:01pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/9 "2018-05-09T15:01:40Z")
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 3:06pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/10 "2018-05-09T15:06:18Z")
i just want to ask you where is the fields ? i didn't found them in your patterns
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 4:09pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/11 "2018-05-09T16:09:44Z")
Look at line 36. You have joined the two lines into one without a semicolon to separate the two statements. That will get you unexpected tIDENTIFIER all day long.
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 5:15pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/12 "2018-05-09T17:15:11Z")
i added the semicolon and always the same wrong ; i think you didn't understand what i wantto parse 🙂
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 5:21pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/13 "2018-05-09T17:21:19Z")
Yeah. I used different names and formatted temp\_totales differently. The important thing is that it joins data from the three lines into one event. Which data does not really matter.
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 9, 2018, 8:04pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/14 "2018-05-09T20:04:13Z")
the first code that i made it work but the problem is i get the result in 3 lines the GREEDYDATA where i put "message 1 or 2 or.. " i don't need those informations , every information i need i give it a significant name like (id\_extraction\_globale,END\_time etc ..)
could you modify it by adding the command "aggregate" in the correct places to put the result in one line in the index without complicating it ? thank you so much badger i m really blocked in this step in my intership..
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 9, 2018, 8:26pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/15 "2018-05-09T20:26:01Z")
```auto
filter {
dissect { mapping => ["message", '%{[@metadata][ts]} [%{}] [%{}] %{[@metadata][text]}' ] }
mutate { add_field => { "[@metadata][static]" => "1" } }
if [@metadata][text] =~ /^Extraction batch ID/ {
grok { match => ["[@metadata][text]", "Extraction batch ID : %{NUMBER:ID_extraction_globale}" ] }
aggregate {
task_id => "%{[@metadata][static]}"
code => "map['id'] = event.get('ID_extraction_globale');
map['timestamp'] = event.get('[@metadata][ts]');"
}
drop {}
}
if [@metadata][text] =~ /^Starting DWH Data Extraction with run timestamp/ {
aggregate {
task_id => "%{[@metadata][static]}"
code => "map['start_time_extraction_globale'] = event.get('[@metadata][ts]');"
}
drop {}
}
if [@metadata][text] =~ /^Solife DWH data EXTRACTION finished/ {
grok { match => ["[@metadata][text]", "Solife\sDWH\sdata\sEXTRACTION\sfinished\sin\s%{GREEDYDATA:temps_totales}" ] }
aggregate {
task_id => "%{[@metadata][static]}"
code => "event.set('ID_extraction_globale', map['id']);
event.set('timestamp', map['timestamp']);
event.set('start_time_extraction_globale', map['start_time_extraction_globale']);"
map_action => "update"
}
}
}
```
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 11, 2018, 2:17pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/17 "2018-05-11T14:17:53Z")
hello it's works but the problem is a have a log in which i have 2000 lines
when i make as un put all the log it parse all the lines
which command should i add to parse only the 3 lines i want to
(like grok parse failure ) thank you for your time
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 11, 2018, 2:30pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/18 "2018-05-11T14:30:55Z")
If you want to throw away the entire file apart from the fields extracted from those 3 lines then change the end of the filter from
```auto
}
}
```
to be
```auto
} else {
drop {}
}
}
```
---
### Author:  [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)
#### Post date: [May 11, 2018, 3:29pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/19 "2018-05-11T15:29:41Z")
filter {
grok {
match =\>["message","%{TIMESTAMP\_ISO8601:Start\_time\_conversion}%{GREEDYDATA:message1}\s+Starting\sstep\s2%{GREEDYDATA:message2}",
"message","%{TIMESTAMP\_ISO8601:ENd\_Time\_Conversion}%{GREEDYDATA:message3}\s+End\sof\sXML\sto\sCSV\sconversion%{GREEDYDATA:message4}%{NUMBER:total\_number}%{GREEDYDATA:message5}\s+were%{GREEDYDATA:status}\s+processed\sin\s%{GREEDYDATA:duration}"
]
}
and those are the lines
2018-02-07T12:24:18,215 [ExtractDwhData] [INFO] Starting step 2 : XML transformation to CSV...
2018-02-07T12:24:36,071 [XmlToCsvConverter] [INFO] End of XML to CSV conversion. 31 XML files were successfully processed in 17.828 seconds
i need only
start\_time\_conversion
end\_time\_conversion
duration
status
how to get those fields in one index like you did in the first time
thank you i am really appriciated
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 8, 2018, 3:29pm UTC](https://discuss.elastic.co/t/multiple-patterns-regrouping-in-one-line-index/131073/20 "2018-06-08T15:29:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.