# Aggregation Filter 2 logs same index **URL:** \ **Category:** Logstash\ **Created:** [August 3, 2020, 8:31pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624 "2020-08-03T20:31:16Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 8:31pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/1 "2020-08-03T20:31:16Z")
Dear Comminuty I need your help to fix my issue. i have 2 logs in same index that should be aggregate to view all information on the Data Table visualize ```auto input { udp { port => 1514 type => "alcatel" } } filter { if [type] == "alcatel" { grok { match => { "message" => ["%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:(%{POSINT:syslog_pid}))\s+\w+:\s+[\w+](?\s+Authentication\s+failed):\s+\w+ %{DATA:job_id},", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:(%{POSINT:syslog_pid}))\s+\w+:\s+[\w+](?\s+Session)%{DATA:job_id},\s+\w+\s+\w+\s+%{SYSLOGHOST:srcip}", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:(%{POSINT:syslog_pid}))\s+\w+:\s+[\w+](?\s+Authentication\s+succeed):\s+\w+ %{DATA:job_id}," ] } add_field => ["received_at", "%{@timestamp}"] add_tag => "Alcatel" } if "_grokparsefailure" in [tags] { drop {} } aggregate { task_id => "%{job_id}" code => " map['syslog_timestamp'] ||= event.get('syslog_timestamp') map['syslog_hostname'] ||= event.get('syslog_hostname') map['syslog_program'] ||= event.get('syslog_program') map['syslog_pid'] ||= event.get('syslog_pid') map['status'] ||= event.get('status') map['srcip'] ||= event.get('srcip') map['job_id'] ||= event.get('job_id') " timeout => 2 timeout_tags => ['aggregated'] map_action => 'create_or_update' push_map_as_event_on_timeout => true } mutate { gsub => ["status", "^ ", ""] } } } ``` I would like to generate only one logs based on job\_id, because in first logs miss the src ip address and the second logs miss the authentication response. For example I received these log: Log 1 \<78\>Aug 03 19:04:41 mi-sg12-sw-01 SESSION(71) Data: [SES] Authentication failed: Session 13, access type Telnet Log 2 Aug 03 22:25:19 mi-sg12-sw-01 SESSION(71) Data: [SES] Session 13, IP address 10.3.1.4 Log finally: Aug 03 19:04:41 mi-sg12-sw-01 Authentication failed 13 IP Address 10.3.1.4 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 3, 2020, 8:32pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/2 "2020-08-03T20:32:28Z")
What is your question? ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 8:39pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/3 "2020-08-03T20:39:11Z")
I updated the first post. Thanks. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 3, 2020, 9:09pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/4 "2020-08-03T21:09:11Z")
logstash will not even start with those grok patterns... ```auto :exception=>#(?:\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|[Mm](?:a|ä)?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|[Oo](?:c|k)?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?!<[0-9])(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9]))) (?(?:(?:(?:(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?.*?)(?:\((?\b(?:[1-9][0-9]*)\b)\))\s+\w+:\s+[\w+](?\s+Authentication\s+failed):\s+\w+ (?.*?),/m>, ``` You will need to fix those before you worry about the aggregate. Try ``` "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Authentication\s+failed:\s+\w+ %{DATA:job_id},", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Session%{DATA:job_id},\s+\w+\s+\w+\s+%{SYSLOGHOST:srcip}", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Authentication\s+succeed:\s+\w+ %{DATA:job_id}," ] ``` ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 9:21pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/5 "2020-08-03T21:21:12Z")
Unfortunately not worked the logs are not aggregated. ![Schermata 2020-08-03 alle 23.18.42](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de4b287a7ec45033198f94da14b8c9b07b336254.png) and now miss the field status with the authentication response ![Schermata 2020-08-03 alle 23.20.08](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b56f49430dfe5178a5719fa210694f1fc0892873.png) the full configuration now is: ```auto filter { if [type] == "alcatel" { grok { match => { "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Authentication\s+failed:\s+\w+ %{DATA:job_id},", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Session%{DATA:job_id},\s+\w+\s+\w+\s+%{SYSLOGHOST:srcip}", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}\(%{POSINT:syslog_pid}\)\s+\w+:\s+\[\w+\]\s+Authentication\s+succeed:\s+\w+ %{DATA:job_id}," ] } add_field => ["received_at", "%{@timestamp}"] add_tag => "Alcatel" } mutate { gsub => ["status", "^ ", ""] } aggregate { task_id => "%{job_id}" code => " map['syslog_timestamp'] ||= event.get('syslog_timestamp') map['syslog_hostname'] ||= event.get('syslog_hostname') map['syslog_program'] ||= event.get('syslog_program') map['syslog_pid'] ||= event.get('syslog_pid') map['status'] ||= event.get('status') map['srcip'] ||= event.get('srcip') map['job_id'] ||= event.get('job_id') " timeout => 2 timeout_tags => ['aggregated'] map_action => 'create_or_update' push_map_as_event_on_timeout => true } if "_grokparsefailure" in [tags] { drop {} } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 3, 2020, 9:40pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/6 "2020-08-03T21:40:28Z")
I do not know what to say. With the grok changes I suggested and your original configuration I get the two parsed log messages and a couple of seconds later I get ``` { "syslog_timestamp" => "Aug 03 22:25:19", "syslog_pid" => "71", "status" => nil, "@version" => "1", "tags" => [ [0] "aggregated" ], "srcip" => "10.3.1.4", "syslog_hostname" => "mi-sg12-sw-01", "syslog_program" => "SESSION", "job_id" => " 13", "@timestamp" => 2020-08-03T21:39:32.389Z } ``` ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 9:47pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/7 "2020-08-03T21:47:15Z")
Could you give me the command that you use to generate it? BTW also in your logs miss the field with "authentication failed", please show the below logs. Log 1 \<78\>Aug 03 19:04:41 mi-sg12-sw-01 SESSION(71) Data: [SES] **Authentication failed** : Session 13, access type Telnet Log 2 Aug 03 22:25:19 mi-sg12-sw-01 SESSION(71) Data: [SES] Session 13, IP address 10.3.1.4 Log finally: Aug 03 19:04:41 mi-sg12-sw-01 **Authentication failed** 13 IP Address 10.3.1.4 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 3, 2020, 10:03pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/8 "2020-08-03T22:03:26Z")
That is probably because you never grok a field called [status] ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 10:04pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/9 "2020-08-03T22:04:58Z")
Show the first post, to generate the status field use **"(?\\s+Authentication\s+failed)"** BTW could you send me the command that you use to generate your output? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 3, 2020, 10:14pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/10 "2020-08-03T22:14:21Z")
Except that does not appear in your first post. The data I posted was generated by ``` output { stdout { codec => rubydebug } } ``` ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 3, 2020, 10:32pm UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/11 "2020-08-03T22:32:38Z")
> [@Badger](#): > > `output { stdout { codec => rubydebug } }` My output is: ```auto "syslog_pid" => "71", "type" => "alcatel", "syslog_program" => "SESSION", "status" => "Authentication succeed", "@version" => "1", "host" => "10.12.100.1", "message" => "<78>Aug 04 02:38:03 mi-sg12-sw-01 SESSION(71) Data: [SES] Authentication succeed: Session 11, access type Telnet\n\u0000", "syslog_timestamp" => "Aug 04 02:38:03", "@timestamp" => 2020-08-04T00:38:03.106Z, "syslog_hostname" => "mi-sg12-sw-01", "job_id" => "11", "received_at" => "2020-08-04T00:38:03.106Z" } { "syslog_pid" => "71", "type" => "alcatel", "syslog_program" => "SESSION", "@version" => "1", "host" => "10.12.100.1", "message" => "<78>Aug 04 02:38:03 mi-sg12-sw-01 SESSION(71) Data: [SES] Session 11, IP address 10.3.1.4\n\u0000", "syslog_timestamp" => "Aug 04 02:38:03", "srcip" => "10.3.1.4", "@timestamp" => 2020-08-04T00:38:03.107Z, "syslog_hostname" => "mi-sg12-sw-01", "job_id" => "11", "received_at" => "2020-08-04T00:38:03.107Z" } { "tags" => [ [0] "Alcatel" ], "syslog_pid" => "71", "syslog_program" => "SESSION", "status" => "Authentication succeed", "host" => "10.12.100.1", "@version" => "1", "syslog_timestamp" => "Aug 04 02:38:03", "srcip" => "10.3.1.4", "@timestamp" => 2020-08-04T00:38:09.878Z, "syslog_hostname" => "mi-sg12-sw-01", "job_id" => "11" ``` I don't understand because each time that restart logstash the output change: ```auto [0] "Alcatel" ], "syslog_pid" => "71", "syslog_program" => "SESSION", "status" => "Authentication succeed", "host" => "10.12.100.1", "@version" => "1", "syslog_timestamp" => "Aug 04 02:38:03", "srcip" => "10.3.1.4", "@timestamp" => 2020-08-04T00:38:09.878Z, "syslog_hostname" => "mi-sg12-sw-01", "job_id" => "11" ``` Reboot 1 ```auto "srcip" => "10.3.1.4", "status" => "Authentication failed", "@version" => "1", "syslog_hostname" => "mi-sg12-sw-01", "job_id" => "12", "@timestamp" => 2020-08-04T00:39:51.135Z, "syslog_pid" => "71", "syslog_program" => "SESSION", "tags" => [ [0] "Alcatel" ], "syslog_timestamp" => "Aug 04 02:39:44", "host" => "10.12.100.1" ``` Reboot 2: ```auto "syslog_hostname" => "mi-sg12-sw-01", "tags" => [ [0] "Alcatel" ], "@version" => "1", "@timestamp" => 2020-08-04T00:42:27.724Z, "syslog_timestamp" => "Aug 04 02:42:21", "syslog_pid" => "71", "srcip" => "10.3.1.4", "job_id" => "13", "status" => "Authentication succeed", "host" => "10.12.100.1", "syslog_program" => "SESSION" ``` Reboot 3 ```auto "host" => "10.12.100.1", "syslog_program" => "SESSION", "syslog_pid" => "71", "srcip" => "10.3.1.4", "syslog_timestamp" => "Aug 04 02:43:34", "job_id" => "14", "status" => "Authentication succeed", "tags" => [ [0] "Alcatel" ], "@timestamp" => 2020-08-04T00:43:38.797Z, "syslog_hostname" => "mi-sg12-sw-01", "@version" => "1" ``` I think that is not normal, BTW from kibana under discovery i saw only the specific log not the aggregate. ---
**Author:** ![rcavezzale](https://avatars.discourse-cdn.com/v4/letter/r/dec6dc/32.png) [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Post date:** [August 4, 2020, 1:38am UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/12 "2020-08-04T01:38:18Z")
Fixed the issue, below the correct configuration, i hope that could be useful for other member of the community. ```auto input { udp { port => 1514 type => "alcatel" } } filter { if [type] == "alcatel" { grok { match => { "message" => ["%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\(%{POSINT:syslog_pid}\))\s+\w+:\s+\[\w+\](?\\s+Authentication\s+failed):\s+\w+ %{DATA:job_id},", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\(%{POSINT:syslog_pid}\))\s+\w+:\s+\[\w+\](?\\s+Authentication\s+succeed):\s+\w+ %{DATA:job_id},", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\(%{POSINT:syslog_pid}\))\s+\w+:\s+\[\w+\]\s+\w+%{DATA:job_id},\s+\w+\s+\w+\s+%{SYSLOGHOST:srcip}" ] } add_field => ["received_at", "%{@timestamp}"] #add_tag => "Alcatel" } mutate { gsub => ["status", "^ ", ""] } mutate { gsub => ["job_id", "^ ", ""] } if "_grokparsefailure" in [tags] { drop {} } aggregate { task_id => "%{job_id}" code => " map['received_at'] ||= event.get('received_at') map['type'] ||= event.get('type') map['syslog_timestamp'] ||= event.get('syslog_timestamp') map['syslog_hostname'] ||= event.get('syslog_hostname') map['syslog_program'] ||= event.get('syslog_program') map['syslog_pid'] ||= event.get('syslog_pid') map['status'] ||= event.get('status') map['srcip'] ||= event.get('srcip') map['host'] ||= event.get('host') map['job_id'] ||= event.get('job_id') " timeout => 2 timeout_tags => ['Alcatel'] map_action => 'create_or_update' push_map_as_event_on_timeout => true } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 1, 2020, 1:38am UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624/13 "2020-09-01T01:38:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.