HELP WITH LOGSTASH PLUGIN

Elastic Stack Logstash
1

I am trying to analyze whatsapp chats on elastic, but my logstash config file seem to be the problem
this is my config file

input {

beats {
type => "whatsapp-alerts"
port => 5012
}
}

filter {
if [type]=="whatsapp-alerts"
{
grok{
break_on_match => false
named_captures_only => true
match => ["message" , "\s*%{NOTSPACE:logdate},\s*%{GREEDYDATA}-%{SPACE}\s*%{GREEDYDATA:PhoneNumber}:\s*%{GREEDYDATA:Message}"]
}
date {
match => [ "logdate", "ISO8601" , "M/d/yy" ]
target => "logdate"
}
mutate { add_field => { "indexType" => "WHATSAPP-LOG" }
}
}

if {"indexType" => "WHATSAPP-LOG"}
}

output {
elasticsearch {
hosts => "http://localhost:9200"
index => "whatsapp-logs-%{+YYYY.MM.dd}"
document_type => doc
user => "indent"
password => "changeme"
}

error message:
][INFO ][org.reflections.Reflections] Reflections took 205 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-23T11:23:32,058][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/filters/if>, :path=>"logstash/filters/if", :type=>"filter", :name=>"if"}
[2020-04-23T11:23:32,090][FATAL][logstash.runner ] The given configuration is invalid. Reason: Unable to configure plugins: (PluginLoadingError) Couldn't find any filter plugin named 'if'. Are you sure this is correct? Trying to load the if filter plugin resulted in this error: no such file to load -- logstash/filters/if
[2020-04-23T11:23:32,138][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

2

You have curly braces surrounding your second conditional.

3

Thank you so much for replying. please i am very new to this. i dont understand the terms yet. what second conditional please?

4

Look at the second if statement and compare to the first.

5

okay. i understand now, let me effect the changes and get back to you. thank you so much

6

Corrected it.

this is the error am getting

[elastic@elkD1 logstash-7.6.2]$ bin/logstash --config.test_and_exit -f /home/elastic/logstash-7.6.2/config/logstash-whatsapp.conf
Sending Logstash logs to /home/elastic/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-04-23T12:14:59,151][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-23T12:15:03,257][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "and", "or", "xor", "nand", "{" at line 25, column 32 (byte 645) after filter {
if [type]=="whatsapp-alerts"
{
grok{
break_on_match => false
named_captures_only => true
match => ["message" , "\s*%{NOTSPACE:logdate},\s*%{GREEDYDATA}-%{SPACE}\s*%{GREEDYDATA:PhoneNumber}:\s*%{GREEDYDATA:Message}"]
}
date {
match => [ "logdate", "ISO8601" , "M/d/yy" ]
target => "logdate"
}
mutate { add_field => { "indexType" => "WHATSAPP-LOG" }
}
}

if [indexType]== "WHATSAPP-LOG"
[2020-04-23T12:15:03,318][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

7

it is still not working, please help

8

@J-JENNY
Below is how your filter block needs to look like. You have to supply what you want to do in the second if conditional as Christian pointed out.

filter {

  if [type] == "whatsapp-alerts" {
    grok{
      break_on_match => false
      named_captures_only => true
      match => ["message" , "\s*%{NOTSPACE:logdate},\s*%{GREEDYDATA}-%{SPACE}\s*%{GREEDYDATA:PhoneNumber}:\s*%{GREEDYDATA:Message}"]
    }
    date {
      match => [ "logdate", "ISO8601" , "M/d/yy" ]
      target => "logdate"
    }
    mutate {
      add_field => { "indexType" => "WHATSAPP-LOG" }
    }
  }

  if [indexType] == "WHATSAPP-LOG" {
    # you have to supply what you want to do here. 
  }

}
9

Alright thank you so much for the reply. i will effect the corrections now.

please is it also possible you suggest what i can input under the second condition? i am very new to this, just following a guide to analyze whatsapp chat with ELK. i do not really know what to input there. thank you for your help

10

this is the current error. am beyond stuck now

][logstash.runner ] The given configuration is invalid. Reason: Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 28, column 17 (byte 583) after filter {

if [type] == "whatsapp-alerts" {
grok{
break_on_match => false
named_captures_only => true
match => ["message" , "\s*%{NOTSPACE:logdate},\s*%{GREEDYDATA}-%{SPACE}\s*%{GREEDYDATA:PhoneNumber}:\s*%{GREEDYDATA:Message}"]
}
date {
match => [ "logdate", "ISO8601" , "M/d/yy" ]
target => "logdate"
}
mutate {
add_field => { "indexType" => "WHATSAPP-LOG" }
}
}

output {

elasticsearch
[2020-04-23T16:52:35,696][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

11

this is the output section

output {

elasticsearch{
hosts => "localhost:5044"
index => "%{whatsapp-logs}-%{+YYYY.MM.dd}"
document_type => doc
user => "indent"
password => "changeme"
}

12

You are missing the } to close the filter section.

13

oh my goodness. thank you so so much @Badger. this finally solved the problem. i have been on this for a long time. i cant believe that was the simple solution. its really not easy being a newbie. wow! thanks

also thanks everyone @Rahul_Kumar4 @Christian_Dahlqvist. your solution also helped for the earlier issues

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.