Filter Error logs only

Adam_Songambele · February 11, 2020, 2:29am

Hi,

i am still very new to the ELK stack and i am wondering how can i filter only error logs through logstash? i am currently feeding application, security and system logs into Logstash.

below is my configuration for logstash:

input {

beats {

port => 5044

type => "log"

}

output {

elasticsearch {

hosts => ["http://localhost:9200"]

manage_template => false

index => "%{[@metadata][beat]}-%{+yyyy.ww}"

document_type => "%{[@metadata][type]}"

}

#jdbc{

#driver_jar_path => “C:\MSSQLJDBC\sqljdbc_4.2\enu\jre8\sqljdbc42.jar”

#connection_string => “jdbc:sqlserver://localhost:1433;databaseName=databasename;user=username;password=userpassword”

#statement => [ “INSERT into DB.TMP.samsamt (w, test1, test2, kkk) VALUES(?,?,?,?)”, “client_num”, “consignee_name”, “@timestamp”,”host” ]

#}

}

Fabio-sama · February 11, 2020, 11:20am

Hi there,

well, first of all make an effort to properly format all your code, otherwise it'll be very unlikely that someone else makes an effort to understand what you are writing.

Then, you have to give us a little bit more info about your data, otherwise we won't know how to help you filter them.

What is a sample input event?

Can you post the output of this pipeline?

input {
  beats {
    port => 5044
    type => "log"
  }
}

filter {}

output {
  stdout{}
}

Adam_Songambele · February 11, 2020, 6:54pm

Hi fabio, thank you for your reply. here is what i have as output, i would like for logstash to only forward the information about the host, the computer name and event id tied to the winlog , the timestamp the event provider and it's action as well as the log level. The message as well. the rest could be disregarded. how can i write a filter for it?

{
"message" => "A user's local group membership was enumerated.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDESKTOP-LMVLRUC$\n\tAccount Domain:\t\tWORKGROUP\n\tLogon ID:\t\t0x3E7\n\nUser:\n\tSecurity ID:\t\tS-1-5-21-3748035913-1059597750-253446-1001\n\tAccount Name:\t\tadamb\n\tAccount Domain:\t\tDESKTOP-LMVLRUC\n\nProcess Information:\n\tProcess ID:\t\t0x29f4\n\tProcess Name:\t\tC:\Windows\System32\svchost.exe",
"@version" => "1",
"host" => {
"architecture" => "x86_64",
"os" => {
"version" => "10.0",
"family" => "windows",
"kernel" => "10.0.18362.628 (WinBuild.160101.0800)",
"name" => "Windows 10 Home",
"build" => "18363.628",
"platform" => "windows"
},
"hostname" => "DESKTOP-LMVLRUC",
"id" => "b0733bdc-476e-4005-9066-b13849a0c783",
"name" => "DESKTOP-LMVLRUC"
},
"type" => "log",
"agent" => {
"type" => "winlogbeat",
"ephemeral_id" => "f634378a-a8a2-46e2-914a-96f8dffc909b",
"hostname" => "DESKTOP-LMVLRUC",
"id" => "ac81bd49-e2b2-4bde-a3e4-b295ce5b52bb",
"version" => "7.5.2"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2020-02-11T19:09:13.821Z,
"winlog" => {
"process" => {
"thread" => {
"id" => 944
},
"pid" => 864
},
"provider_name" => "Microsoft-Windows-Security-Auditing",
"provider_guid" => "{54849625-5478-4994-a5ba-3e3b0328c30d}",
"event_id" => 4798,
"task" => "User Account Management",
"computer_name" => "DESKTOP-LMVLRUC",
"activity_id" => "{bd1bfc0b-e109-0003-2cfc-1bbd09e1d501}",
"opcode" => "Info",
"record_id" => 35109,
"api" => "wineventlog",
"event_data" => {
"TargetSid" => "S-1-5-21-3748035913-1059597750-253446-1001",
"SubjectUserName" => "DESKTOP-LMVLRUC$",
"CallerProcessName" => "C:\Windows\System32\svchost.exe",
"SubjectLogonId" => "0x3e7",
"TargetDomainName" => "DESKTOP-LMVLRUC",
"CallerProcessId" => "0x29f4",
"SubjectUserSid" => "S-1-5-18",
"SubjectDomainName" => "WORKGROUP",
"TargetUserName" => "adamb"
},
"keywords" => [
[0] "Audit Success"
],
"channel" => "Security"
},
"ecs" => {
"version" => "1.1.0"
},
"event" => {
"action" => "User Account Management",
"code" => 4798,
"provider" => "Microsoft-Windows-Security-Auditing",
"created" => "2020-02-11T19:09:15.179Z",
"kind" => "event"
},
"log" => {
"level" => "information"
}
}

Fabio-sama · February 11, 2020, 8:08pm

Man, you gotta format your code. Highlight the part of the text to format (with proper newline before and after that part) and click on this icon

After that, try to properly write what you are searching for, making a list of the needed fields, INCLUDING their nesting level. If your looking for the event_id nested in winlog, then you write

[winlog][event_id] or winlog.event_id

so it is immediately clear for the readers where to look for that field. If you write

event id tied to the winlog

you're forcing the reader to do extra work to help you.

Adam_Songambele · February 12, 2020, 12:55am

i will try rephrase and space out the code as best i can... i am not used to posting here as you can see.

i am using winlogbeat to collect application, security and system logs, these seem to be the most common feilds accross all three, i require them as i would like to forward information to external database later on

I am looking to extract these feilds:

[message] [host.architecture] [host.os.version] [host.os.name] [host.os.kernel] [host.os.build] [host.hostname] [host.id] [host.name], [type] [agent.type] [agent.id] [agent.version], [@timestamp], [winlog.provider_name] [winlog.process.thread.id] [winlog.keyword] [winlog.event_data.TargetUserName] [winlog.channel] [event.created] [event.kind] [log.level]

below is the ouput of my pipeline in the cmd, i have applied no filter and would like to know how to create one using grok.

  {
       "message" => "A user's local group membership was enumerated.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDESKTOP-LMVLRUC$\n\tAccount Domain:\t\tWORKGROUP\n\tLogon ID:\t\t0x3E7\n\nUser:\n\tSecurity ID:\t\tS-1-5-21-3748035913-1059597750-253446-1001\n\tAccount Name:\t\tadamb\n\tAccount Domain:\t\tDESKTOP-LMVLRUC\n\nProcess Information:\n\tProcess ID:\t\t0x29f4\n\tProcess Name:\t\tC:\\Windows\\System32\\svchost.exe",
      "@version" => "1",
          "host" => {
        "architecture" => "x86_64",
                  "os" => {
             "version" => "10.0",
              "family" => "windows",
              "kernel" => "10.0.18362.628 (WinBuild.160101.0800)",
                "name" => "Windows 10 Home",
               "build" => "18363.628",
            "platform" => "windows"
        },
            "hostname" => "DESKTOP-LMVLRUC",
                  "id" => "b0733bdc-476e-4005-9066-b13849a0c783",
                "name" => "DESKTOP-LMVLRUC"
    },
          "type" => "log",
         "agent" => {
                "type" => "winlogbeat",
        "ephemeral_id" => "f634378a-a8a2-46e2-914a-96f8dffc909b",
            "hostname" => "DESKTOP-LMVLRUC",
                  "id" => "ac81bd49-e2b2-4bde-a3e4-b295ce5b52bb",
             "version" => "7.5.2"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "@timestamp" => 2020-02-11T19:09:13.821Z,
        "winlog" => {
              "process" => {
            "thread" => {
                "id" => 944
            },
               "pid" => 864
        },
        "provider_name" => "Microsoft-Windows-Security-Auditing",
        "provider_guid" => "{54849625-5478-4994-a5ba-3e3b0328c30d}",
             "event_id" => 4798,
                 "task" => "User Account Management",
        "computer_name" => "DESKTOP-LMVLRUC",
          "activity_id" => "{bd1bfc0b-e109-0003-2cfc-1bbd09e1d501}",
               "opcode" => "Info",
            "record_id" => 35109,
                  "api" => "wineventlog",
           "event_data" => {
                    "TargetSid" => "S-1-5-21-3748035913-1059597750-253446-1001",
              "SubjectUserName" => "DESKTOP-LMVLRUC$",
            "CallerProcessName" => "C:\\Windows\\System32\\svchost.exe",
               "SubjectLogonId" => "0x3e7",
             "TargetDomainName" => "DESKTOP-LMVLRUC",
              "CallerProcessId" => "0x29f4",
               "SubjectUserSid" => "S-1-5-18",
            "SubjectDomainName" => "WORKGROUP",
               "TargetUserName" => "adamb"
        },
             "keywords" => [
            [0] "Audit Success"
        ],
              "channel" => "Security"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
         "event" => {
          "action" => "User Account Management",
            "code" => 4798,
        "provider" => "Microsoft-Windows-Security-Auditing",
         "created" => "2020-02-11T19:09:15.179Z",
            "kind" => "event"
    },
           "log" => {
        "level" => "information"
    }
}

thank you!

Badger · February 12, 2020, 1:59am

No, you are not. logstash can support both fields whose names contain periods, and fields nested inside other fields. [host.architecture] and [host][architecture] are different. The first is a field whose name contains a period. The second is field called [host] that is an object that contains another field called [architecture].

Thankyou for posting the rubydebug output. Here is why we so often ask for it...

      "host" => {
    "architecture" => "x86_64",
              "os" => {
         "version" => "10.0",

The [host] field is an object that contains an architecture field. In logstash that is called

[host][architecture]

The [host] object also contains an [os] object, that contains a [version] field. So that is

[host][os][version]

in logstash. host.os.version in kibana could be a [host] field that contain an [os] object, that contains a [version] field. Or a [host] object that contains an [os.version] field. Or a [host.os] object that contains a [version] field.

logstash clearly distinguishes between those. kibana, not so much. There may be someone on the planet who has has to deal with both [host][os.version] and [host.os][version] in the same elasticsearch/kibana instance (but I doubt it). If you can find them they may be able to explain how that works.

It wouldn't surprise me if there were some wicked X-Pack ML bugs related to that. Correlation, or auto-correlation

Adam_Songambele · February 12, 2020, 2:14am

Hi Badger,
Thank you for the reply,

but could you provide me with a small example of a filter related to my fields so i could understand how grok work? as the only documentation they provide is quite criptic. also would i need to specify the json as type in the filter ?

Badger · February 12, 2020, 2:28am

That's difficult because the output looks like well structured rubydebug output. In which case logstash has done its work and you would be able to save the structured data.

Adam_Songambele · February 12, 2020, 2:48am

yes but i would like to filter the fields so i can only save about 7 and remove the rest, should i mutate the fields by removing those that i don't want first ? or will the filtering only keep the ones i want ?

Fabio-sama · February 13, 2020, 12:25pm

If you're receiving as input a json with those fields, you could try something like

filter {
  ruby {
    code => "
      def whitelist_fields(whitelist, event)
        root = '[whitelisted_fields]'
        whitelist.each { |field| event.set(root + field, event.get(field)) unless event.get(field).nil? }
        event.to_hash.each { |k, v| event.remove(k) unless k == 'whitelisted_fields' }
        event.get('whitelisted_fields').each { |k, v| event.set(k, v) }
        event.remove('whitelisted_fields')
      end

      whitelist = [
        '[message]',
        '[type]',
        '[@timestamp]',
        '[host][architecture]',
        '[host][hostname]',
        '[host][id]',
        '[host][name]',
        '[host][os][version]',
        '[host][os][name]',
        '[host][os][kernel]',
        '[host][os][build]',
        '[winlog][provider_name]',
        '[winlog][process][thread][id]',
        '[winlog][keyword]',
        '[winlog][event_data][TargetUserName]',
        '[winlog][channel]',
        '[agent][type]',
        '[agent][id]',
        '[agent][version]',
        '[event][created]',
        '[event][kind]',
        '[log][level]'
      ]

      whitelist_fields(whitelist, event)
    "
  }
}

Adam_Songambele · February 15, 2020, 5:27pm

Hi fabio thank you for the reply,

i have used the filter you have provided and it worked initially. However when i tried adding an additionnal field using ruby filter or mutate option it did not working, i had no output in the command line.

However, even after removing and running

> .\logstash -f C:\ProgramData\Elastic\logstash-7.5.2\config\logstash.conf it still did not output anything and i do not understand it because the service is starting but not outputing

Fabio-sama · February 15, 2020, 6:04pm

You added it AFTER the whitelist_fields(whitelist, event) line? If you added it BEFORE that line and didn't whitelist the field, it obviously didn't work.

Can you show your pipeline (formatted)? And can you show the output of your pipeline (again formatted) commenting out the whole part of the filter I wrote (so the ruby plugin basically)?

Adam_Songambele · February 15, 2020, 9:01pm

First of all sorry for all the question as i am not used to developpe in this environment.

Yes the line was added after the whiteListing function, i realsied that it wasn't working because
of the output configuration as i am trying to output to both a local instance of mongoDB and elastic search, do you mind having a look at this and tell me why the elastic search is giving me a "field not found " error and why the mongo output is not working ? here is my output configuration :

 output {
  
  if [log][level] == "warning" or [log][level] == "error"{
      
      stdout { codec => rubydebug } 
      elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{+yyyy.ww}"
        document_type => "%{[@metadata][type]}"
      }

      mongodb {
      uri => "mongodb://localhost:27017" # Database URL 
      database => "rasadb"
      collection => "rasaCollection"
      isodate => true
      }

  }

}

Adam_Songambele · February 15, 2020, 9:12pm

I have just solved the mongoDB issue which was a conversion problem with the isodate variable so i have just removed it. but your help would very much appreciated with the elasticsearch output

Fabio-sama · February 15, 2020, 10:20pm

Well, you sure you do have those [@metadata][beat] and [@metadata][type] fields?

On which of those two does it throw the "field not found" error?

system · March 14, 2020, 10:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.