# Filter Error logs only **URL:** \ **Category:** Logstash\ **Created:** [February 11, 2020, 2:29am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701 "2020-02-11T02:29:48Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 11, 2020, 2:29am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/1 "2020-02-11T02:29:48Z")
Hi, i am still very new to the ELK stack and i am wondering how can i filter only error logs through logstash? i am currently feeding application, security and system logs into Logstash. below is my configuration for logstash: input { beats { port =\> 5044 type =\> "log" } } output { elasticsearch { ``` hosts => ["http://localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{+yyyy.ww}" document_type => "%{[@metadata][type]}" ``` } #jdbc{ ``` #driver_jar_path => “C:\MSSQLJDBC\sqljdbc_4.2\enu\jre8\sqljdbc42.jar” #connection_string => “jdbc:sqlserver://localhost:1433;databaseName=databasename;user=username;password=userpassword” #statement => [“INSERT into DB.TMP.samsamt (w, test1, test2, kkk) VALUES(?,?,?,?)”, “client_num”, “consignee_name”, “@timestamp”,”host”] ``` #} } ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [February 11, 2020, 11:20am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/2 "2020-02-11T11:20:49Z")
Hi there, well, first of all make an effort to properly format all your code, otherwise it'll be very unlikely that someone else makes an effort to understand what you are writing. Then, you have to give us a little bit more info about your data, otherwise we won't know how to help you filter them. What is a sample input event? Can you post the output of this pipeline? ``` input { beats { port => 5044 type => "log" } } filter {} output { stdout{} } ``` ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 11, 2020, 6:54pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/3 "2020-02-11T18:54:50Z")
Hi fabio, thank you for your reply. here is what i have as output, i would like for logstash to only forward the information about the host, the computer name and event id tied to the winlog , the timestamp the event provider and it's action as well as the log level. The message as well. the rest could be disregarded. how can i write a filter for it? { "message" =\> "A user's local group membership was enumerated.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDESKTOP-LMVLRUC$\n\tAccount Domain:\t\tWORKGROUP\n\tLogon ID:\t\t0x3E7\n\nUser:\n\tSecurity ID:\t\tS-1-5-21-3748035913-1059597750-253446-1001\n\tAccount Name:\t\tadamb\n\tAccount Domain:\t\tDESKTOP-LMVLRUC\n\nProcess Information:\n\tProcess ID:\t\t0x29f4\n\tProcess Name:\t\tC:\Windows\System32\svchost.exe", "@version" =\> "1", "host" =\> { "architecture" =\> "x86\_64", "os" =\> { "version" =\> "10.0", "family" =\> "windows", "kernel" =\> "10.0.18362.628 (WinBuild.160101.0800)", "name" =\> "Windows 10 Home", "build" =\> "18363.628", "platform" =\> "windows" }, "hostname" =\> "DESKTOP-LMVLRUC", "id" =\> "b0733bdc-476e-4005-9066-b13849a0c783", "name" =\> "DESKTOP-LMVLRUC" }, "type" =\> "log", "agent" =\> { "type" =\> "winlogbeat", "ephemeral\_id" =\> "f634378a-a8a2-46e2-914a-96f8dffc909b", "hostname" =\> "DESKTOP-LMVLRUC", "id" =\> "ac81bd49-e2b2-4bde-a3e4-b295ce5b52bb", "version" =\> "7.5.2" }, "tags" =\> [ [0] "beats\_input\_codec\_plain\_applied" ], "@timestamp" =\> 2020-02-11T19:09:13.821Z, "winlog" =\> { "process" =\> { "thread" =\> { "id" =\> 944 }, "pid" =\> 864 }, "provider\_name" =\> "Microsoft-Windows-Security-Auditing", "provider\_guid" =\> "{54849625-5478-4994-a5ba-3e3b0328c30d}", "event\_id" =\> 4798, "task" =\> "User Account Management", "computer\_name" =\> "DESKTOP-LMVLRUC", "activity\_id" =\> "{bd1bfc0b-e109-0003-2cfc-1bbd09e1d501}", "opcode" =\> "Info", "record\_id" =\> 35109, "api" =\> "wineventlog", "event\_data" =\> { "TargetSid" =\> "S-1-5-21-3748035913-1059597750-253446-1001", "SubjectUserName" =\> "DESKTOP-LMVLRUC$", "CallerProcessName" =\> "C:\Windows\System32\svchost.exe", "SubjectLogonId" =\> "0x3e7", "TargetDomainName" =\> "DESKTOP-LMVLRUC", "CallerProcessId" =\> "0x29f4", "SubjectUserSid" =\> "S-1-5-18", "SubjectDomainName" =\> "WORKGROUP", "TargetUserName" =\> "adamb" }, "keywords" =\> [ [0] "Audit Success" ], "channel" =\> "Security" }, "ecs" =\> { "version" =\> "1.1.0" }, "event" =\> { "action" =\> "User Account Management", "code" =\> 4798, "provider" =\> "Microsoft-Windows-Security-Auditing", "created" =\> "2020-02-11T19:09:15.179Z", "kind" =\> "event" }, "log" =\> { "level" =\> "information" } } ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [February 11, 2020, 8:08pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/4 "2020-02-11T20:08:50Z")
Man, you gotta format your code. Highlight the part of the text to format (with proper newline before and after that part) and click on this icon ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e264eb41037f704118a2dbdc2a8640ee9fe01ba5.png) After that, try to properly write what you are searching for, making a list of the needed fields, INCLUDING their nesting level. If your looking for the event\_id nested in winlog, then you write `[winlog][event_id]` or `winlog.event_id` so it is immediately clear for the readers where to look for that field. If you write > event id tied to the winlog you're forcing the reader to do extra work to help you. ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 12, 2020, 12:55am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/5 "2020-02-12T00:55:14Z")
i will try rephrase and space out the code as best i can... i am not used to posting here as you can see. i am using winlogbeat to collect application, security and system logs, these seem to be the most common feilds accross all three, i require them as i would like to forward information to external database later on I am looking to extract these feilds: `[message] [host.architecture] [host.os.version] [host.os.name] [host.os.kernel] [host.os.build] [host.hostname] [host.id] [host.name], [type] [agent.type] [agent.id] [agent.version], [@timestamp], [winlog.provider_name] [winlog.process.thread.id] [winlog.keyword] [winlog.event_data.TargetUserName] [winlog.channel] [event.created] [event.kind] [log.level]` below is the ouput of my pipeline in the cmd, i have applied no filter and would like to know how to create one using grok. ``` { "message" => "A user's local group membership was enumerated.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDESKTOP-LMVLRUC$\n\tAccount Domain:\t\tWORKGROUP\n\tLogon ID:\t\t0x3E7\n\nUser:\n\tSecurity ID:\t\tS-1-5-21-3748035913-1059597750-253446-1001\n\tAccount Name:\t\tadamb\n\tAccount Domain:\t\tDESKTOP-LMVLRUC\n\nProcess Information:\n\tProcess ID:\t\t0x29f4\n\tProcess Name:\t\tC:\\Windows\\System32\\svchost.exe", "@version" => "1", "host" => { "architecture" => "x86_64", "os" => { "version" => "10.0", "family" => "windows", "kernel" => "10.0.18362.628 (WinBuild.160101.0800)", "name" => "Windows 10 Home", "build" => "18363.628", "platform" => "windows" }, "hostname" => "DESKTOP-LMVLRUC", "id" => "b0733bdc-476e-4005-9066-b13849a0c783", "name" => "DESKTOP-LMVLRUC" }, "type" => "log", "agent" => { "type" => "winlogbeat", "ephemeral_id" => "f634378a-a8a2-46e2-914a-96f8dffc909b", "hostname" => "DESKTOP-LMVLRUC", "id" => "ac81bd49-e2b2-4bde-a3e4-b295ce5b52bb", "version" => "7.5.2" }, "tags" => [ [0] "beats_input_codec_plain_applied" ], "@timestamp" => 2020-02-11T19:09:13.821Z, "winlog" => { "process" => { "thread" => { "id" => 944 }, "pid" => 864 }, "provider_name" => "Microsoft-Windows-Security-Auditing", "provider_guid" => "{54849625-5478-4994-a5ba-3e3b0328c30d}", "event_id" => 4798, "task" => "User Account Management", "computer_name" => "DESKTOP-LMVLRUC", "activity_id" => "{bd1bfc0b-e109-0003-2cfc-1bbd09e1d501}", "opcode" => "Info", "record_id" => 35109, "api" => "wineventlog", "event_data" => { "TargetSid" => "S-1-5-21-3748035913-1059597750-253446-1001", "SubjectUserName" => "DESKTOP-LMVLRUC$", "CallerProcessName" => "C:\\Windows\\System32\\svchost.exe", "SubjectLogonId" => "0x3e7", "TargetDomainName" => "DESKTOP-LMVLRUC", "CallerProcessId" => "0x29f4", "SubjectUserSid" => "S-1-5-18", "SubjectDomainName" => "WORKGROUP", "TargetUserName" => "adamb" }, "keywords" => [ [0] "Audit Success" ], "channel" => "Security" }, "ecs" => { "version" => "1.1.0" }, "event" => { "action" => "User Account Management", "code" => 4798, "provider" => "Microsoft-Windows-Security-Auditing", "created" => "2020-02-11T19:09:15.179Z", "kind" => "event" }, "log" => { "level" => "information" } } ``` thank you! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2020, 1:59am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/6 "2020-02-12T01:59:31Z")
> [@Adam\_Songambele](#): > > I am looking to extract these feilds: > > `[message] [host.architecture] [host.os.version]` No, you are not. logstash can support both fields whose names contain periods, and fields nested inside other fields. [host.architecture] and [host][architecture] are _different_. The first is a field whose name contains a period. The second is field called [host] that is an object that contains another field called [architecture]. Thankyou for posting the rubydebug output. Here is why we so often ask for it... ``` "host" => { "architecture" => "x86_64", "os" => { "version" => "10.0", ``` The [host] field is an object that contains an architecture field. In logstash that is called ``` [host][architecture] ``` The [host] object also contains an [os] object, that contains a [version] field. So that is ``` [host][os][version] ``` in logstash. host.os.version in kibana could be a [host] field that contain an [os] object, that contains a [version] field. Or a [host] object that contains an [os.version] field. Or a [host.os] object that contains a [version] field. logstash clearly distinguishes between those. kibana, not so much. There may be someone on the planet who has has to deal with both [host][os.version] and [host.os][version] in the same elasticsearch/kibana instance (but I doubt it). If you can find them they may be able to explain how _that_ works. It wouldn't surprise me if there were some wicked X-Pack ML bugs related to that. Correlation, or auto-correlation 😃 ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 12, 2020, 2:14am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/7 "2020-02-12T02:14:58Z")
Hi Badger, Thank you for the reply, but could you provide me with a small example of a filter related to my fields so i could understand how grok work? as the only documentation they provide is quite criptic. also would i need to specify the json as type in the filter ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 12, 2020, 2:28am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/8 "2020-02-12T02:28:49Z")
That's difficult because the output looks like well structured rubydebug output. In which case logstash has done its work and you would be able to save the structured data. ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 12, 2020, 2:48am UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/9 "2020-02-12T02:48:35Z")
yes but i would like to filter the fields so i can only save about 7 and remove the rest, should i mutate the fields by removing those that i don't want first ? or will the filtering only keep the ones i want ? ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [February 13, 2020, 12:25pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/10 "2020-02-13T12:25:55Z")
If you're receiving as input a json with those fields, you could try something like ``` filter { ruby { code => " def whitelist_fields(whitelist, event) root = '[whitelisted_fields]' whitelist.each { |field| event.set(root + field, event.get(field)) unless event.get(field).nil? } event.to_hash.each { |k, v| event.remove(k) unless k == 'whitelisted_fields' } event.get('whitelisted_fields').each { |k, v| event.set(k, v) } event.remove('whitelisted_fields') end whitelist = [ '[message]', '[type]', '[@timestamp]', '[host][architecture]', '[host][hostname]', '[host][id]', '[host][name]', '[host][os][version]', '[host][os][name]', '[host][os][kernel]', '[host][os][build]', '[winlog][provider_name]', '[winlog][process][thread][id]', '[winlog][keyword]', '[winlog][event_data][TargetUserName]', '[winlog][channel]', '[agent][type]', '[agent][id]', '[agent][version]', '[event][created]', '[event][kind]', '[log][level]' ] whitelist_fields(whitelist, event) " } } ``` ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 15, 2020, 5:27pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/11 "2020-02-15T17:27:00Z")
Hi fabio thank you for the reply, i have used the filter you have provided and it worked initially. However when i tried adding an additionnal field using ruby filter or mutate option it did not working, i had no output in the command line. However, even after removing and running `> .\logstash -f C:\ProgramData\Elastic\logstash-7.5.2\config\logstash.conf` it still did not output anything and i do not understand it because the service is starting but not outputing ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [February 15, 2020, 6:04pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/12 "2020-02-15T18:04:33Z")
You added it AFTER the `whitelist_fields(whitelist, event)` line? If you added it BEFORE that line and didn't whitelist the field, it obviously didn't work. Can you show your pipeline (formatted)? And can you show the output of your pipeline (again formatted) commenting out the whole part of the filter I wrote (so the ruby plugin basically)? ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 15, 2020, 9:01pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/13 "2020-02-15T21:01:42Z")
First of all sorry for all the question as i am not used to developpe in this environment. Yes the line was added after the `whiteListing function`, i realsied that it wasn't working because of the output configuration as i am trying to output to both a local instance of mongoDB and elastic search, do you mind having a look at this and tell me why the elastic search is giving me a `"field not found " error` and why the `mongo output` is not working ? here is my output configuration : ``` output { if [log][level] == "warning" or [log][level] == "error"{ stdout { codec => rubydebug } elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{+yyyy.ww}" document_type => "%{[@metadata][type]}" } mongodb { uri => "mongodb://localhost:27017" # Database URL database => "rasadb" collection => "rasaCollection" isodate => true } } } ``` ---
**Author:** ![Adam\_Songambele](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_songambele/32/47708_2.png) [@Adam\_Songambele](https://discuss.elastic.co/u/Adam_Songambele)\ **Post date:** [February 15, 2020, 9:12pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/14 "2020-02-15T21:12:46Z")
I have just solved the mongoDB issue which was a conversion problem with the `isodate` variable so i have just removed it. but your help would very much appreciated with the elasticsearch output ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [February 15, 2020, 10:20pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/15 "2020-02-15T22:20:26Z")
Well, you sure you do have those `[@metadata][beat]` and `[@metadata][type]` fields? On which of those two does it throw the "field not found" error? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 14, 2020, 10:20pm UTC](https://discuss.elastic.co/t/filter-error-logs-only/218701/16 "2020-03-14T22:20:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.