The grok part from logstash didn't take effect in elasticsearch

Elastic Stack Logstash
1

Below is my logstash config:

logstash
logstash.png1489×596 19.4 KB

And I find only the document_type is effect after I start the logstash. I go to elasticsearch: hostname:9200/indexName and find as below:
elasticsearch
elasticsearch.png1587×194 15.4 KB

So I find the grok part didn't take effect. Is there anything can help to debug or any other idea?

Notes: I have check all the logs but no erro or warning.

2

I have restart the ELK part but also the same behavior.

3

I debug run logstash and find this as below；

debug
debug.png1587×401 13.2 KB

Any idea about this issue?

4

Yes, I think I got why can't parsing as Filter config. My origin log will not generate as grok format. So it can't match with grok.
My question is: How to intellect matching the logs with diff format in logs? use condition? But how to use it?

6

Don't use grok to parse JSON! Use a json filter or json codec.

7

Thanks， But my log have diff format of json. How to handle that.
For example: 2 types
1.{"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}}

2.{"elapsed time":2.666, "request":{"method":"POST","uri":"jobs","payload":{"application":"ASmp","db":"Sample","jobtype":"maxl","parameters":{"script":"createfilters_ASmp.msh"}}}}

8

I will hit [1] "_jsonparsefailure" during the log format as:
{"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}}

But does this is also a json format?

9

I mean if there is non-standard json format in my log. Then how to deal with that?

10

In your first example, the colon after "request" isn't actually a colon but some other character that looks like a colon. Your second example doesn't have that problem.

Apart from that those JSON strings are fine.

11

Thanks,magnusbaeck.

I have to verify my json is valid here: https://jsonlint.com/

{"elapsed time":40.666,"executed by":{"user":"BDServiceAdministrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"CocaCola.zip"}}}}

but my logstash still can't parse it, logstash point to line line 1 column 16 which always point to "elapsed time":40.666, I try to remove it and re-parsing but will point to next one "executed by":

12

Hey,

I find this https://github.com/logstash-plugins/logstash-codec-json/issues/35
which is said empty key is valid in json but logstash can't parsing it. So I change my json format to

{"elapsedtime":40.666,"executedby":{"user":"Administrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"Coc.zip"}}}}

And logstash can parsing successful. But the my second json format didn't have this issue.

Second format as:
{"elapsed time":0.675, "request":{"method":"PUT","uri":"files/applications/ASOsamp/Sample/createfilters_ASOsamp.msh?overwrite=true"}}
it always works well. Do you know why?

13

If I copy/paste the JSON string above into jsonlint.com it works just fine.

14

Yes, It work find and valid. But for logstash parse part it will failed.

15

Works fine for me with Logstash 6.1:

$ cat test.config 
input { stdin { codec => json_lines } }
output { stdout { codec => rubydebug } }
$ cat data 
{"elapsed time":40.666,"executed by":{"user":"BDServiceAdministrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"CocaCola.zip"}}}}
$ logstash -f test.config < data
Sending Logstash's logs to /home/magnus/logstash/logstash-6.1.0/logs which is now configured via log4j2.properties
[2018-07-24T11:39:58,360][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/magnus/logstash/logstash-6.1.0/modules/fb_apache/configuration"}
[2018-07-24T11:39:58,372][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/magnus/logstash/logstash-6.1.0/modules/netflow/configuration"}
[2018-07-24T11:39:58,778][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-24T11:39:59,313][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.0"}
[2018-07-24T11:39:59,765][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-07-24T11:40:02,160][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x1bc2d1a0 run>"}
[2018-07-24T11:40:02,224][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"main"}
[2018-07-24T11:40:02,363][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
     "executed by" => {
        "password" => "Welcome1",
            "user" => "BDServiceAdministrator"
    },
        "@version" => "1",
      "@timestamp" => 2018-07-24T09:40:02.376Z,
            "host" => "bertie",
    "elapsed time" => 40.666,
         "request" => {
            "uri" => "jobs",
        "payload" => {
               "jobtype" => "lcmImport",
            "parameters" => {
                "zipFileName" => "CocaCola.zip"
            }
        },
         "method" => "POST"
    }
}
[2018-07-24T11:40:03,004][INFO ][logstash.pipeline        ] Pipeline terminated {"pipeline.id"=>"main"}
16

Thanks, I am failed with below json but not the one which you use. Could you help to use below json to take a try in 6.1? I am working with 5.6.3, I think it doesn't matter with diff version.

{"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}}

17

As I said earlier, the colon after "request" isn't actually a colon but some other character that looks like a colon. To be extremely explicit:

..."request"：{"method"...
            ^
            |
             ----- this is not a colon
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.