Extract rows from log using http plugin

Elastic Stack Logstash
1

Hi,

I'm trying to use http plugin to receive log with this format :
2017-10-31-07:29:05 - DEBUG Waiting for callback, for command id : 56232
2017-10-31-07:29:05 - DEBUG Waiting for callback...
2017-10-31-07:29:06 - DEBUG Callback called after 1000

I have this configuration file :

input {
        http {
        } 

    } 

    filter
    {

        split{
            }
        grok {
            match => { "message" => "(?<date>\d{4}-\d{2}-\d{2}-\d{2}:\d{2}:\d{2}) - %{WORD:loglevel} %{GREEDYDATA:description}"}
        }


    }
    output { 
        stdout { 
            codec => rubydebug
            } 
    }

Unfortunately, logstash reads all the log input in one event. (as if it was an one line log file).
How should I configure logstash?
thank you very much by advance

Mohamed

1 Like
2

Works fine for me. Please show an example from your stdout output.

3

Thank you for answering

I have this result :

{
           "date" => "2017-10-31-07:29:05",
        "headers" => {
            "http_accept" => "*/*",
           "content_type" => "application/x-www-form-urlencoded",
           "request_path" => "/twitter/tweet/1",
           "http_version" => "HTTP/1.1",
         "request_method" => "PUT",
              "http_host" => "localhost:8080",
            "request_uri" => "/twitter/tweet/1",
         "content_length" => "177",
        "http_user_agent" => "curl/7.38.0"
    },
     "@timestamp" => 2017-11-06T09:20:42.873Z,
       "loglevel" => "DEBUG",
       "@version" => "1",
           "host" => "127.0.0.1",
    "description" => "Waiting for callback, for command id : 562322017-10-31-07:29:05 - DEBUG Waiting for callback...2017-10-31-07:29:06 - DEBUG Callback called after 1000",
        "message" => "2017-10-31-07:29:05 - DEBUG Waiting for callback, for command id : 562322017-10-31-07:29:05 - DEBUG Waiting for callback...2017-10-31-07:29:06 - DEBUG Callback called after 1000"
}

I would like something like :

{
           "date" => "2017-10-31-07:29:05",
        "headers" => {
            "http_accept" => "*/*",
           "content_type" => "application/x-www-form-urlencoded",
           "request_path" => "/twitter/tweet/1",
           "http_version" => "HTTP/1.1",
         "request_method" => "PUT",
              "http_host" => "localhost:8080",
            "request_uri" => "/twitter/tweet/1",
         "content_length" => "177",
        "http_user_agent" => "curl/7.38.0"
    },
     "@timestamp" => 2017-11-06T09:20:42.873Z,
       "loglevel" => "DEBUG",
       "@version" => "1",
           "host" => "127.0.0.1",
    "description" => "Waiting for callback, for command id : 562322",
        "message" => "2017-10-31-07:29:05 - DEBUG Waiting for callback, for command id : 56232"
},

{
           "date" => "2017-10-31-07:29:05",
        "headers" => {
            "http_accept" => "*/*",
           "content_type" => "application/x-www-form-urlencoded",
           "request_path" => "/twitter/tweet/1",
           "http_version" => "HTTP/1.1",
         "request_method" => "PUT",
              "http_host" => "localhost:8080",
            "request_uri" => "/twitter/tweet/1",
         "content_length" => "177",
        "http_user_agent" => "curl/7.38.0"
    },
     "@timestamp" => 2017-11-06T09:20:42.873Z,
       "loglevel" => "DEBUG",
       "@version" => "1",
           "host" => "127.0.0.1",
    "description" => "Waiting for callback, for command id : 562322",
        "message" => "2017-10-31-07:29:05 - DEBUG Waiting for callback..."
},

{
           "date" => "2017-10-31-07:29:06",
        "headers" => {
            "http_accept" => "*/*",
           "content_type" => "application/x-www-form-urlencoded",
           "request_path" => "/twitter/tweet/1",
           "http_version" => "HTTP/1.1",
         "request_method" => "PUT",
              "http_host" => "localhost:8080",
            "request_uri" => "/twitter/tweet/1",
         "content_length" => "177",
        "http_user_agent" => "curl/7.38.0"
    },
     "@timestamp" => 2017-11-06T09:20:42.873Z,
       "loglevel" => "DEBUG",
       "@version" => "1",
           "host" => "127.0.0.1",
    "description" => "Callback called after 1000",
        "message" => "2017-10-31-07:29:06 - DEBUG Callback called after 1000"
}

thank you

4

Okay, weird. I get the expected result with Logstash 2.4.

5

I have the 5.6.0 version (result of logstash --version)

I'm using the last version of ELK

6

Any other idea maybe?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.