How do I get first line from multiline message by using logstash 1.5?

Elastic Stack Logstash
1

Here is a sample:

{
     "@timestamp" => "2015-06-03T11:12:43.776Z",
        "message" => "2015-06-03 16:40:30,356 [DefaultQuartzScheduler_Worker-9] ERROR StackTrace  - Full Stack Trace:\norg.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only\nhawkeyes.logat hawkeyes.quartz.JobTemplate.execute(JobTemplate.groovy:10)\nhawkeyes.logat org.quartz.core.JobRunShell.run(JobRunShell.java:202)\nhawkeyes.logat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)",
       "@version" => "1",
           "tags" => [
        [0] "multiline",
        [1] "hawkeyes_log",
        [2] "log4j"
    ],
           "type" => "hawkeyes",
           "host" => "hawkeyesTest",
           "path" => "/mnt/log/hawkeyeslogs/hawkeyes.log",
           "date" => "2015-06-03 16:40:30,356",
    "thread_name" => "DefaultQuartzScheduler_Worker-9",
      "log_level" => "ERROR",
        "content" => "StackTrace  - Full Stack Trace:\norg.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only\nhawkeyes.logat hawkeyes.quartz.JobTemplate.execute(JobTemplate.groovy:10)\nhawkeyes.logat org.quartz.core.JobRunShell.run(JobRunShell.java:202)\nhawkeyes.logat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)"

Here is my grok config:

        grok {
            match => { "message" => 
                "%{TIMESTAMP_ISO8601:date} \[(?<thread_name>.+?)\] (?<log_level>\w+)\s*(?<content>.*)"
            }
        }

In logstash 1.4, the content variable will only get the first line content("StackTrace - Full Stack Trace:"). But in logstash 1.5, this variable will hold the whole multiline message.

How do I make this variable only the first line?

2

I solved this issue by myself. I find the filter mutate can solve my issue.

Here is my config:

filter{
   mutate {
       gsub => [
           "content", "\n.*", ""
       ]
   }
}
1 Like
4

But the grok in logstash 1.4.2 could not match the multiline. I used logstash since 1.4.2. So I find that the grok in 1.5.0 could match multiline.

5

@abcfy2 AFAIR grok in logstash 1.4.2 supports multiline matches. Do you have manual page where it says otherwise ? Have you tried to add '(?m)' at beginning of your match ? Just like suggested here in first response.

6

Oh sorry. I don't know about ?m. This grok regex in 1.4.2 could not match the multiline, so I thought grok could not match multiline. Got the new tech.

7

I'm using Grok in 1.4.2 with multiline codec, works fine. You can test your Grok pattern with multiline context in this online testing tool.

1 Like
8

If use Grok 1.5.x, and want to let it work like Grok 1.4.x, you can add (?-m) before your pattern, disable the default multiline function of 1.5.