# Logstash not giving the expected output

**URL:** <https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034>\
**Category:** Logstash\
**Created:** [February 3, 2025, 10:40pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034 "2025-02-03T22:40:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 3, 2025, 10:40pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/1 "2025-02-03T22:40:22Z")

</div>

I am trying to parse my sample catalina.out file.  
here is my tomcat.conf file i am giving to the logstash  
` ./logstash -f conf.d/tomcat.conf -r`

here is my tomcat.conf

```auto
input {
  file {
    path => "/elkstack/logs/input/catalina-sample.out"
    codec => multiline {
      pattern => "^\[<\d{1,2}"
      negate => "true"
      what => "previous"
      }
    }
  }

filter {
  grok {
    match => { "message" => "%{GREEDYDATA:message}" }
  }

output {
    stdout{}
  }

```

i am not seeing any parsing or any results. what am i doing wrong?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 3, 2025, 11:17pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/2 "2025-02-03T23:17:43Z")

</div>

> [@nkknkk](#):
>
> i am not seeing any parsing or any results. what am i doing wrong?

What is the result you are getting? Please share the output from Logstash.

Also, your grok filter is doing nothing, so your output will be the same as the original message in your log file.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 3, 2025, 11:26pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/3 "2025-02-03T23:26:04Z")

</div>

And if the file has been read already, it will not be reread unless you set

` sincedb_path => "/dev/null"`

in the input file config.

```auto
input {
  file {
    path => "/elkstack/logs/input/catalina-sample.out"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^\[<\d{1,2}"
      negate => "true"
      what => "previous"
      }
    }
  }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 4, 2025, 3:09am UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/4 "2025-02-04T03:09:12Z")

</div>

Also, you may need to set [auto\_flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto_flush_interval) on the codec.

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 5:50pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/5 "2025-02-10T17:50:58Z")

</div>

thank you Stephen. This solution seems to be working for a static file. however, I have the log file generated dynamically and the log file gets deleted after every restart of the tomcat.

now, the parsing rules are only getting the lines that starts with `d{1,2}`. it is completely ignoring the other lines that starts with `[<`. I have no clue why is this behavior?

This is how my config file looks like.

```auto
# cat elkstack.conf
input {
  file {
    path => "/opt/app/logs/catalina.out"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^(\[<\d{1,2}|\d{2}-\w{3}-\d{4})"
      negate => "true"
      what => "previous"
      auto_flush_interval => 5
    }
  }
}

filter {
  mutate {
    remove_field => ["host", "@version", "event"]
  }
}

output {
  elasticsearch {
    hosts => ["http://elkstack:9200"]
    index => "dev01"
     }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 6:03pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/6 "2025-02-10T18:03:47Z")

</div>

Are you sure the generated files are the same patterns ... post a sample (anonymize) more often than not the `pattern` is incorrect

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 6:13pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/7 "2025-02-10T18:13:20Z")

</div>

here is the sample log

```auto
[<2 10, 2025 11:17:15 AM>:
Yr=2025
]
[<2 10, 2025 11:17:15 AM>: CurrentYear: close statement]
10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
[<2 10, 2025 11:18:15 AM>:
Yr=2025
]
[<2 10, 2025 11:18:15 AM>: CurrentYear: close statement]
10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 6:23pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/8 "2025-02-10T18:23:59Z")

</div>

And why not just

```
  pattern => "^(\[<|\d{2}-\w{3}-\d{4})"

```

According to [regex101.com](http://regex101.com) that should work... I see you have single line as well... hmm... perhaps that is the issue

I will try on my side...

Can you supply a sample that has both types please...

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 6:53pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/9 "2025-02-10T18:53:43Z")

</div>

Ok This ...

```auto
    codec => multiline {
      pattern => "^\["
      negate => "true"
      what => "previous"
    }

```

Works for these

```auto
[<2 10, 2025 11:17:15 AM>:
Yr=2025
]
[<2 10, 2025 11:17:15 AM>: CurrentYear: close statement]
10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
[<2 10, 2025 11:18:15 AM>:
Yr=2024
]
[<2 10, 2025 11:18:15 AM>: CurrentYear: close statement]
11-Feb-2025 11:17:09.201 ERROR [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
]
[<2 10, 2025 11:18:15 AM>:
Yr=2024
]

```

Output

```auto
{
    "@timestamp" => 2025-02-10T18:53:18.165960Z,
       "message" => "[<2 10, 2025 11:18:15 AM>: CurrentYear: close statement]\n11-Feb-2025 11:17:09.201 ERROR [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference\n java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T18:53:18.165458Z,
       "message" => "[<2 10, 2025 11:18:15 AM>:\nYr=2024\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T18:53:18.165005Z,
       "message" => "[<2 10, 2025 11:17:15 AM>: CurrentYear: close statement]\n10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference\n java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T18:53:18.164136Z,
       "message" => "[<2 10, 2025 11:17:15 AM>:\nYr=2025\n]",
      "@version" => "1"
}

```

Now provide a mixed file or is this it

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:27pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/10 "2025-02-10T20:27:27Z")

</div>

that is the mixed sample i gave you.  
the first message should only be until `: close statement]`  
message 2 should start with `11-feb-2025`

the pattern i gave in the above response works perfectly on a static file.  
its just not doing the same thing on a dynamic log file that gets deleted and populated with logs often.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 8:29pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/11 "2025-02-10T20:29:23Z")

</div>

Ahh OK... let me check...

OK so this worked for me note missing the `(`  
so it says

Starts with this `^\[` or starts with `^\d{1,2}|\d{2}-\w{3}-\d{4}`

```auto
    codec => multiline {
      pattern => "^\[|^\d{1,2}|\d{2}-\w{3}-\d{4}"
      negate => "true"
      what => "previous"
    }

```

Input

```auto
[<2 10, 2025 11:17:15 AM>:
Yr=2025
]
[<2 10, 2025 11:17:15 AM>: CurrentYear: close statement]
10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
[<2 10, 2025 11:18:15 AM>:
Yr=2024
]
[<2 10, 2025 11:18:15 AM>: CurrentYear: close statement]
11-Feb-2025 11:17:09.201 ERROR [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference
        java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
]
[<2 10, 2025 11:18:15 AM>:
Yr=2024
]

```

Output

```auto
{
    "@timestamp" => 2025-02-10T20:31:35.653999Z,
       "message" => "[<2 10, 2025 11:18:15 AM>:\nYr=2024\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.288355Z,
       "message" => "[<2 10, 2025 11:17:15 AM>:\nYr=2025\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.290290Z,
       "message" => "11-Feb-2025 11:17:09.201 ERROR [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference\n java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.289482Z,
       "message" => "[<2 10, 2025 11:18:15 AM>:\nYr=2024\n]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.289715Z,
       "message" => "[<2 10, 2025 11:18:15 AM>: CurrentYear: close statement]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.288636Z,
       "message" => "[<2 10, 2025 11:17:15 AM>: CurrentYear: close statement]",
      "@version" => "1"
}
{
    "@timestamp" => 2025-02-10T20:31:36.289027Z,
       "message" => "10-Feb-2025 11:17:09.201 WARNING [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference\n java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection",
      "@version" => "1"
}

```

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:34pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/12 "2025-02-10T20:34:02Z")

</div>

ok, let me try by removing `( )`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 8:34pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/13 "2025-02-10T20:34:33Z")

</div>

Try exactly what I have....

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:36pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/14 "2025-02-10T20:36:15Z")

</div>

I can't have exactly what you have. because i have messages that have `[` in the middle and i don't want to break the line there. that is the reason i had `[<d{1,2}`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 8:38pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/15 "2025-02-10T20:38:55Z")

</div>

I Don't understand .... '  
`^` means anchor at the beginning... that is regex

`^\[`means **only** `[` **at the beginning of the line**... so a `[` in the middle will not be matched that is the whole point

You can see it worked that way

note the `[` in the middle of the line

```auto
{
    "@timestamp" => 2025-02-10T20:31:36.290290Z,
       "message" => "11-Feb-2025 11:17:09.201 ERROR [main] org.apache.naming.NamingContext.lookup Unexpected exception resolving reference\n java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection\n]",
      "@version" => "1"
}

```

and like wise

`^\d{1,2}|\d{2}-\w{3}-\d{4}"` means only at the beginning of the line

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:42pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/16 "2025-02-10T20:42:06Z")

</div>

oh right ok. my bad. I can try exactly what you have

`"^\[|^\d{1,2}|\d{2}-\w{3}-\d{4}"`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 8:47pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/17 "2025-02-10T20:47:37Z")

</div>

And I think you really only need

` pattern => "^\[|^\d{1,2}-\w{3}-\d{4}"`

because `d{1,2}` matches 1 or 2 digits

I tried with

```auto
11-Feb-2025 
9-Feb-2025 

```

Both matched use this to test and then I test with logstash... both verified

> **[regex101: build, test, and debug regex](https://regex101.com/)**
>
> Regular expression tester with syntax highlighting, explanation, cheat sheet for PHP/PCRE, Python, GO, JavaScript, Java, C#/.NET, Rust.

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:54pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/18 "2025-02-10T20:54:14Z")

</div>

well it works with the static file. I will have to wait for tomorrow to see what happens.

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [February 10, 2025, 8:56pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/19 "2025-02-10T20:56:32Z")

</div>

This is what is happening like I mentioned before. every day we have a deployment and a server restart. when we do that, the old log file gets deleted and new log file with the same name and format will be recreated with the new content. this is where the problem is happening. i will have to wait for tomorrow and see if this works

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 10, 2025, 9:01pm UTC](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034/20 "2025-02-10T21:01:55Z")

</div>

Sounds good if works on this static file but not on a new file...

Either the file is not the same or somehow you logstash config is altered or something...

You will need to show exactly then perhaps we can help.

IF that is the case I would say open a new thread with an exact subject and very much details...

[Next page](https://discuss.elastic.co/t/logstash-not-giving-the-expected-output/374034.md?page=2)