Logstash only ingesting data when shutdown/closing files

Elastic Stack Logstash
1

My logstash config file seems to running successfully, however, it doesn't pull in any data. It isn't until I manually shutdown the process using 'Ctrl+C', that it pulls in data. This doesn't seem to normal behavior. Is this so? And if not how can I correct this?

P.S. I get no errors.

2

That would happen if the input expects a separator between events (e.g. a tcp input looking for newline) and it never sees the separator. Or a multiline codec that never sees a line that causes it to flush an event. What is your input configuration?

3

The input configuration is

input {
file {
path => "/data/xmlFiles/*.xml"
sincedb_path => "/dev/null"
start_position => "beginning"
type => "xml"
codec => multiline {
pattern => "^()"
negate => true
what => previous
max_lines => 105000
}
}
}

I assume I'm missing the line "auto_flush_interval => 1" in the codec from what it sounds like.

4

Yes, if you are trying to consume an entire file as a single event you need that.

5

It's more like multiple files in a single event. The Auto_flush didn't pull them in would it have to be a different value?

6

Sorry scratch that last comment. It did work there was a change that needed to be done elsewhere. A gsub to separate where the pattern was located, on the last line.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.