Logstash big files

[txisme]

Hey, recently i sent to logstash with filebeat a 5GB log text data and it just broke logstash. Is there any problem with sending big files at once?

[eperry (Ed)]

How did it break logstash?

Logs of logstash, error message and maybe your filebeat and logstash config would help diagnose it.

[txisme]

Logstash and filebeat work perfect if im sending relatively small logs.

There are no logs messages, it simply stops working after processing some logs of that file
Only way to fix is to stop filebeat to harvest that specific file and restart logstash

[txisme]

Is it possible that logstash or filebeat simply are on hold while they process the big file? And that if i wait a long time everything will appear?

EDIT: after 30min of waiting i notice that every index is getting bigger but at a very very slow rate. Less than 1000 docs were indexed.

[eperry (Ed)]

no it would have to be the config, since filebeat and logstash break files in to lines you could have infinate size files

A size of the file makes no difference it is what filebeat and logstash are doing with it.

For exxample an misconfigured multiline will eat up heap space. and you should have OOM errors in logstash

filebeat I am less familar with but I would imagine as fast as it can process the file and ship it off that is all

but this is all speculation without seeing logs and configs.

there is no known issue with log sizes I personally send 100G files with no problem.

[txisme]

Filebeat 1.3
Logstash 1.4 EDIT: 2.4

Filebeat prospector config:

    -
  paths:
    - "/root/tracy/logs/*.out"
  input_type: log

  fields:
    log_type: tracy

  multiline:
    pattern: '^Offset'
    negate: true
    match: after

Filebeat Log:
It doesnt even create a log file

Logstash config:

input {
    beats{
            port => 5044
    }
}
filter{
 if([fields][log_type]=="tracy"){
            grok{
                    match => {"message" => "(?m)%{WORD:check} = %{NUMBER:offset2:int}, %{WORD} = %{WORD:topic2}, %{WORD} = %{WORD:source2}, %{WORD} = %{WORD:type2}, %{WORD} = %{GREEDYDATA:value2}"}
            }
    }
}
output {
     if([fields][log_type]=="tracy"){
            elasticsearch{
                    hosts => [ "localhost:9200" ]
                    index => [ "tracy" ]
            }
}

Logstash Log:

Sending logstash logs to /var/log/logstash/logstash.log.
{:timestamp=>"2016-11-17T00:11:50.044000+0000", :message=>"Pipeline main started"}

[txisme]

Ok, i found the problem.

I was sending the output to ElasticSearch and to a fileoutput, it was that that was slowing down everything

[Christian_Dahlqvist]

I would recommend upgrading to a more recent Logstash release as 1.4 is almost anchient.

[txisme]

Sorry, i meant 2.4

[Christian_Dahlqvist]

That is much better.