Logstash won't read file?

Elastic Stack Logstash
1

I'm playing with the some logs files I ..... found on the internet, which comes in JSON files. So I'm using LS to ingest them and send them to ES. But LS won't read them and I'm having a really hard time diagnosing the problem.

My conf file:

input {
#       stdin { codec => json }

    file {
            path => "/opt/BOTS/botsv1.iis.json"
            sincedb_path => "/dev/null"
            codec => json
    }
}

output {

    stdout { codec => rubydebug }

}

The only lines in logstash.yml that are active:

path.data: /var/lib/logstash
log.level: debug
path.logs: /var/log/logstash

Everything else is commented out.

When I run

/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/testtest.conf

nothing prints to the screen.

The file has these permissions:

-rwxr-xr-x. 1 logstash logstash 27M Jun 8 18:52 /opt/BOTS/botsv1.iis.json

Here is the log file:

[2018-06-13T19:24:24,094][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-06-13T19:24:24,102][DEBUG][logstash.inputs.file     ] _globbed_files: /opt/BOTS/botsv1.iis.json: glob is: ["/opt/BOTS/botsv1.iis.json"]
[2018-06-13T19:24:24,103][DEBUG][logstash.inputs.file     ] _discover_file: /opt/BOTS/botsv1.iis.json: new: /opt/BOTS/botsv1.iis.json (exclude is [])
[2018-06-13T19:24:24,196][DEBUG][logstash.inputs.file     ] _open_file: /opt/BOTS/botsv1.iis.json: opening
[2018-06-13T19:24:24,197][DEBUG][logstash.inputs.file     ] /opt/BOTS/botsv1.iis.json: initial create, no sincedb, seeking to end 28215738
[2018-06-13T19:24:24,199][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:25,201][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:26,203][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:27,206][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:27,977][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-06-13T19:24:27,979][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-06-13T19:24:28,209][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:29,097][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x15f64d55@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-06-13T19:24:29,211][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:30,214][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:31,215][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:32,218][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
--> [2018-06-13T19:24:32,990][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-06-13T19:24:32,991][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-06-13T19:24:33,220][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:34,099][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x15f64d55@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-06-13T19:24:34,221][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:35,224][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:36,227][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738
[2018-06-13T19:24:37,229][DEBUG][logstash.inputs.file     ] each: file grew: /opt/BOTS/botsv1.iis.json: old size 0, new size 28215738

The part from the --> (my addition) repeats over and over again until I stop LS. But no data ever reads.

Does this mean that LS has marked the file as read and therefore won't reread it?
I was told that using sincedb_path => "/dev/null" would allow logstash to re-read the file over and over again, but it doesn't seem to be working.

Suggestions? Am I missing something? Did I misconfigure?

Thanks

(running CentOS 7 with 8 GB RAM & Quad core i3 processor.)

2

I was told that using sincedb_path => "/dev/null" would allow logstash to re-read the file over and over again, but it doesn't seem to be working.

You also need start_position => "beginning".

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.