Logstash 7.9.1 file input plugin is not able to read json file

Elastic Stack Logstash
1

I have created sample.conf file for reading json file but it is not showing any output.
below is my sample.cong file

input {
	file {
		path => "C:\Users\Sagar\sample.json"
		start_position => "beginning"
		codec => "json"
	}
}
filter{
    json {
        source => "message"
    }
}
output {
	stdout { 
        codec => json 
    }
}

below is my json file, it is contains just one value for testing purpose:

{"nativeId": 1001}

Below is logstash log:

C:\Users\Sagar\ELK\logstash-7.9.1>.\bin\logstash.bat -f .\config\sample.conf
Sending Logstash logs to C:/Users/Sagar/ELK/logstash-7.9.1/logs which is now configured via log4j2.properties
[2021-02-17T15:59:33,433][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2021-02-17T15:59:33,689][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"C:/Users/Sagar/ELK/logstash-7.9.1/data/queue"}
[2021-02-17T15:59:33,704][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"C:/Users/Sagar/ELK/logstash-7.9.1/data/dead_letter_queue"}
[2021-02-17T15:59:33,914][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-17T15:59:33,996][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"2de0f081-6c80-4f62-a866-f5404b6d6f71", :path=>"C:/Users/Sagar/ELK/logstash-7.9.1/data/uuid"}
[2021-02-17T15:59:37,157][INFO ][org.reflections.Reflections] Reflections took 64 ms to scan 1 urls, producing 22 keys and 45 values
[2021-02-17T15:59:38,021][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/Sagar/ELK/logstash-7.9.1/config/sample.conf"], :thread=>"#<Thread:0x68e54919 run>"}
[2021-02-17T15:59:39,175][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.14}
[2021-02-17T15:59:40,174][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Users/Sagar/ELK/logstash-7.9.1/data/plugins/inputs/file/.sincedb_2241c9a4d00df54d36f52b07672d9ebc", :path=>["C:\\Users\\Sagar\\sample.json"]}
[2021-02-17T15:59:40,215][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-02-17T15:59:40,294][INFO ][filewatch.observingtail  ][main][65b2f32534e23b57e62a0d4c276d2cf74b9d94a1edf9b2f80db0699ef82c6301] START, creating Discoverer, Watch with file and sincedb collections
[2021-02-17T15:59:40,303][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-17T15:59:41,077][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Please let me know if i am missing anything here.

2

Try path => "C:/Users/Sagar/sample.json"

3
  • I have tried with path => "C:/Users/Sagar/sample.json" but is not working.
  • converted file to .txt but it is not working.
  • added mode => read it is not working

everytime i have deleted data folder and try different approach but not working.

4

Hi @sagarpatel

Most likely Logstash has already read the file once Logstash keeps track and so will not read it again... No matter how many times you run it. This is a common mistake when getting started

During debug and test you should probably set see here

sincedb_path => "/dev/null"

In the file input section.

Or you can erase the since DB data every time but the /dev/null setting is easier

Also you don't need the multilpe json parsing... For a file with ndjson take it out of the json codec in the file input and just use the json filter.

In the output the just use the rubydebug

output {
      stdout { codec => rubydebug }
    }
5

I have tried below configuration as i am running logstash 7.9.1 on windows. But it is still not printing anything on log or prompt. one more thing i have noticed that logstash 7.9.1 download page not showing OS based download link, that means is it supporting only Linux ? because i can see OS based download for 7.10.x version.

input {
	file {
		path => "C:/Users/Sagar/sample.txt"
		start_position => "beginning"
		sincedb_path => "NUL"
	}
}
filter{
    json {
        source => "message"
    }
}
output {
	stdout { codec => rubydebug }
}
6

Can you post your Logstash output again to see if anything changed?

7

@sagarpatel

To answer your question Yes Logstash 7.9 is supported on Windows Server 2016 and Windows Server 2019.

How are you starting logstash?

and as @aaron-nimocks asked can you please post your logstash output?

There is something basic going on... like a typo in the path perhaps... the config about should just read the file... is there more than 1 line in the file?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.