Logstash doesn't reads JSON file on Windows

[VSKMurali]

Hello,

I am trying to configure a following setup on Windows machine.

JSON file (creates every 5 mins with the same file name) and Elasticsearch should read the file and push to Index and this is my Logstash config file.

input {
  file {
    path => "D:\logs_folder\logs.json"
    start_position => "beginning"
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] index => "in_elk_test"}
  stdout { codec => rubydebug }
}

My JSON file would be like below (new line there after JSON message)
{"mode":"LIVE","user":2418,"platform":"IOS_Mobile"}

I have tried following file plugin also

  file {
    path => "D:\logs_folder\logs.json"
    start_position => "beginning"
    codec => "json"
    sincedb_path => " D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/data/plugins/inputs/file/null"
  }

Tried sincedb_path=>"nul" also

Logstash config stop at the line Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Command used to run Logstash config file
D:\elk\logstash-8.9.0-windows-x86_64\logstash-8.9.0\bin> .\logstash -f logstash-simple.conf

Config file name: logstash-simple.conf

Please help me on the logstash config for reading from file.

[Badger]

@VSKMurali

path => "D:\logs_folder\logs.json"

Do not use backslash in the path option of a file input, it is treated as an escape. You could use path => "D:/logs_folder/logs.json"

When you use backslash logstash ends up looking for "D:logs_folderlogs.json" in the root directory.

[VSKMurali]

Hello @Badger , thanks for the comment, I stuck at the same place "Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}" when I use "/"

  file {
    path => "D:/logs_folder/logs.json"
    start_position => "beginning"
  }

[Badger]

Try sincedb_path => "NUL"

If logstash successfully writes the contents of the file to elasticsearch then you should expect to see exactly what you get in the logstash log. It may be working.

You could try setting TRACE level logging and see what filewatch has to say.

[VSKMurali]

Hello @Badger I have tried with sincedb_path => "NUL", but still ending up with the same place.

Logs:
I have converted following curl to REST API and got the following result

CURL statement:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logger.filewatch.discoverer" : "TRACE",
    "logger.filewatch.observingtail" : "TRACE",
    "logger.filewatch.sincedbcollection" : "TRACE",
    "logger.filewatch.tailmode.handlers.createinitial" : "TRACE",
    "logger.filewatch.tailmode.handlers.grow" : "TRACE",
    "logger.filewatch.tailmode.processor" : "TRACE"
}

Result:

{
    "status": 500,
    "request_method": "PUT",
    "path_info": "/_node/logging",
    "query_string": "pretty",
    "http_version": "HTTP/1.1",
    "http_accept": "*/*",
    "error": "Unexpected Internal Error",
    "class": "LogStash::Json::ParserError",
    "message": "Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])\"{\"; line: 1, column: 1])\n at [Source: (byte[])\"{\"; line: 1, column: 2]",
    "backtrace": [
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/json.rb:32:in `jruby_load'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/api/modules/logging.rb:43:in `block in PUT /'",
        "org/jruby/RubyMethod.java:116:in `call'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `block in compile!'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block in route!'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1049:in `route_eval'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block in route!'",
        "D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1078:in `block in process_route'",
.............
]
}

[Badger]

I get that error if I leave out the final } of the JSON.

[VSKMurali]

Hello,

@VSKMurali

{"mode":"LIVE","user":2418,"platform":"IOS_Mobile"}

do you mean "}" in the last?
How can I fix that?

[Badger]

I was referring to the } after "logger.filewatch.tailmode.processor" : "TRACE"

[VSKMurali]

I have taken the curl expression from

Logstash when started using nohup, it is logging too much resulting in huge size

we are running logstash and its output has file and elasticsearch for every opening and closing the file plugin is logging in... resulting the nohup.out file being very huge... How do i avoid so much of logging, or is there a way better way to handle this?

How can I prevent "}", is it causing files not reading issue?

[Badger]

Apparently not. Try running with --log.level TRACE on the logstash command line instead.

[VSKMurali]

command used .\logstash -f logstash-simple.conf --log.level trace

Found following logs

[2023-08-20T04:54:21,373][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-08-20T04:54:21,679][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2023-08-20T04:54:21,680][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2023-08-20T04:54:24,079][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2023-08-20T04:54:24,108][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>1}
[2023-08-20T04:54:24,122][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2023-08-20T04:54:24,128][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::StopAndDelete/pipeline_id:main}
[2023-08-20T04:54:24,137][DEBUG][logstash.javapipeline    ] Closing inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x5c0b1644@D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2023-08-20T04:54:24,139][DEBUG][logstash.inputs.file     ] Stopping {:plugin=>"LogStash::Inputs::File"}
[2023-08-20T04:54:24,146][INFO ][filewatch.observingtail  ] QUIT - closing all files and shutting down.
[2023-08-20T04:54:24,147][DEBUG][logstash.javapipeline    ] Closed inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x5c0b1644@D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2023-08-20T04:54:24,410][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2023-08-20T04:54:24,664][TRACE][filewatch.sincedbcollection][main][a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36] caller requested sincedb write (tail mode subscribe complete - shutting down)
[2023-08-20T04:54:24,665][TRACE][filewatch.sincedbcollection][main][a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36] sincedb_write: NUL (time = 2023-08-20 04:54:24 +0530)
[2023-08-20T04:54:24,665][TRACE][filewatch.sincedbcollection][main][a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36] non_atomic_write:  {:time=>2023-08-20 04:54:24 +0530}
[2023-08-20T04:54:24,674][DEBUG][logstash.inputs.file     ][main][a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36] Closing {:plugin=>"LogStash::Inputs::File"}
[2023-08-20T04:54:24,681][DEBUG][logstash.pluginmetadata  ][main][a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36] Removing metadata for plugin a2a2eab1293da13e934e5dc003981e308db427ac8e61c2feeb0ce40dc00f1b36
[2023-08-20T04:54:24,688][DEBUG][logstash.javapipeline    ][main] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"main", :thread=>"#<Thread:0x5c0b1644@D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[

[2023-08-20T04:54:24,796][DEBUG][logstash.outputs.elasticsearch][main] Closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2023-08-20T04:54:24,798][DEBUG][logstash.outputs.elasticsearch][main] Stopping sniffer
[2023-08-20T04:54:24,798][DEBUG][logstash.outputs.elasticsearch][main] Stopping resurrectionist
[2023-08-20T04:54:25,235][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=972487998} post-append result (captures: `6` span: `PT25.0754648S` }
[2023-08-20T04:54:25,236][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=1738346506} post-append result (captures: `18` span: `PT1M25.2969402S` }
[2023-08-20T04:54:25,236][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_5_minutes id=1582722571} post-append result (captures: `25` span: `PT3M55.8568842S` }
[2023-08-20T04:54:25,236][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=996325092} post-append result (captures: `6` span: `PT25.074679S` }
[2023-08-20T04:54:25,237][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=1755963538} post-append result (captures: `18` span: `PT1M25.2960993S` }
[2023-08-20T04:54:25,237][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_5_minutes id=1056074974} post-append result (captures: `25` span: `PT3M55.8475895S` }
[2023-08-20T04:54:25,237][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=1668172458} post-append result (captures: `6` span: `PT25.0740602S` }
[2023-08-20T04:54:25,237][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=1820068749} post-append result (captures: `18` span: `PT1M25.2944415S` }
[2023-08-20T04:54:25,237][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_5_minutes id=1094733648} post-append result (captures: `25` span: `PT3M55.8476024S` }
[2023-08-20T04:54:25,240][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1213668314} post-append result (captures: `11` span: `PT3M54.854053S` }
[2023-08-20T04:54:25,241][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=273161937} post-append result (captures: `5` span: `PT20.0612887S` }
[2023-08-20T04:54:25,241][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=1531597356} post-append result (captures: `17` span: `PT1M20.2659609S` }
[2023-08-20T04:54:25,241][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1285758357} post-append result (captures: `11` span: `PT3M54.8538561S` }
[2023-08-20T04:54:25,752][DEBUG][logstash.outputs.elasticsearch][main] Waiting for in use manticore connections
[2023-08-20T04:54:25,756][DEBUG][logstash.outputs.elasticsearch][main] Closing adapter #<LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x3f32277e>
[

Some importance logs

[2023-08-20T04:54:24,139][DEBUG][logstash.inputs.file     ] Stopping {:plugin=>"LogStash::Inputs::File"}
[2023-08-20T04:54:24,146][INFO ][filewatch.observingtail  ] QUIT - closing all files and shutting down.
[2023-08-20T04:54:24,147][DEBUG][logstash.javapipeline    ] Closed inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x5c0b1644@D:/elk/logstash-8.9.0-windows-x86_64/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2023-08-20T04:54:24,410][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu