How to Filebeat and logstash working with a JSON file

[thewasta (Schenier Lopez)]

Filebeat, Logstash version: 7.10

Given this json

{
  "killStreakData": {
    "12344412441": {
      "Name": "nickName",
      "highestKS": 2
    },
    "53134441634": {
      "Name": "nickName2",
      "highestKS": 3
    }
  }
}

I would like to know how to stream (tail) JSON file, where "highestKS" can increase/change and a new object inside "killStreakData" could also be added. I want to output to elasticsearch like this:

userid: 12344412441
nick: "nickName"
kills: 2

I have tried configuration

File: /etc/logstash/conf.d/data-example.conf

input {
  file {
    path => "/home/userxample/example-log/kills_stats.json",
    start_position => "beginning"
    codec => "json"
  }
}

output {
  stdout { codec => rubydebug }
}

File: /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/userxample/example-log/kills_stats.json

And run this command /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/data-example.conf --path.settings=/etc/logstash and return error

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-82682/jruby383258714223377497jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2021-02-04T09:44:17,764][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2021-02-04T09:44:18,284][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-04T09:44:19,124][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 3, column 67 (byte 84) after input {\n  file {\n    path => \"/home/userxample/example-log/kills_stats.json\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2021-02-04T09:44:19,351][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-04T09:44:24,410][INFO ][logstash.runner          ] Logstash shut down.
[2021-02-04T09:44:24,420][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I don't really know at all how logstash works with JSON, but like I said "highestKS" value will change, will this be inserted as a new value in my document? Or does it update the current value?

Thanks