I am trying to index AWS CloudTrail logs in elastic search using the file input plugin for logstash.
The format of the JSON file it is reading is multiline events of json, separated by new line. From my understanding for this reason I should be using json_lines codec.
Here is input/output configuration:
path => "/home/centos/cloudtrailproject/cloudtraillog.json"
codec => "json_lines"
hosts => [ "localhost:9200" ]
index => "cloudtrail"
user = > omitted
password => omitted
I can't really tell if the input plugin is running or not, but I am assuming it indexes any time new info is appended to the file. I am not seeing any errors in the logstash logs. Also not seeing data in the index regardless of time filter. I even tried creating the index in kibana without a time filter and it still did not show any data so my assumption that the data is not making it into elasticsearch (however, the index is created in elasticsearch and in kibana you can see all of the fields that would be indexed if the data were displaying). I also made sure the service logstash is running as has read permissions on the file.
Can anyone point my in the right direction here or have any idea of what might be going wrong?