JSON file processing - looking for direction

Elastic Stack Logstash
1

To start, I have been successful procssing single JSON files with the following LS config file. I use the 'exec' command to cat the JSON file. Works fine.

The issue I have is that I want to be able to process 100 different JSON files in a directory and this method will not work.

I have attempted to shift to the 'file' input, but haven't had successful results. Basically, I cannot get it to work.

I'm looking to the community for direction as to the best way to process 'x' number of JSON files.

Thanks in advance.

Logstash configuration file:

input {

   #file {
         #path => "C:\ELK_Stack\logstash-5.6.3\br_configs\sampleD_embedded.json"
		 #tags => ["FileNet", "nmon"]
		#codec => "json"
		#}
        
    exec {
         command => "cat C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json"
		 codec => "json"
		 interval => 3600
		 tags => ["FileNet", "nmon"]
		 type => "nmon_output"
		}
}

filter {
json {
source => "message"
}

	mutate {
			add_field => [ "serverTZ", "US/Central"]
			replace => [ "@source_host", "%{host}" ]
		   }
		   
    date {
          match => [ "timestamp", "HH:mm:ss'T'dd-MMM-yyyy" ]
		  timezone => "America/Chicago"
		  target => "@timestamp"
         }

}

output {
stdout {
codec => "rubydebug"
}
#elasticsearch {
#hosts => "localhost:9200"
#index => "nmon-es-index"
#}

}

2

The issue I have is that I want to be able to process 100 different JSON files in a directory and this method will not work.

So what happens?

Have you tried using the json_lines codec instead of json for your exec input?

3

So, to start, logstash does nothing (just sits there) as if you're trying to process a file yet the file isn't there.

I have bounced back and forth between json and json_lines without success.

input {
   
   file {
         path => "C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json"
		 tags => ["FileNet", "nmon"]
		codec => "json_lines"
        sincedb_path => "NUL" 
        start_position => "beginning" 
		}
        
    #exec {
         #command => "cat C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json"
		 #codec => "json"
		 #interval => 3600
		 #tags => ["FileNet", "nmon"]
		 #type => "nmon_output"
		#}
}

filter {
    json {
	      source => "message"
		 }

But, from a run with 'debug' enabled it does look like the JSON file is read through, values notes, etc.

[2017-12-14T13:05:03,431][DEBUG][logstash.inputs.file ] _globbed_files: C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json: glob is: []
[2017-12-14T13:05:03,432][DEBUG][logstash.inputs.file ] _globbed_files: C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json: glob is: ["C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json"] because glob did not work
[2017-12-14T13:05:03,435][DEBUG][logstash.inputs.file ] _discover_file: C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json: new: C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json (exclude is [])
[2017-12-14T13:05:03,449][DEBUG][logstash.inputs.file ] _open_file: C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json: opening
[2017-12-14T13:05:03,455][DEBUG][logstash.inputs.file ] C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json: initial create, no sincedb, seeking to beginning of file
[2017-12-14T13:05:03,456][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>"["}
[2017-12-14T13:05:03,460][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>"{"}
[2017-12-14T13:05:03,462][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>""timestamp": "00:05:32T24-MAR-2016","}
[2017-12-14T13:05:03,464][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>""filename": "sampleD.nmon","}
[2017-12-14T13:05:03,465][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>""host": "sampleN","}
[2017-12-14T13:05:03,466][DEBUG][logstash.inputs.file ] Received line {:path=>"C:\ELK_Stack\logstash-6.0.0\br_configs\sampleD_embedded.json", :text=>""date": "24-MAR-2016","}

If that is the case, is it just that I don't have the remainder of the LS configuration correct?

Thus, am I using the incorrect filter, etc.?

Thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.