JSON File Input

Elastic Stack Logstash
1

Hi,
I am just trying to take in a JSON file and spit it out to the stdout. The pipeline starts and nothing happens after that. There are no error messages and no other movements. I have the following in my configuration file:

input {
file {
type => "json"
path => "/opt/bin/samplejson_single.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
json {
source => "message"
}
}

output {
stdout {}
}

2

Please show the command you are running and any applicable logs/output.

3

logstash_screenshot
logstash_screenshot.PNG1185×205 30.5 KB

The contents of samplejson_single.json are:

{"prodId":"00583BE8-BE33-E011-BF97-001018721196","vendorName":"Omega","vendorId":"212770E7-E633-E011-BF97-001018721196","model":"DishStar®","categoryName":"Blender, Food, Countertop","fob":{"city":"Ft. Lauderdale","state":"FL","zip":"33331"},"spec":"Omega Blender, 48 oz. Tritan coopolyester BPA-Free container, toggle control with high/low speed, heavy duty locking ring, recessed coupling system, removable stainless steel blade assembly, silver, 1 HP, 600 watts, CE, ETL","listPrice":164.0,"netPrice":0.0,"numPerCase":1,"sellingUnit":"ea","shippingCube":0.0,"shippingWeight":7.0,"freightClass":"85","height":15.5,"width":8.0,"depth":7.0,"catalogItem":true,"multiplier":0,"hideModel":false,"callForPricing":false,"pictureId":"9B6AC60C-CF96-4858-BF6E-02A0F91E2FAA","cutSheetId":"D01FB5BA-A9E0-4B43-B4D4-568FED71A2BD","hasConfig":true,"configAccessories":"ZZOMEGASHIP1;ZZWARRANTY03;ZZOMEGAVLT02A01;ZZOMEGAVLT02A02A;","values":[{"property":"Motor HP","value":" 1 HP"},{"property":"Motor speed","value":"2-speed"},{"property":"Exterior Finish","value":"polycarbonate"},{"property":"Capacity","value":" 32 - 95 oz (1 - 2 qt)"},{"property":"Blender Type","value":"removable plastic container"}],"certifications":[{"text":"CE"},{"text":"ETL"}],"utils":{"electrical":[{"order":1,"amps":"","kw":".6","hp":"1","mca":"","mocp":"","voltage":"","cycle":"","nema":"","remarks":"","phase":"*","electricalConnectionHeight":"","connectionType":""}]},"changeState":"added"}

4

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

5

Sorry about that. I am new to discussion forums. Initially I was talking about Ubuntu Linux which I was running on my Virtual Machine (Virtualbox). Unfortunately, that has crashed. I have now installed Logstash on my original Windows environment. Following are the details of my trial:

In Windows Powershell, I ran this command:
.\bin\logstash -f grok_trial.conf

I got the following output:
Sending Logstash's logs to C:/Users/prati/Desktop/ELK/logstash/logs which is now configured via log4j2.properties
[2017-09-19T11:10:26,223][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Users/prati/Desktop/ELK/logstash/modules/fb_apache/configuration"}
[2017-09-19T11:10:26,230][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/prati/Desktop/ELK/logstash/modules/netflow/configuration"}
[2017-09-19T11:10:26,712][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-09-19T11:10:27,046][INFO ][logstash.pipeline ] Pipeline main started
[2017-09-19T11:10:27,177][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

The contents of grok_trial.conf are:
input {
file {
path => ["C:\Users\prati\Desktop\CE_Training\singlejson.json"]
start_position => "beginning"
type => "json"
}
}
output {
stdout {
codec => "rubydebug"
}
}

The contents of the file singlejson.json are:
{"prodId":"00583BE8-BE33-E011-BF97-001018721196","vendorName":"Omega","vendorId":"212770E7-E633-E011-BF97-001018721196","model":"DishStar®","categoryName":"Blender, Food, Countertop","fob":{"city":"Ft. Lauderdale","state":"FL","zip":"33331"},"spec":"Omega Blender, 48 oz. Tritan coopolyester BPA-Free container, toggle control with high/low speed, heavy duty locking ring, recessed coupling system, removable stainless steel blade assembly, silver, 1 HP, 600 watts, CE, ETL","listPrice":164.0,"netPrice":0.0,"numPerCase":1,"sellingUnit":"ea","shippingCube":0.0,"shippingWeight":7.0,"freightClass":"85","height":15.5,"width":8.0,"depth":7.0,"catalogItem":true,"multiplier":0,"hideModel":false,"callForPricing":false,"pictureId":"9B6AC60C-CF96-4858-BF6E-02A0F91E2FAA","cutSheetId":"D01FB5BA-A9E0-4B43-B4D4-568FED71A2BD","hasConfig":true,"configAccessories":"ZZOMEGASHIP1;ZZWARRANTY03;ZZOMEGAVLT02A01;ZZOMEGAVLT02A02A;","values":[{"property":"Motor HP","value":" 1 HP"},{"property":"Motor speed","value":"2-speed"},{"property":"Exterior Finish","value":"polycarbonate"},{"property":"Capacity","value":" 32 - 95 oz (1 - 2 qt)"},{"property":"Blender Type","value":"removable plastic container"}],"certifications":[{"text":"CE"},{"text":"ETL"}],"utils":{"electrical":[{"order":1,"amps":"","kw":".6","hp":"1","mca":"","mocp":"","voltage":"","cycle":"","nema":"","remarks":"","phase":"*","electricalConnectionHeight":"","connectionType":""}]},"changeState":"added"}

6

That all looks fine.
If you have processed the file before then you are likely running into sincedb being aware of that and not processing the file, even with the before directive. You may need to set the sincedb path to NUL to process the file.

7

Thanks, I fixed the problem by adding a sensible path for 'sincedb_path' which logstash has permission to write to. Also, we need to make sure that the input file has an extra blank line at the end of the file. I don't know why this needs to be done (maybe Mark you can throw light on this) but it started working just fine when I did.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.