Pipeline starts but data does not ingest - json or xml files

Elastic Stack Logstash
1

I'm using API call to pull data into json docs and ingesting through logstash, but when I go to run my config file, I get this in log stash

[2017-09-15T16:08:02,579][INFO ][logstash.pipeline ] Pipeline main started
[2017-09-15T16:08:02,658][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

No actual data is ingested however and an index is not created.

Same thing happens with JSON or XML.

I have no clue what I'm doing wrong because the logs do not show an error.

2

What does your configuration look like?

3

input {
file {
path => "/Users/S.AWDRSRV/Downloads/MDMCSV/json/*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
#filters not added yet
}
}
output {
stdout {codec => rubydebug }
elasticsearch {
hosts => ["http://localhost:9200"]
index => "test"
}
}

4

What if you comment out the elasticsearch output and focus on the stdout output? Are you getting anything? Are you sure Logstash has read access to the JSON files?

5

Even I am facing the same issue. My logstash.conf looks like this

input {
file {
path => "C:/logs/msglogin/*.log"
start_position => "beginning"
sincedb_path => "null"
codec=>"json"
}
}
filter {
json {
source => "message"
}
}
output {
stdout {
codec => rubydebug
}
}

and json file data is
{"_index": "customer", "_type": "product", "_id": 1}

6

@Samireddy_Keerthi, please start a separate thread for your problem.

7

If I comment it out, will it still pipe to the index? How can I check its read permissions? I run it as admin.

8

If I comment it out, will it still pipe to the index?

No, and that's the whole point. We want to simplify the configuration to make it easier to debug it.

How can I check its read permissions? I run it as admin.

Make sure the file itself is readable to the user that runs Logstash and that all directories leading up to the directory are at least executable to that user. The last directory, /Users/S.AWDRSRV/Downloads/MDMCSV/json, also needs to be readable.

You can increase Logstash's logging verbosity (there's a command line option for it) and it'll log additional details about what it's doing, including what filename patterns are expanded to. If your filename pattern expands to nothing that usually means that it doesn't have sufficient permissions.

9

this is the main output that I get:

Successfully started Logstash API endpoint {:port=>9600}
[2017-10-05T17:06:11,384][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162
[2017-10-05T17:06:12,388][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162
[2017-10-05T17:06:13,391][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162
[2017-10-05T17:06:14,396][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162
[2017-10-05T17:06:15,370][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-10-05T17:06:15,400][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162
[2017-10-05T17:06:16,402][DEBUG][logstash.inputs.file ] each: file grew: /Users/S.AWDRSRV/Downloads/MDMCSV/json/AWoutput.json: old size 0, new size 24162

It just loops this repeatedly and never creates the index.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.