Logstash ('No configuration found in the configured sources')

matteo001 · March 1, 2021, 9:04am

Hi everyone, I tried to load the JSON file in elastic search through Logstash in windows10.

I show you my "MATTEOLOGSTASHCONF.conf" file:
input {
file {
start_position => "beginning"
path => "C:/Users/matte/Desktop/Magistrale/"2 ANNO"/TESI/query_2.json"
sincedb_path => "/dev/null"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
index => "Mqattep"
document_type => "doc_tty"
}
stdout{}
}

Following, this is my command line in cmd:
logstash -f C:/Users/matte/Desktop/TESI/logstash-7.11.1/config/MATTEOLOGSTASHCONF.conf

And this is what the program returns:
Using JAVA_HOME defined java: C:\Program Files\Java\jdk-12.0.1
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/Users/matte/Desktop/TESI/logstash-7.11.1/logs which is now configured via log4j2.properties
[2021-03-01T09:50:50,261][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 12.0.1+12 on 12.0.1+12 +indy +jit [mswin32-x86_64]"}
[2021-03-01T09:50:50,544][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-03-01T09:50:51,159][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-03-01T09:50:51,667][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-03-01T09:50:56,380][INFO ][logstash.runner ] Logstash shut down.
[2021-03-01T09:50:56,418][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.Users.matte.Desktop.TESI.logstash_minus_7_dot_11_dot_1.lib.bootstrap.environment.(C:\Users\matte\Desktop\TESI\logstash-7.11.1\lib\bootstrap\environment.rb:89) ~[?:?]

What are my errors?

Also, I tried to write the pipelines.yml:
pipeline.id: test
pipeline.workers: 1
pipeline.batch.size: 500
queue.type: persisted
queue.max_bytes: 1gb # Default = 1024mb
queue.drain: true
pipeline.output.workers: 1
path.config: "C:/Users/matte/Desktop/TESI/logstash-7.11.1/config/MATTEOLOGSTASHCONF.conf"

I run the command line both with pipelines.yml file and without it.

aaron-nimocks · March 1, 2021, 1:39pm

The error No configuration found in the configured sources. means it can't find the file you entered. I would verify that path.

Also on Windows it will be sincedb_path => "NUL" instead of sincedb_path => "/dev/null" but that is not your current problem.

matteo001 · March 1, 2021, 4:58pm

Thank you,

matteo001 · March 1, 2021, 4:58pm

The path on "2 ANNO" was wrong.

matteo001 · March 1, 2021, 6:39pm

@aaron-nimocks please, can you help me again?

I want to insert into ES (elasticsearch) the json file, NOT line for line but the datas of the json file.
Example of json:

[{"_id": {"$oid": "603bf573e462e7ea44be9432" }, "NUM_TEL": "89123732", "Assente": true},{ "_id": { "$oid": "603bf573e462e7ea44be9433" }, "NUM_TEL": "318731123", "Assente": true}]

When I load the json file in elasticsearch and i get wirh command: GET /mattep/_search,
it returns:
hit:{
[...]
"message" : " "$oid": "603bf573e462e7ea44be9432"",
[...]},
hit{[...]
"message" : " "$NUM_TEL": "89123732"",
[...]}

I tried to write different .conf files. It is the last file:

input {

file {
		type => "json"
		start_position => "beginning"
		path => "C:/Users/matte/Desktop/Magistrale/query_2.json"
		sincedb_path => "NUL"
}

}

filter { json { source => "message" } }

output {
elasticsearch {
hosts => "localhost:9200"
index => "mattep"
document_type => "json"
}

stdout {  codec => rubydebug  }

}

My goal is to put the data into Elasticsearch with the json file so that I can show it with Kibana.
I would to see, when I run "GET /MY_INDEX/_search [...]":

hit:{
[...]
"_id" : "603bf573e462e7ea44be9432",
"NUM_TEL": 2990129039,
[...]}

I've been reading every topic and document all afternoon.

Badger · March 1, 2021, 7:45pm

Use a multiline codec on the file input. There is an example here.

matteo001 · March 1, 2021, 8:18pm

I thank you tour reply but I don't understand it.
This is my conf file:
input {
file {
start_position => "beginning"
path => "C:/Users/matte/Desktop/Magistrale/query_2.json"
sincedb_path => "NUL"
codec => multiline { pattern => "^}" negate => true what => next
auto_flush_interval => 1 }
}
}
filter { json { source => "message" } }
output {
elasticsearch {
hosts => "localhost:9200"
index => "matte0_jud"
document_type => "json"
}
stdout { codec => rubydebug }
}

I m sorry if I tell you some bulllshit (if it is the term in english) .
SO, what is the problem in my conf file?

Badger · March 1, 2021, 8:26pm

I have no idea. What do you not like about the result?

matteo001 · March 2, 2021, 3:42pm

This not good because I Have:
[...]
"hits" : [
{
"_index" : "mattep",
"_type" : "json",
"_id" : "uanr7ncB22tqv8EEWKTF",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.506Z",
"@version" : "1",
"type" : "json",
"message" : " "$oid": "603bf573e462e7ea44be9432"",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
{
"_index" : "mattep",
"_type" : "json",
"_id" : "u6nr7ncB22tqv8EEWKTF",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.509Z",
"@version" : "1",
"type" : "json",
"message" : "},{",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
{
"_index" : "mattep",
"_type" : "json",
"_id" : "Iqnr7ncB22tqv8EEWKbG",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.646Z",
"@version" : "1",
"type" : "json",
"message" : " "$oid": "603bf574e462e7ea44be945f"",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
[....]

Instead, I want to have NO MESSAGE
"message" : "$_id": "603bf573e462e7ea44be9432"",

but I want:
{
'_id': "603bf573e462e7ea44be9432",
'NUM_TEL' : '089328192123',
...}

Did you understand?
In a nutshell, I would like to insert a list of KEY: VALUE in elasticsearch by giving it a json file as input.
When I do a get I would like to see the data as JSON output in ES, not have the json unpacked in rows, the json must be inserted into the ES data.

matteo001 · March 2, 2021, 5:48pm

I'd like to insert the value in ES so, i can shoe it with get command.
Example:
JSon datas:

I want it in ES, so:
.

After, I 'd like to show these results in kibana canvas (and also here i have a few problems)

I don't understand what is my problem! .conf file? Json file? ....????

Thank you for your responses in advance.

system · March 30, 2021, 5:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.