Logstash ('No configuration found in the configured sources')

matteo001 · March 1, 2021, 9:04am

Hi everyone, I tried to load the JSON file in elastic search through Logstash in windows10.

I show you my "MATTEOLOGSTASHCONF.conf" file:
input {
file {
start_position => "beginning"
path => "C:/Users/matte/Desktop/Magistrale/"2 ANNO"/TESI/query_2.json"
sincedb_path => "/dev/null"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
index => "Mqattep"
document_type => "doc_tty"
}
stdout{}
}

Following, this is my command line in cmd:
logstash -f C:/Users/matte/Desktop/TESI/logstash-7.11.1/config/MATTEOLOGSTASHCONF.conf

And this is what the program returns:
Using JAVA_HOME defined java: C:\Program Files\Java\jdk-12.0.1
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/Users/matte/Desktop/TESI/logstash-7.11.1/logs which is now configured via log4j2.properties
[2021-03-01T09:50:50,261][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 12.0.1+12 on 12.0.1+12 +indy +jit [mswin32-x86_64]"}
[2021-03-01T09:50:50,544][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-03-01T09:50:51,159][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-03-01T09:50:51,667][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-03-01T09:50:56,380][INFO ][logstash.runner ] Logstash shut down.
[2021-03-01T09:50:56,418][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.Users.matte.Desktop.TESI.logstash_minus_7_dot_11_dot_1.lib.bootstrap.environment.(C:\Users\matte\Desktop\TESI\logstash-7.11.1\lib\bootstrap\environment.rb:89) ~[?:?]

What are my errors?

Also, I tried to write the pipelines.yml:
pipeline.id: test
pipeline.workers: 1
pipeline.batch.size: 500
queue.type: persisted
queue.max_bytes: 1gb # Default = 1024mb
queue.drain: true
pipeline.output.workers: 1
path.config: "C:/Users/matte/Desktop/TESI/logstash-7.11.1/config/MATTEOLOGSTASHCONF.conf"

I run the command line both with pipelines.yml file and without it.

aaron-nimocks · March 1, 2021, 1:39pm

The error No configuration found in the configured sources. means it can't find the file you entered. I would verify that path.

Also on Windows it will be sincedb_path => "NUL" instead of sincedb_path => "/dev/null" but that is not your current problem.

matteo001 · March 1, 2021, 4:58pm

Thank you,

matteo001 · March 1, 2021, 4:58pm

The path on "2 ANNO" was wrong.

matteo001 · March 1, 2021, 6:39pm

@aaron-nimocks please, can you help me again?

I want to insert into ES (elasticsearch) the json file, NOT line for line but the datas of the json file.
Example of json:

[{"_id": {"$oid": "603bf573e462e7ea44be9432" }, "NUM_TEL": "89123732", "Assente": true},{ "_id": { "$oid": "603bf573e462e7ea44be9433" }, "NUM_TEL": "318731123", "Assente": true}]

When I load the json file in elasticsearch and i get wirh command: GET /mattep/_search,
it returns:
hit:{
[...]
"message" : " "$oid": "603bf573e462e7ea44be9432"",
[...]},
hit{[...]
"message" : " "$NUM_TEL": "89123732"",
[...]}

I tried to write different .conf files. It is the last file:

input {

file {
		type => "json"
		start_position => "beginning"
		path => "C:/Users/matte/Desktop/Magistrale/query_2.json"
		sincedb_path => "NUL"
}

}

filter { json { source => "message" } }

output {
elasticsearch {
hosts => "localhost:9200"
index => "mattep"
document_type => "json"
}

stdout {  codec => rubydebug  }

}

My goal is to put the data into Elasticsearch with the json file so that I can show it with Kibana.
I would to see, when I run "GET /MY_INDEX/_search [...]":

hit:{
[...]
"_id" : "603bf573e462e7ea44be9432",
"NUM_TEL": 2990129039,
[...]}

I've been reading every topic and document all afternoon.

Badger · March 1, 2021, 7:45pm

Use a multiline codec on the file input. There is an example here.

matteo001 · March 1, 2021, 8:18pm

I thank you tour reply but I don't understand it.
This is my conf file:
input {
file {
start_position => "beginning"
path => "C:/Users/matte/Desktop/Magistrale/query_2.json"
sincedb_path => "NUL"
codec => multiline { pattern => "^}" negate => true what => next
auto_flush_interval => 1 }
}
}
filter { json { source => "message" } }
output {
elasticsearch {
hosts => "localhost:9200"
index => "matte0_jud"
document_type => "json"
}
stdout { codec => rubydebug }
}

I m sorry if I tell you some bulllshit (if it is the term in english) .
SO, what is the problem in my conf file?

Badger · March 1, 2021, 8:26pm

I have no idea. What do you not like about the result?

matteo001 · March 2, 2021, 3:42pm

This not good because I Have:
[...]
"hits" : [
{
"_index" : "mattep",
"_type" : "json",
"_id" : "uanr7ncB22tqv8EEWKTF",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.506Z",
"@version" : "1",
"type" : "json",
"message" : " "$oid": "603bf573e462e7ea44be9432"",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
{
"_index" : "mattep",
"_type" : "json",
"_id" : "u6nr7ncB22tqv8EEWKTF",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.509Z",
"@version" : "1",
"type" : "json",
"message" : "},{",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
{
"_index" : "mattep",
"_type" : "json",
"_id" : "Iqnr7ncB22tqv8EEWKbG",
"_score" : 1.0,
"_source" : {
"tags" : [
"_jsonparsefailure"
],
"@timestamp" : "2021-03-01T17:51:41.646Z",
"@version" : "1",
"type" : "json",
"message" : " "$oid": "603bf574e462e7ea44be945f"",
"path" : "C:/Users/matte/Desktop/Magistrale/query_2.json",
"host" : "DESKTOP-6FJNOJN"
}
},
[....]

Instead, I want to have NO MESSAGE
"message" : "$_id": "603bf573e462e7ea44be9432"",

but I want:
{
'_id': "603bf573e462e7ea44be9432",
'NUM_TEL' : '089328192123',
...}

Did you understand?
In a nutshell, I would like to insert a list of KEY: VALUE in elasticsearch by giving it a json file as input.
When I do a get I would like to see the data as JSON output in ES, not have the json unpacked in rows, the json must be inserted into the ES data.

matteo001 · March 2, 2021, 5:48pm

I'd like to insert the value in ES so, i can shoe it with get command.
Example:
JSon datas:

I want it in ES, so:
.

After, I 'd like to show these results in kibana canvas (and also here i have a few problems)

I don't understand what is my problem! .conf file? Json file? ....????

Thank you for your responses in advance.

system · March 30, 2021, 5:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Move json file to elasticsearch Logstash	27	1093	September 2, 2019
Pipeline error in logstash on windows 10 Logstash windows	2	756	April 20, 2021
Logstash : error No configuration found in the configured sources Logstash	1	1161	May 24, 2019
[ERROR][logstash.config.sourceloader] No configuration found in the configured sources Logstash docker	3	6008	April 16, 2021
Json load in elasticserach thru logstash Logstash	21	1568	July 6, 2017

Logstash ('No configuration found in the configured sources')

Related topics