Unable to create logstash.conf for log file

Elastic Stack Logstash
1

Please correct my logstash.conf as i am getting below error message and in kibana i can see my data in dev tool (GET /_search) but not able to view data in discover tab
Q1.I need to add any index in logstash.conf?
Q2.i need to add any filter

ERROR StatusLogger No log4j2 configuration file found. Using default configurati
on: logging only errors to the console.
Sending Logstash's logs to C:/logstash-5.5.2/logs which is now configured via lo
g4j2.properties
[2017-08-29T19:40:56,025][INFO ][logstash.pipeline ] Starting pipeline {"
id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.
delay"=>5, "pipeline.max_inflight"=>500}
[2017-08-29T19:40:56,649][INFO ][logstash.pipeline ] Pipeline main starte
d
[2017-08-29T19:40:56,821][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}

For all general issues, please provide the following details for fast resolution:

Version:5.5.2

Operating System:windows 2008 r2 server

Config File (if you have sensitive info, please remove it): Logstash.conf
input {
file{
path=> "\logstash-5.5.2\bin\test.log"
start_position=>"beginning"
sincedb_path=>"/dev/null"
}
}
output {
elasticsearch {
hosts =>["localhost:9200"]

}

stdout{}
}

Sample Data: test.log
2017-08-29 14:49:28 INFO jupiter:? - 'ApplicationName':string, 'EventStatus':string, 'SeverityLevel':string, 'EventTime':string, 'ErrorDescription':string, 'ErrorCode':string, 'IsSecurityIncident':string, 'UrlRequested':string, 'MethodName':string, 'CurrentUser':string, 'RequestedIp':string, 'TransactionId':string, 'ActionMethod':string, 'MessageFormat':string, 'MessageParameters':{}
Steps to Reproduce:
logstash -f logstash.conf

2

sincedb_path=>”/dev/null”

On Windows use "nul" instead of "/dev/null".

3

as suggested i modified logstash.conf asbelow but In Kibana i not able to see any data:
input {
file{
path=> "/logstash-5.5.2/bin/test1.log"
start_position=>"beginning"
sincedb_path=>"nul"
}
}
output { stdout {}
elasticsearch { hosts => ["localhost:9200"] }
}

Logstash cmd:
C:\logstash-5.5.2\bin>logstash -f logstash.conf
ERROR StatusLogger No log4j2 configuration file found. Using default configurati
on: logging only errors to the console.
Sending Logstash's logs to C:/logstash-5.5.2/logs which is now configured via lo
g4j2.properties
[2017-08-31T12:35:21,325][INFO ][logstash.outputs.elasticsearch] Elasticsearch p
ool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-08-31T12:35:21,466][INFO ][logstash.outputs.elasticsearch] Running health
check to see if an Elasticsearch connection is working {:healthcheck_url=>http:/
/localhost:9200/, :path=>"/"}
[2017-08-31T12:35:21,700][WARN ][logstash.outputs.elasticsearch] Restored connec
tion to ES instance {:url=>"http://localhost:9200/"}
[2017-08-31T12:35:21,700][INFO ][logstash.outputs.elasticsearch] Using mapping t
emplate from {:path=>nil}
[2017-08-31T12:35:21,903][INFO ][logstash.outputs.elasticsearch] Attempting to i
nstall template {:manage_template=>{"template"=>"logstash-", "version"=>50001,
"settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-08-31T12:35:21,934][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2017-08-31T12:35:21,949][INFO ][logstash.pipeline ] Starting pipeline {"
id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.
delay"=>5, "pipeline.max_inflight"=>500}
[2017-08-31T12:35:23,185][INFO ][logstash.pipeline ] Pipeline main started
[2017-08-31T12:35:23,473][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}

4

If you increase the log level Logstash will tell you more details about what's going on with the reading of the input file.

5

But my question that In Kibana i not able to see any data:...??? can u please provide conf file to load my test.log in ES ..???? please

6

I am trying to help you figure out why your configuration file isn't working. If you don't play along we're not going to get anywhere.

7

i changed conf file as below ,now i am not able to read file from my local path as no output in stdout.i didnt find any reference and vedios for logstash to start dev.i had to test and everything good then my company going to use this in production.
input {
file{
path=> "C:/logstash-5.5.2/bin/test1.log"
start_position=>"beginning"
sincedb_path=>"/dev/null"
}
}
output {
stdout {}

}

8

Please can some one fix this issue.

9

sincedb_path=>”/dev/null”

As I've already explained to you this line is wrong.

I have also asked you to increase the log level of Logstash so that we can get more clues to aid the debugging. That might reveal what the problem is.

Until you provide the information asked for we're all just wasting our time.

10

sorry, i changed to "nul" even though it is not showing in stdout

11

can u please elobrate what mean to increase log level of logstash
.If posible can u share any reference to read a log file in to logstash

12

can u please elobrate what mean to increase log level of logstash

Logstash's logging verbosity can be changed via command-line options. Bumping it up to "debug" should provide us with additional clues.

13

Logstash --debug -f logstash.conf ranin cmd
in cmd:
[2017-08-31T16:15:02,829][DEBUG][logstash.inputs.file ] _globbed_files: c:/logstash-5.5.2/bin/test1.log: glob is: ["c:/logstash-5.5.2/bin/test1.log"]
[2017-08-31T16:15:02,829][DEBUG][logstash.inputs.file ] _discover_file: c:/logstash-5.5.2/bin/test1.log: new: c:/logstash-5.5.2/bin/test1.log (exclude is )
[2017-08-31T16:15:03,843][DEBUG][logstash.inputs.file ] _open_file: c:/logstash-5.5.2/bin/test1.log: opening
[2017-08-31T16:15:03,843][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb last value 0, cur size 62
[2017-08-31T16:15:03,843][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb: seeking to 0
[2017-08-31T16:15:03,843][DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1504176303)
[2017-08-31T16:15:03,859][DEBUG][logstash.inputs.file ] :delete for c:/logstash-5.5.2/bin/test1.log, closing file
[2017-08-31T16:15:06,027][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-08-31T16:15:11,039][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-08-31T16:15:16,049][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-08-31T16:15:18,014][DEBUG][logstash.inputs.file ] _globbed_files: c:/logstash-5.5.2/bin/test1.log: glob is: ["c:/logstash-5.5.2/bin/test1.log"]
[2017-08-31T16:15:18,014][DEBUG][logstash.inputs.file ] _discover_file: c:/logstash-5.5.2/bin/test1.log: new: c:/logstash-5.5.2/bin/test1.log (exclude is )
[2017-08-31T16:15:19,028][DEBUG][logstash.inputs.file ] _open_file: c:/logstash-5.5.2/bin/test1.log: opening
[2017-08-31T16:15:19,028][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb last value 0, cur size 62
[2017-08-31T16:15:19,028][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb: seeking to 0
[2017-08-31T16:15:19,028][DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1504176319)[2017-08-31T16:15:19,028][DEBUG][logstash.inputs.file ] :delete for c:/logstash-5.5.2/bin/test1.log, closing file
[2017-08-31T16:15:21,058][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-08-31T16:15:26,068][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-08-31T16:15:33,184][DEBUG][logstash.inputs.file ] _globbed_files: c:/logstash-5.5.2/bin/test1.log: glob is: ["c:/logstash-5.5.2/bin/test1.log"]
[2017-08-31T16:15:35,368][DEBUG][logstash.inputs.file ] _discover_file: c:/logstash-5.5.2/bin/test1.log: new: c:/logstash-5.5.2/bin/test1.log (exclude is )
[2017-08-31T16:15:36,382][DEBUG][logstash.inputs.file ] _open_file: c:/logstash-5.5.2/bin/test1.log: opening
[2017-08-31T16:15:36,382][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb last value 0, cur size 62
[2017-08-31T16:15:36,382][DEBUG][logstash.inputs.file ] c:/logstash-5.5.2/bin/test1.log: sincedb: seeking to 0
[2017-08-31T16:15:36,382][DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1504176336)
[2017-08-31T16:15:36,382][DEBUG][logstash.inputs.file ] :delete for c:/logstash-5.5.2/bin/test1.log, closing file

14

it is very worse that to load a file in logstash is this mush time taken ,better i should switch to other tool.

15

please help to fix config file issue, other wise leave it.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.