Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input

Datakids · January 18, 2019, 1:21pm

Hi @ all,

I mess arround with follwing and hope you have some Ideas:

input {
    file {
        path => ["C:\EsKiLo_6.5.4\Test.csv"]
		start_position => "beginning"
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

------ vs: ------

input {
 beats {
    port => 5044
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

Filebeat:

filebeat.prospectors:
- input_type: log
  paths:
    - C:\EsKiLo_6.5.4\Test.csv
  fields:
    logfiletype: test

Path = no errors in log AND no Index is created?
Beats = boom ready

What's up with Logstash using a path on a Windows 64bit PC
For Quick Testes I need to cleanup Filebeat instead of just using the logstash.conf, thats a bit hm hm hm...

Thanks in advance for suggestions

Badger · January 18, 2019, 1:44pm

Does adding this to the file input help?

sincedb_path => "NUL"

Datakids · January 18, 2019, 2:12pm

Hi,
no nfortunately not

[2019-01-18T15:04:33,844][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x15d7664 run>"}
[2019-01-18T15:04:33,941][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-18T15:04:33,992][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-01-18T15:04:34,454][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

grafik

It is ok If I have to use Beat (however why), but what I'm really interested in is why it doesn't work with Logstsh.

Badger · January 18, 2019, 2:54pm

Enable trace level logging and look for filewatch. You can turn on trace globally on the command line or through logstash.yml, or you can enable trace just for the filewatch modules through the api as described here.

Datakids · January 18, 2019, 4:22pm

Hi,
OK tried but (due to lack of experiences not right!)

curl -XGET localhost:9600/_node/logging?pretty = it works and give me an json overview

curl -XPUT localhost:9600/_node/logging?pretty -H 'Content-Type: application/json' -d'
{
"logger.filewatch.discoverer" : "TRACE"
}
'
Result:
.... block in spawn_thread'"]}curl: (6) Could not resolve host: application

Logstash log:

[2019-01-18T17:13:14,515][ERROR][logstash.agent           ] Internal API server error {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Json::ParserError", :message=>"Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])\"'\"; line: 1, column: 2]", :backtrace=>["C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/json.rb:16:in `jruby_load'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/modules/logging.rb:28:in `block in PUT /'", "org/jruby/RubyMethod.java:127:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/xss_header.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/path_traversal.rb:16:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/json_csrf.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/frame_options.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/nulllogger.rb:9:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/head.rb:13:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:66:in `block in call'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:50:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:57:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:153:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}
[2019-01-18T17:13:14,539][ERROR][logstash.agent           ] API HTTP Request {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}

I'm off here !

Thanks for reply

Datakids · January 18, 2019, 9:38pm

Finaly made it!

I add thus to Input:

codec => plain {
charset => "ISO-8859-1"
}

And used this commandline toi start for debug:

logstash --log.level debug -f aTestfile_path_3cols.conf

and found that entry in logstash logs:

[2019-01-18T22:18:09,404][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

Search the Forum and find this:

Yeah made my day when changed from \ to / in path!

Lesson learned ELK Stack is quite a bit hypersensitive based on whatever matter because in Filebeat I can still use \ instead of / ?
...time for beer...

Regards

system · February 15, 2019, 9:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash parsing input filter beat csv file Logstash	1	252	September 22, 2020
Help with logstash csv and beats conf file Logstash	2	311	January 9, 2019
Logstash input beat issue Logstash	1	218	September 27, 2020
Csv -> filebeat -> logstash not parsing? Beats filebeat	2	1081	July 1, 2018
Logstash errors out-Couldn't find any input plugin named 'beats' Logstash	2	1103	February 6, 2017

Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input

Related topics