Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input

Datakids · January 18, 2019, 1:21pm

Hi @ all,

I mess arround with follwing and hope you have some Ideas:

input {
    file {
        path => ["C:\EsKiLo_6.5.4\Test.csv"]
		start_position => "beginning"
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

------ vs: ------

input {
 beats {
    port => 5044
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

Filebeat:

filebeat.prospectors:
- input_type: log
  paths:
    - C:\EsKiLo_6.5.4\Test.csv
  fields:
    logfiletype: test

Path = no errors in log AND no Index is created?
Beats = boom ready

What's up with Logstash using a path on a Windows 64bit PC
For Quick Testes I need to cleanup Filebeat instead of just using the logstash.conf, thats a bit hm hm hm...

Thanks in advance for suggestions

Badger · January 18, 2019, 1:44pm

Does adding this to the file input help?

sincedb_path => "NUL"

Datakids · January 18, 2019, 2:12pm

Hi,
no nfortunately not

[2019-01-18T15:04:33,844][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x15d7664 run>"}
[2019-01-18T15:04:33,941][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-18T15:04:33,992][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-01-18T15:04:34,454][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

grafik

It is ok If I have to use Beat (however why), but what I'm really interested in is why it doesn't work with Logstsh.

Badger · January 18, 2019, 2:54pm

Enable trace level logging and look for filewatch. You can turn on trace globally on the command line or through logstash.yml, or you can enable trace just for the filewatch modules through the api as described here.

Datakids · January 18, 2019, 4:22pm

Hi,
OK tried but (due to lack of experiences not right!)

curl -XGET localhost:9600/_node/logging?pretty = it works and give me an json overview

curl -XPUT localhost:9600/_node/logging?pretty -H 'Content-Type: application/json' -d'
{
"logger.filewatch.discoverer" : "TRACE"
}
'
Result:
.... block in spawn_thread'"]}curl: (6) Could not resolve host: application

Logstash log:

[2019-01-18T17:13:14,515][ERROR][logstash.agent           ] Internal API server error {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Json::ParserError", :message=>"Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])\"'\"; line: 1, column: 2]", :backtrace=>["C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/json.rb:16:in `jruby_load'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/modules/logging.rb:28:in `block in PUT /'", "org/jruby/RubyMethod.java:127:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/xss_header.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/path_traversal.rb:16:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/json_csrf.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/frame_options.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/nulllogger.rb:9:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/head.rb:13:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:66:in `block in call'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:50:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:57:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:153:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}
[2019-01-18T17:13:14,539][ERROR][logstash.agent           ] API HTTP Request {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}

I'm off here !

Thanks for reply

Datakids · January 18, 2019, 9:38pm

Finaly made it!

I add thus to Input:

codec => plain {
charset => "ISO-8859-1"
}

And used this commandline toi start for debug:

logstash --log.level debug -f aTestfile_path_3cols.conf

and found that entry in logstash logs:

[2019-01-18T22:18:09,404][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

Search the Forum and find this:

Yeah made my day when changed from \ to / in path!

Lesson learned ELK Stack is quite a bit hypersensitive based on whatever matter because in Filebeat I can still use \ instead of / ?
...time for beer...

Regards

system · February 15, 2019, 9:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat logs are not being send to logstash Beats filebeat	5	503	April 23, 2024
Logstash and Filebeat on Windows Kibana	8	315	July 7, 2023
Filebeat not sending logs to logstash (port 5044 ) Beats filebeat	18	3389	March 19, 2021
Filebeat Elastic working, but filebeat logstash not working Logstash	6	3217	May 19, 2017
How to troubleshoot Windows Filebeat to logstash Beats filebeat	4	2419	February 26, 2019

Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input

Related Topics