Hello everyone,
Am working on a search engine project and am new experiencing with the ELK stack , so after installing and configuring it , I am now called to pass csv files which came originally from parsed output via Hachoir python library.
for the initial conducted tests I chose to feed logstash the syslogs from my operating system (Ubuntu 18.0.4) , I used the filter from the documentation.
running ./logstash --path.settings /etc/logstash/ returned this:
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-01-20T05:10:44,138][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.5.1"}
[2020-01-20T05:10:47,736][INFO ][org.reflections.Reflections] Reflections took 46 ms to scan 1 urls, producing 20 keys and 40 values
[2020-01-20T05:10:48,990][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-01-20T05:10:49,249][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-01-20T05:10:49,323][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-01-20T05:10:49,330][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-01-20T05:10:49,415][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2020-01-20T05:10:49,981][INFO ][logstash.filters.geoip ][main] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[2020-01-20T05:10:50,146][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-01-20T05:10:50,154][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/10-syslog-filter.conf"], :thread=>"#<Thread:0x217552f2 run>"}
[2020-01-20T05:10:50,833][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-01-20T05:10:50,856][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-01-20T05:10:51,025][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-01-20T05:10:51,094][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-01-20T05:10:51,679][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
All am asking is a sanity check from anyone who has been the same road I am walking today:
1/ how can I view the indexed syslogs on kibana and what are the possible operations one can perform ?
2/ what would be the appropriate filter (i wouldn't mind the input/output too ) needed for the kind of csv files i want to index on kibana ?
any links related are very much appreciated too.
Thank you in advance.