Help in Data collection and indexing

Elastic Stack Elasticsearch
21

I run the follow command to enable logstash

systemctl start logstash.service

22

Then it's probably in logstash logs.

I'm running logstash manually while I'm still developing instead of running as a service so I can see immediately the logs in my console.

23

Do you mean logstash-plain.log?

24

The log in logstash as below,

[2018-05-17T19:03:38,851][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-17T19:03:38,856][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-17T19:03:39,006][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/modules/arcsight/configuration"}
[2018-05-17T19:03:39,048][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["apache", "cloudwatch_logs"], :poll_interval=>"5000000000ns"}
[2018-05-17T19:03:39,167][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"Bad scheme 'localhost' found should be one of http/https", :class=>"LogStash::ConfigurationError"}
[2018-05-17T19:03:39,168][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2018-05-17T19:03:39,186][ERROR][logstash.configmanagement.elasticsearchsource] Configuration Management is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.management.elasticsearch settings.
[2018-05-17T19:03:39,192][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Configuration Management is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.management.elasticsearch settings.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/license_checker/licensed.rb:78:in `with_license_check'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/config_management/elasticsearch_source.rb:48:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/config_management/hooks.rb:52:in `after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in `block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in `fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in `<main>'"]}
[2018-05-17T19:03:39,198][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
25

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

26

For a walk through of building an integration from raw data all the way to Kibana Dashboards this might help...

https://github.com/robcowart/eslog_tutorial/blob/master/eslog_tutorial.pdf

Also, to get started with syslog, take a look at this...

27

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.