Help in Data collection and indexing


(Mark Walkom) #8

Is that not working? It looks ok.


(Thompson) #9

I just create the above config file. And there is syslog sent to my Elastic server.

Should I update and configure anything further such that Elasticsearch is collecting those syslog and build an index for further processing on Kibana?


(Mark Walkom) #10

If it's working, then see what the analysis looks like and go from there.


(Thompson) #11

From the view of Kibana, I can find system indices only.
I didn't find the one I am trying to create. Do I miss any steps?


(Mark Walkom) #12

What does the output from _cat/indices?v show?


(Thompson) #13

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-kibana-6-2018.05.07 7qS0tPxOQa-N6Q7OZj03EQ 1 0 7107 0 1.9mb 1.9mb
green open .kibana 8b9DbLcMTaaOtUYo-j59PQ 1 0 1 0 4kb 4kb
green open .monitoring-es-6-2018.05.08 mtDWvwHCRq-UgQ0F8xlfDw 1 0 198731 228 107.1mb 107.1mb
green open .monitoring-es-6-2018.05.06 6m1XXKo7QvCk3Inp4hzO-g 1 0 146896 78 65.9mb 65.9mb
green open .watcher-history-7-2018.05.05 YE0pokcCRECJr2N4iCa9hA 1 0 8628 0 11.7mb 11.7mb
green open .monitoring-es-6-2018.05.10 xkWc-cOxRouax2g61F6znQ 1 0 92061 333 51.8mb 51.8mb
green open .monitoring-alerts-6 N9R4IKxiQMiXzo5I4wURKQ 1 0 2 0 12kb 12kb
green open .watcher-history-7-2018.05.06 LiojSCnURVSuAUDPss8AyQ 1 0 8634 0 11.7mb 11.7mb
green open .monitoring-kibana-6-2018.05.09 OLOv908_RM61ZlZolioC7w 1 0 8638 0 2mb 2mb
green open .watcher-history-7-2018.05.04 xsRjEun5SgmV9JMHNrfiQw 1 0 7030 0 9.5mb 9.5mb
green open .monitoring-es-6-2018.05.04 BIha6WuGRGKKopPjAF1keQ 1 0 74383 63 32mb 32mb
green open .monitoring-kibana-6-2018.05.05 pK00cdlLQMm8_LGmyUP8Xg 1 0 8637 0 1.9mb 1.9mb
green open .watches kTDTEu2xRq-hSz45dfWd0A 1 0 0 0 268b 268b
green open .monitoring-kibana-6-2018.05.06 MnX0ObLWQ1-fVlR0wa3qPg 1 0 8638 0 1.9mb 1.9mb
green open .monitoring-kibana-6-2018.05.08 rl9-m2-5S3qiwD6Dcfj1ag 1 0 8637 0 2.1mb 2.1mb
yellow open test flziY85sTy-LFua8vnxx6Q 5 1 1 0 4.4kb 4.4kb
green open .triggered_watches dOPdzOQbRRqDyuxEHwpxdw 1 0 0 0 3.2mb 3.2mb
green open .monitoring-kibana-6-2018.05.10 TmdojM3UTk-sOF3mYF7l-A 1 0 3394 0 1mb 1mb
green open .monitoring-es-6-2018.05.07 GckqTampTU22W449XQaUvQ 1 0 178671 102 95.8mb 95.8mb
green open .monitoring-kibana-6-2018.05.04 z-8JMgWZRJ2FuvRWnAHiJw 1 0 5525 0 1.5mb 1.5mb
green open .monitoring-es-6-2018.05.09 JRklvsD2QXCeiOOAxP4faA 1 0 216013 252 114.5mb 114.5mb
green open .security-6 wdydXTI2QO-AQJBp9tWRrA 1 0 3 0 9.8kb 9.8kb
green open .monitoring-es-6-2018.05.05 eEeFACUgQNKS9s75yzoX3Q 1 0 120968 80 53.9mb 53.9mb
green open .watcher-history-7-2018.05.07 JkZbccpKRh2Azb4WES4urw 1 0 2832 0 3.9mb 3.9mb


(Mark Walkom) #14

I'd put a stdout section in the output to make sure that things are coming in and making it to the output.


(Thompson) #15

Sorry, I didn't get it


(David Pilato) #16

Replace

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

With

output {
  stdout { codec => json }
}

(Thompson) #17

I check that there is a lot of syslog sent to the ELK server with all essential modules installed.

I configured as above said. Found that 514 port is not listening, is it the source of problem?

How can I check if the log is successfully sent to input and then output to elasticsearch?

Thanks


(David Pilato) #18
input {
  udp {
    port => "514"
    type => "syslog"
  }
}

filter {
}

output {
  stdout { codec => json }
}

If something is received on UDP / 514, then you will see it in the logstash stdout.


(Thompson) #19

How can I see the logstash stdout?


(David Pilato) #20

How do you launch Logstash? Does it print anything?


(Thompson) #21

I run the follow command to enable logstash

systemctl start logstash.service


(David Pilato) #22

Then it's probably in logstash logs.

I'm running logstash manually while I'm still developing instead of running as a service so I can see immediately the logs in my console.


(Thompson) #23

Do you mean logstash-plain.log?


(Thompson) #24

The log in logstash as below,

[2018-05-17T19:03:38,851][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-05-17T19:03:38,856][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-05-17T19:03:39,006][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/modules/arcsight/configuration"}
[2018-05-17T19:03:39,048][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["apache", "cloudwatch_logs"], :poll_interval=>"5000000000ns"}
[2018-05-17T19:03:39,167][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"Bad scheme 'localhost' found should be one of http/https", :class=>"LogStash::ConfigurationError"}
[2018-05-17T19:03:39,168][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2018-05-17T19:03:39,186][ERROR][logstash.configmanagement.elasticsearchsource] Configuration Management is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.management.elasticsearch settings.
[2018-05-17T19:03:39,192][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Configuration Management is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.management.elasticsearch settings.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/license_checker/licensed.rb:78:in `with_license_check'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/config_management/elasticsearch_source.rb:48:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/lib/config_management/hooks.rb:52:in `after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in `block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in `fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in `<main>'"]}
[2018-05-17T19:03:39,198][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

(David Pilato) #25

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.


(Robert Cowart) #26

For a walk through of building an integration from raw data all the way to Kibana Dashboards this might help...

https://github.com/robcowart/eslog_tutorial/blob/master/eslog_tutorial.pdf

Also, to get started with syslog, take a look at this...


(system) closed #27

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.