# Help in Data collection and indexing

**URL:** <https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802>\
**Category:** Elasticsearch\
**Created:** [May 7, 2018, 9:21am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802 "2018-05-07T09:21:24Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 7, 2018, 9:21am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/1 "2018-05-07T09:21:24Z")

</div>

I have already finished installation and configuration of Elastic, Kibana and Logstash.

I found some existing indices after browsing to localhost:9200/\_cat/indices?v

Now I want to send logs from different devices via JDBC & Syslog to ElasticSearch and view it in Kibana.

How can I do so?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 7, 2018, 9:22am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/2 "2018-05-07T09:22:30Z")

</div>

Have you seen Beats and Logstash?

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 7, 2018, 9:54am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/3 "2018-05-07T09:54:54Z")

</div>

I go for agentless approach at first so Beats is not useful at this moment.

For Logstash, I get confused which part mentions collecting different logs

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 7, 2018, 10:19am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/4 "2018-05-07T10:19:19Z")

</div>

You will usually want an input, filter and an output.

Start with a file input, and go from there. The docs walk you through some basic processing that should get you started.

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 7, 2018, 10:36am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/5 "2018-05-07T10:36:19Z")

</div>

May I have the URL? Which step to start with?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 7, 2018, 8:47pm UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/6 "2018-05-07T20:47:53Z")

</div>

[https://www.elastic.co/guide/en/logstash/6.2/getting-started-with-logstash.html](https://www.elastic.co/guide/en/logstash/6.2/getting-started-with-logstash.html)

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 10, 2018, 2:57am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/7 "2018-05-10T02:57:25Z")

</div>

I got a bit confused.

I try to create a syslog input config file under conf.d directory as below,

input {  
udp {  
port =\> "514"  
type =\> "syslog"  
}  
}

filter {  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
}

What should I do next to make it effective and then view data from Kibana?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 10, 2018, 6:02am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/8 "2018-05-10T06:02:03Z")

</div>

Is that not working? It looks ok.

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 10, 2018, 7:13am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/9 "2018-05-10T07:13:49Z")

</div>

I just create the above config file. And there is syslog sent to my Elastic server.

Should I update and configure anything further such that Elasticsearch is collecting those syslog and build an index for further processing on Kibana?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 10, 2018, 7:16am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/10 "2018-05-10T07:16:01Z")

</div>

If it's working, then see what the analysis looks like and go from there.

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 10, 2018, 7:59am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/11 "2018-05-10T07:59:51Z")

</div>

From the view of Kibana, I can find system indices only.  
I didn't find the one I am trying to create. Do I miss any steps?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 10, 2018, 8:49am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/12 "2018-05-10T08:49:45Z")

</div>

What does the output from `_cat/indices?v` show?

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 10, 2018, 9:25am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/13 "2018-05-10T09:25:56Z")

</div>

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
green open .monitoring-kibana-6-2018.05.07 7qS0tPxOQa-N6Q7OZj03EQ 1 0 7107 0 1.9mb 1.9mb  
green open .kibana 8b9DbLcMTaaOtUYo-j59PQ 1 0 1 0 4kb 4kb  
green open .monitoring-es-6-2018.05.08 mtDWvwHCRq-UgQ0F8xlfDw 1 0 198731 228 107.1mb 107.1mb  
green open .monitoring-es-6-2018.05.06 6m1XXKo7QvCk3Inp4hzO-g 1 0 146896 78 65.9mb 65.9mb  
green open .watcher-history-7-2018.05.05 YE0pokcCRECJr2N4iCa9hA 1 0 8628 0 11.7mb 11.7mb  
green open .monitoring-es-6-2018.05.10 xkWc-cOxRouax2g61F6znQ 1 0 92061 333 51.8mb 51.8mb  
green open .monitoring-alerts-6 N9R4IKxiQMiXzo5I4wURKQ 1 0 2 0 12kb 12kb  
green open .watcher-history-7-2018.05.06 LiojSCnURVSuAUDPss8AyQ 1 0 8634 0 11.7mb 11.7mb  
green open .monitoring-kibana-6-2018.05.09 OLOv908\_RM61ZlZolioC7w 1 0 8638 0 2mb 2mb  
green open .watcher-history-7-2018.05.04 xsRjEun5SgmV9JMHNrfiQw 1 0 7030 0 9.5mb 9.5mb  
green open .monitoring-es-6-2018.05.04 BIha6WuGRGKKopPjAF1keQ 1 0 74383 63 32mb 32mb  
green open .monitoring-kibana-6-2018.05.05 pK00cdlLQMm8\_LGmyUP8Xg 1 0 8637 0 1.9mb 1.9mb  
green open .watches kTDTEu2xRq-hSz45dfWd0A 1 0 0 0 268b 268b  
green open .monitoring-kibana-6-2018.05.06 MnX0ObLWQ1-fVlR0wa3qPg 1 0 8638 0 1.9mb 1.9mb  
green open .monitoring-kibana-6-2018.05.08 rl9-m2-5S3qiwD6Dcfj1ag 1 0 8637 0 2.1mb 2.1mb  
yellow open test flziY85sTy-LFua8vnxx6Q 5 1 1 0 4.4kb 4.4kb  
green open .triggered\_watches dOPdzOQbRRqDyuxEHwpxdw 1 0 0 0 3.2mb 3.2mb  
green open .monitoring-kibana-6-2018.05.10 TmdojM3UTk-sOF3mYF7l-A 1 0 3394 0 1mb 1mb  
green open .monitoring-es-6-2018.05.07 GckqTampTU22W449XQaUvQ 1 0 178671 102 95.8mb 95.8mb  
green open .monitoring-kibana-6-2018.05.04 z-8JMgWZRJ2FuvRWnAHiJw 1 0 5525 0 1.5mb 1.5mb  
green open .monitoring-es-6-2018.05.09 JRklvsD2QXCeiOOAxP4faA 1 0 216013 252 114.5mb 114.5mb  
green open .security-6 wdydXTI2QO-AQJBp9tWRrA 1 0 3 0 9.8kb 9.8kb  
green open .monitoring-es-6-2018.05.05 eEeFACUgQNKS9s75yzoX3Q 1 0 120968 80 53.9mb 53.9mb  
green open .watcher-history-7-2018.05.07 JkZbccpKRh2Azb4WES4urw 1 0 2832 0 3.9mb 3.9mb

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 10, 2018, 9:26am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/14 "2018-05-10T09:26:48Z")

</div>

I'd put a stdout section in the output to make sure that things are coming in and making it to the output.

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 14, 2018, 6:27am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/15 "2018-05-14T06:27:32Z")

</div>

Sorry, I didn't get it

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 14, 2018, 6:51am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/16 "2018-05-14T06:51:58Z")

</div>

Replace

```auto
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

```

With

```auto
output {
  stdout { codec => json }
}

```

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 15, 2018, 11:13am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/17 "2018-05-15T11:13:10Z")

</div>

I check that there is a lot of syslog sent to the ELK server with all essential modules installed.

I configured as above said. Found that 514 port is not listening, is it the source of problem?

How can I check if the log is successfully sent to input and then output to elasticsearch?

Thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 16, 2018, 10:08am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/18 "2018-05-16T10:08:12Z")

</div>

```auto
input {
  udp {
    port => "514"
    type => "syslog"
  }
}

filter {
}

output {
  stdout { codec => json }
}

```

If something is received on UDP / 514, then you will see it in the logstash stdout.

---

<div class="post-metadata">

**Author:** ![thompsonlau](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@thompsonlau](https://discuss.elastic.co/u/thompsonlau)\
**Post date:** [May 16, 2018, 10:54am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/19 "2018-05-16T10:54:34Z")

</div>

How can I see the logstash stdout?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 16, 2018, 11:30am UTC](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802/20 "2018-05-16T11:30:32Z")

</div>

How do you launch Logstash? Does it print anything?

[Next page](https://discuss.elastic.co/t/help-in-data-collection-and-indexing/130802.md?page=2)