Logstash output to Elastic search is not working

zaluxa · January 13, 2015, 9:05am

Hi all,

I've started experimenting ELK today, unfortunately not succeeded.
Everything installed properly and running without any error. When I start
Logstash with the following command, output to STDOUT is fine. But nothing
is seen in elastic search:

#./logstash agent -e "input { stdin {} } output { elasticsearch { host =>
localhost } stdout { codec => rubydebug}}"

What should I do?

Elastic search's console output is:

[2015-01-13 15:55:48,072][INFO ][node ] [Apollo] started
[2015-01-13 15:55:51,392][INFO ][gateway ] [Apollo]
recovered [1] indices into cluster_state
[2015-01-13 15:55:51,422][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false}])
[2015-01-13 15:57:44,028][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:01:29,656][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}])
[2015-01-13 16:21:07,373][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:25:07,143][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false}])

Can't see anything from the following command output:
#curl http://localhost:9200/_search?pretty

Please help me on this.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f380776d-7c74-4aae-b999-9c93454e3eee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marc_2 · January 13, 2015, 12:39pm

It all looks ok to me, since one can see that the logstash process is added
as a node.
However, you should try to remove the .sincedb files in your home directory.
If sincedb files exist and you are trying to analyze identical log files it
will know that it already read in the info and wait for new log entries in
the file... ergo nothing will happen

On Tuesday, January 13, 2015 at 10:05:10 AM UTC+1, zal...@gmail.com wrote:

Hi all,

I've started experimenting ELK today, unfortunately not succeeded.
Everything installed properly and running without any error. When I start
Logstash with the following command, output to STDOUT is fine. But nothing
is seen in Elasticsearch:

#./logstash agent -e "input { stdin {} } output { elasticsearch { host =>
localhost } stdout { codec => rubydebug}}"

What should I do?

Elastic search's console output is:

[2015-01-13 15:55:48,072][INFO ][node ] [Apollo]
started
[2015-01-13 15:55:51,392][INFO ][gateway ] [Apollo]
recovered [1] indices into cluster_state
[2015-01-13 15:55:51,422][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false}])
[2015-01-13 15:57:44,028][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:01:29,656][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}])
[2015-01-13 16:21:07,373][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:25:07,143][INFO ][cluster.service ] [Apollo] added
{[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false}])

Can't see anything from the following command output:
#curl http://localhost:9200/_search?pretty

Please help me on this.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e47a908-3a8b-4c89-9ab9-923a67b5105e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

zaluxa · January 14, 2015, 2:27am

Hi Marc,

I didn't find any .sincedb file from the file system.The problem is still.

On Tuesday, January 13, 2015 at 8:39:57 PM UTC+8, Marc wrote:

It all looks ok to me, since one can see that the logstash process is
added as a node.
However, you should try to remove the .sincedb files in your home
directory.
If sincedb files exist and you are trying to analyze identical log files
it will know that it already read in the info and wait for new log entries
in the file... ergo nothing will happen

On Tuesday, January 13, 2015 at 10:05:10 AM UTC+1, zal...@gmail.com wrote:

Hi all,

I've started experimenting ELK today, unfortunately not succeeded.
Everything installed properly and running without any error. When I start
Logstash with the following command, output to STDOUT is fine. But nothing
is seen in Elasticsearch:

#./logstash agent -e "input { stdin {} } output { elasticsearch { host =>
localhost } stdout { codec => rubydebug}}"

What should I do?

Elastic search's console output is:

[2015-01-13 15:55:48,072][INFO ][node ] [Apollo]
started
[2015-01-13 15:55:51,392][INFO ][gateway ] [Apollo]
recovered [1] indices into cluster_state
[2015-01-13 15:55:51,422][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false}])
[2015-01-13 15:57:44,028][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:01:29,656][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}])
[2015-01-13 16:21:07,373][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:25:07,143][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false}])

Can't see anything from the following command output:
#curl http://localhost:9200/_search?pretty

Please help me on this.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce8c16ca-d336-4d0d-8ff4-6f3a61ca749d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marc_2 · January 15, 2015, 11:46am

What do you mean by
"Can't see anything from the following command output:
#curl http://localhost:9200/_search?pretty"

from your first post?

On Wednesday, January 14, 2015 at 3:27:57 AM UTC+1, zal...@gmail.com wrote:

Hi Marc,

I didn't find any .sincedb file from the file system.The problem is still.

On Tuesday, January 13, 2015 at 8:39:57 PM UTC+8, Marc wrote:

It all looks ok to me, since one can see that the logstash process is
added as a node.
However, you should try to remove the .sincedb files in your home
directory.
If sincedb files exist and you are trying to analyze identical log files
it will know that it already read in the info and wait for new log entries
in the file... ergo nothing will happen

On Tuesday, January 13, 2015 at 10:05:10 AM UTC+1, zal...@gmail.com
wrote:

Hi all,

I've started experimenting ELK today, unfortunately not succeeded.
Everything installed properly and running without any error. When I start
Logstash with the following command, output to STDOUT is fine. But nothing
is seen in Elasticsearch:

#./logstash agent -e "input { stdin {} } output { elasticsearch { host
=> localhost } stdout { codec => rubydebug}}"

What should I do?

Elastic search's console output is:

[2015-01-13 15:55:48,072][INFO ][node ] [Apollo]
started
[2015-01-13 15:55:51,392][INFO ][gateway ] [Apollo]
recovered [1] indices into cluster_state
[2015-01-13 15:55:51,422][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-suricata-3299-4010][cKVoEM8zT8KPVIAelpMSsg][suricata][inet[/172.16.4.88:9301]]{client=true,
data=false}])
[2015-01-13 15:57:44,028][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-21484-2010][O0emX_s0SmauCfqAC_YaTA][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:01:29,656][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}])
[2015-01-13 16:21:07,373][INFO ][cluster.service ] [Apollo]
removed
{[logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason:
zen-disco-node_failed([logstash-0.0.0.0-22435-2010][LdUiD4llTY6S7eiN8Z97ag][inet[/172.16.4.88:9302]]{client=true,
data=false}), reason transport disconnected
[2015-01-13 16:25:07,143][INFO ][cluster.service ] [Apollo]
added
{[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-0.0.0.0-24108-2010][k2ToeYbPRtW_LH4PLBcL-A][inet[/172.16.4.88:9302]]{client=true,
data=false}])

Can't see anything from the following command output:
#curl http://localhost:9200/_search?pretty

Please help me on this.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/33030a85-786c-46e6-b24c-b9de6403b79a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.