ES load testing?

Elastic Stack Elasticsearch
1

I had tested my ELK setup on a small set of logs and got it working.

Today I started it's testing on actual log files of a server. So one
server's one month data with around 10 type of logs.
So approximately 500 files to be monitored each approx 1k lines, comes upto
5 lac log lines.

**: I am using default setting for elasticsearch.

So when I started logstash ES, first I saw logstash kept running for 2
hours just reading logs and after that it gave following error on console:

log4j, [2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.transport.netty:
[logstash-XX-] disconnected from
[[#zen_unicast_2#][XX[inet[localhost/127.0.0.1:9301]]]log4j,
[2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.discovery.zen:
[logstashXX] filtered ping responses: (filter_client[true],
filter_data[false]) {none}log4j, [2014-12-02T16:03:19.089] DEBUG:
org.elasticsearch.transport.netty: [logstashXX] connected to node
[[#zen_unicast_2#][inet[localhost/127.0.0.1:9301]]]
ES console:

[2014-12-02 12:23:24,289][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [default][2014-12-02 12:23:24,972][INFO
][cluster.metadata ] [Mantis] [logstash-2014.12.02] update_mapping
[logs] (dynamic)[2014-12-02 12:23:39,675][INFO ][cluster.metadata ]
[Mantis] [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:13,975][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:26,389][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:18,142][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:31,564][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)Write failed: Broken
pipe
I guess the connection is lost bw logstash and ES.

I am using ELK first time with such amount data .

Any guidance??

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0d820088-3604-426e-b50e-d5c3dccd7ee5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I have used Tsung to load test my clusters. It's very easy to install and
configure.
It will give more insight about memory usage, i/o, network i/o etc.

I suggest that you read this blog entry
https://engineering.helpshift.com/2014/tsung/ about it.

You'll find here http://tsung.erlang-projects.org/ the official page for
Tsung.

Le mardi 2 décembre 2014 12:38:37 UTC+1, Siddharth Trikha a écrit :

I had tested my ELK setup on a small set of logs and got it working.

Today I started it's testing on actual log files of a server. So one
server's one month data with around 10 type of logs.
So approximately 500 files to be monitored each approx 1k lines, comes
upto 5 lac log lines.

**: I am using default setting for elasticsearch.

So when I started logstash ES, first I saw logstash kept running for 2
hours just reading logs and after that it gave following error on console:

log4j, [2014-12-02T16:03:19.087] DEBUG:
org.elasticsearch.transport.netty: [logstash-XX-] disconnected from
[[#zen_unicast_2#][XX[inet[localhost/127.0.0.1:9301]]]log4j,
[2014-12-02T16:03:19.087] DEBUG: org.elasticsearch.discovery.zen:
[logstashXX] filtered ping responses: (filter_client[true],
filter_data[false]) {none}log4j, [2014-12-02T16:03:19.089] DEBUG:
org.elasticsearch.transport.netty: [logstashXX] connected to node
[[#zen_unicast_2#][inet[localhost/127.0.0.1:9301]]]
ES console:

[2014-12-02 12:23:24,289][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [default][2014-12-02 12:23:24,972][INFO
][cluster.metadata ] [Mantis] [logstash-2014.12.02] update_mapping
[logs] (dynamic)[2014-12-02 12:23:39,675][INFO ][cluster.metadata ]
[Mantis] [logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:13,975][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:51:26,389][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:18,142][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)[2014-12-02
12:55:31,564][INFO ][cluster.metadata ] [Mantis]
[logstash-2014.12.02] update_mapping [logs] (dynamic)Write failed: Broken
pipe
I guess the connection is lost bw logstash and ES.

I am using ELK first time with such amount data .

Any guidance??

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/25189c47-a9c4-4933-a29e-9599bec454d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.