# Logstash sending less data to ElasticSearch Cluster
**URL:** https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749
**Category:** Logstash
**Created:** [July 29, 2019, 5:14pm UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749 "2019-07-29T17:14:00Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 29, 2019, 5:14pm UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/1 "2019-07-29T17:14:00Z")
Hello,
I am sending data from filebeats to a logstash machine and then to ElasticSearch cluster.
I have persistent queue enabled on logstash.The queue gets filled very quickly.I can't figure out out reason why it is not sending data to ElasticSearch.The network monitor on logstash machine looks odd.Can anyone help me with the issue.
The graph is MiB per minute.So its 43MiB per minute at 19:53 in image
Green is receiving and yellow is transmitting.I dont know why it is exactly symmetric.I have done speed test on this machine and it can give speed upto 500 Mb/s download and 500 Mb/s upload speed.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 29, 2019, 7:38pm UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/2 "2019-07-29T19:38:47Z")
What is the hardware specification of the Elasticsearch cluster?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 30, 2019, 6:45am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/3 "2019-07-30T06:45:35Z")
1 co-ordination node,1 master node and 4 data nodes.
2 data nodes are master eligible as well.
**Data Nodes:**
`Ram : 31.548 GB`
`Total disk : 2.3 TB`
**Co-ordination node:**
| Model | vCPU | Memory (GiB) | Instance Storage (GiB) | Network Bandwidth (Gbps) | EBS Bandwidth (Mbps) |
| --- | --- | --- | --- | --- | --- |
| c5n.large | 2 | 5.25 | EBS-Only | Up to 25 | Up to 3,500 |
**MASTER NODE:**
`Ram : 3.549 GB`
`Total disk : 31 GB`
> Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
> Stepping: 3
> CPU MHz: 3000.000
> BogoMIPS: 6000.00
> Hypervisor vendor: KVM
> Virtualization type: full
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 1024K
> L3 cache: 25344K
This shouldn't be ingestion problem I think.Why is the network graph symmetric.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2019, 6:56am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/4 "2019-07-30T06:56:24Z")
What instance types are you using for the data nodes? If it is all deployed on AWS it would be great if you could list all instance types used.
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 30, 2019, 8:04am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/5 "2019-07-30T08:04:40Z")
**Data nodes** :r5.xlarge (4 machines)
**Master node** :c5.large
**Co-ordinating node** :c5n.large
**Logstash machine** :c4.xlarge
I have given half of the ram in each machine to jvm
Logstash is receiving data at rate of 1MB/s from several filebeats
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 31, 2019, 6:31am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/6 "2019-07-31T06:31:41Z")
Any idea?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2019, 6:39am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/7 "2019-07-31T06:39:47Z")
Is there anything in Elasticsearch logs or stats that is indicating that it is struggling or limited by resources so Logstash would apply back pressure? What does CPU usage, GC and disk I/O look like on the Elasticsearch nodes?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 31, 2019, 6:40am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/8 "2019-07-31T06:40:58Z")
No,
I am just sending from this 1 logstash machine.
Logstash machine is running on centos6.Can this be a problem?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2019, 6:42am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/9 "2019-07-31T06:42:36Z")
I don’t know but would check Elasticsearch first. Do you output to any other systems?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [July 31, 2019, 7:11am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/10 "2019-07-31T07:11:56Z")
yes I write to disk as well as send to ElasticSearch
You have any idea about why upload speed+download speed is a constant.Though my machine can support higher speed transfers
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [August 1, 2019, 6:23pm UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/11 "2019-08-01T18:23:57Z")
Any help?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 1, 2019, 6:29pm UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/12 "2019-08-01T18:29:35Z")
Did you check Elasticsearch?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [August 2, 2019, 5:09am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/13 "2019-08-02T05:09:05Z")
Yes,
No errors or warnings in elasticsearch logs
So logging has not stopped.But logstash is ingesting at lowser rate .And also will adding 1 more co-ordinator help?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 2, 2019, 5:10am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/14 "2019-08-02T05:10:57Z")
Do you have monitoring installed on Elasticsearch? If so, what is the reported indexing rate? What does heap usage look like? What about CPU usage?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [August 2, 2019, 5:14am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/15 "2019-08-02T05:14:57Z")
Indexing at 14,000/s
And heap usage of whole cluster 50%
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 2, 2019, 5:18am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/16 "2019-08-02T05:18:01Z")
What about CPU usage? What about disk I/O and iowait, e.g. through `netstat -x`?
How much CPU is the Logstash machine using?
---
### Author:  [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)
#### Post date: [August 2, 2019, 11:06am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/17 "2019-08-02T11:06:31Z")
On Kibana dashboard:
JVM heap usage = 50% avg of all
CPU usage:
Co-ordinator=12%
[node-data-1]=37%,#shards=190
[node-data-2]=78%,#shards=190
[node-data-3]=40%,#shards=190
[node-data-4]=36%,#shards=190
master = 0%
Logstash machine =60% cpu usage
only logstash is running on logstash machine
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 30, 2019, 11:17am UTC](https://discuss.elastic.co/t/logstash-sending-less-data-to-elasticsearch-cluster/192749/18 "2019-08-30T11:17:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.