# Huge concurrent data ingestion to ElasticSearch
**URL:** https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318
**Category:** Elasticsearch
**Created:** [July 31, 2018, 9:05am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318 "2018-07-31T09:05:49Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [July 31, 2018, 9:05am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/1 "2018-07-31T09:05:49Z")
Hi Guys, I encountered an issue in ElasticSearch. We use Logstash listen a TCP connection, then filter and output. We only have a few dissect filters in Logstash to improve Logstash processing speed.
Our logstash settings is
```auto
pipeline.batch.size: 2500
pipeline.batch.delay: 5
pipeline.workers: 24
pipeline.output.workers: 24
config.reload.automatic: true
config.reload.interval: 100s
```
We have a 32 Cores CPU and 96GB mem in LS server.
In order to relieve Logstash pressure, we move almost parsing code to ingest node of ES. In that ingest node, we have some processors to do some calculation and regular expression to our data. Meanwhile there is another master/data node combining a ES cluster. The ingest node is only used for ingest, disabling data/master node. The data node is in the contrary.
Ingest node has 4G JVM, Data node has 16G JVM. They are in different servers. Both of servers have 96G mem. The refresh\_interval is 180 seconds for each indices.
Now the issue is that we want to send 700K events in TCP connection in one minutes, after several intervals, 8 or 9, ElasticSearch is not able to receive data any more, I queries in \_cat/indices, the disk storage would not change. and data generator (another app, not logstash input plugin)will crash. However I can not see any errors in both ES and LS logs. Does anyone have ideas to improve ingestion behavior for ES and LS? I much appreciate your help!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2018, 9:48am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/2 "2018-07-31T09:48:27Z")
Do you have monitoring installed for Elasticsearch? What is the full specification of the Elasticsearch data node(s), e.g. CPU cores and type of storage? How many indices/shards are you actively indexing into? What is the average size of your documents? Is there anything else running on the hosts?
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [July 31, 2018, 10:02am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/3 "2018-07-31T10:02:00Z")
Hi Christian, thanks for your reply.
- We do have monitoring tools like ElasticHQ and cerebro installed in our server. However I can hardly find out the root cause, all metrics looked good for me..
- We have two servers, one has Logstash, Kibana and Ingset node, another has data node only. Both of servers have 96GB memories and 32 cores. The storage is normal hard disk, no RAID I guess.
- there is only one index being ingested with 5 shards. This index has less than 20 fields in each document. See below, you may know about doc size.
```auto
elastic@node011:~/elasticsearch-6.1.0$ curl node012:9200/_cat/indices
yellow open .elastichq DZsFO0BUSNOckwp8Bg6T1g 5 1 1 0 7kb 7kb
yellow open .kibana O4fgIRL-SiKSlWSPR9vmQQ 1 1 325 5 276.6kb 276.6kb
yellow open rawType-host15-20180730 CuM6rCtmSlCRZQpwQoNaiA 5 1 485733 0 315.7mb 315.7mb
yellow open rawType-host15-20180731 28TF8H_6Si6nrwUlX7R_pw 5 1 115201 0 72.3mb 72.3mb
```
I opened debug in Logstash, it seems Logstash is still able to receive data from TCP connection. however index size of ES would not increase any more.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2018, 10:31am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/4 "2018-07-31T10:31:31Z")
What does `iostat` give while you are indexing? Anything in the logs? Are your mappings static?
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [July 31, 2018, 11:50am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/5 "2018-07-31T11:50:43Z")
I've not had a chance to do that, I'll try it tomorrow. (iostat)
But in basic monitoring, CPU and Mem are OK (top & free & vmstat)
There is no logs in both ES and LS when message level is info. and we used template to make sure no dynamic mapping, avoiding "convert" in Logstash. By the way, I add "pool\_max" in ES output plugin and resize it to 10000. I don't know whether it would improve performance or not. Do you need other details? thanks in advance.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2018, 11:53am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/6 "2018-07-31T11:53:02Z")
That will allow Elasticsearch to queue up more data, which will use more heap, but is unlikely to speed up ingestion. See [this blog post](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster) for details and a discussion.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [July 31, 2018, 11:56am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/7 "2018-07-31T11:56:30Z")
OK, Got it. But previously I just kept it as default, still failed to ingest data to data node before indexing. Do you think we should put ingest node and data node in the same server to save cost of network communication?
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [July 31, 2018, 12:07pm UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/8 "2018-07-31T12:07:35Z")
I dissected your link, it mentioned "rejected events". When we have some heavy queries using HighLevelRestClient, I saw some "rejected events" indeed in ES log and LS log, indicating \_bulk API was full and some requests haven been denied. But in this case, I see nothing in my log and there is no queuing events in ElasticHQ dashboard.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [August 1, 2018, 8:57am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/9 "2018-08-01T08:57:06Z")
@Christian_Dahlqvist as you suggested. today we have a better monitoring in Elastic 6.3.0 (previous 6.1.0). I set both ES and LS 32GB memory. ES has just a little workload however Logstash used up its memory in a few minutes, about 10m. And our application threw an exception, data did not increase any more. Do you have other ideas?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 1, 2018, 9:14am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/10 "2018-08-01T09:14:45Z")
What does your Logstash config look like? Logstash should not use that much heap even if you have a large batch size and a lot of worker threads. I would recommend enabling debug logging in Logstash to see if anything surfaces.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [August 1, 2018, 9:27am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/11 "2018-08-01T09:27:26Z")
We have nearly 20 config files in one folder. Each filter plugin has a dissect filter to enrich "plain text" to JSON. Previously we used "csv filter" and it seemed it had bad performance than "dissect". Generally, we have a "split" filter for splitting new line, "mutate" to convert some fields lower case and "date" for timestamp matching. Here is a sample:
```auto
filter {
split {
field => "message"
}
mutate {
lowercase => ["host", "sourceType"]
add_field => {
"SYSPLEX_SYSTEM" => "%{sysplexName}.%{systemName}"
}
}
}
filter {
if [sourceType] != "syslog-console" {
date {
match => ["TIMESTAMP", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
timezone => "%{timeZone}"
target => "@timestamp"
}
}
}
filter {
if [sourceType] == "special_type" {
if [message] {
dissect {
mapping => {
"message" => "%{},%{},%{TIMESTAMP},%{MVS_SYSTEM_ID},%{DB2_SYSTEM_ID},%{PRIMARY_AUTH_ID},%{END_USER_USERID},%{END_USER_TRAN},%{END_USER_WORKSTN},%{SSID},%{AUTHID},%{CORRELATIONID},%{PLAN},%{LOCK_LATCH_SEC},%{LOCK_LATCH_EVENTS},%{IRLM_LOCKLATCH_SEC},%{DB2_LATCH_SEC},%{ELAPSED_SEC_CL1},%{CP_CPU_SEC_CL1},%{COMMIT_COUNT},%{ABORT_COUNT},%{SQLCALL_SEC_INSP},%{UDF_REQUESTS_SEC},%{IO_WAIT_SEC},%{BP4K_GETPAGE},%{BP32_GETPAGE},%{BP8K_GETPAGE},%{BP16_GETPAGE}"
}
}
mutate {
add_field => { "tags" => "pipeline" }
}
}
else {
mutate {
add_field => { "tags" => "no_message" }
}
}
}
}
```
I also opened debug to print a large nohup log. Not many useful info. Our scenario is sending plenty of data in a short time with TCP connection, and continuing on in 1 minute interval. Each interval may have 700K events. The data size is 0.6K per one event in Linux system by index storage size / doc count, the single JSON files is 4K in filesystem which copy from Kibana document.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [August 2, 2018, 12:05am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/12 "2018-08-02T00:05:37Z")
@Christian_Dahlqvist by the way, I tried to resize batch size \* workers in Logstash at several times from 8000 to 36000 in a single flight. It seems no impact to performance. Logstash was always busy.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 2, 2018, 4:06am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/13 "2018-08-02T04:06:15Z")
I was suggesting decreasing it rather than increasing it. Larger values does necessarily not always give better performance.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [August 7, 2018, 8:04am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/14 "2018-08-07T08:04:06Z")
@Christian_Dahlqvist Thanks for your reply, as we tested, the bottle neck is output to ES. If we just change output to a file or stdout, then it works. Any ideas to optimize ES bulk? I can not use flush\_size in LS 6.0 and above.
---
### Author:  [@supergunme](https://discuss.elastic.co/u/supergunme)
#### Post date: [August 21, 2018, 2:31am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/15 "2018-08-21T02:31:17Z")
Can we still have a discussion here or post a new topic 😃
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 21, 2018, 5:22am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/16 "2018-08-21T05:22:02Z")
You will need to look at the Elasticsearch cluster and try to find what is limiting performance. Is it CPU or memory on the ingest node? Is it CPU or perhaps disk I/O on the data nodes?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 18, 2018, 5:22am UTC](https://discuss.elastic.co/t/huge-concurrent-data-ingestion-to-elasticsearch/142318/17 "2018-09-18T05:22:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.