Could I remove _type field?

Elastic Stack Elasticsearch
1

Hi, due to the high loading, I want to remove the default field _type, _score, _id.
is it possible ? if can, how to do it?

thanks a lot.

2

Either of those three are necessary for Elasticsearch to index (_type) or to search (_id, _score) your data.
I don't think it is possible.

3

ok, thanks for your reply :slight_smile:

4

REmove from what? The search response?

5

from logstash. because I found my hard drive is too late to write data .

6

So you can't remove those fields. Note that _score is not stored on disk and just computed at search time.

I think you want to solve an issue but using the wrong way of doing so.
May be explain what the problem is instead?

2 Likes
7

I store data through logstash into ES, and I found that the data can't write to ES in time, so there has drop flow.After I remove some fields which I don't need, and the collecting performance is better.that's why I want to remove the default field.
maybe my hard drive isn't good enough, but it seems not bad.

hdparm -t --direct /dev/sda

/dev/sda:
 Timing O_DIRECT disk reads: 6524 MB in  3.00 seconds = 2174.35 MB/sec
8

What does disk I/O and iowait look like while indexing? Have you tuned your cluster for indexing speed?

9

[root@10-255-254-24 ~]# hdparm -t --direct /dev/sda

/dev/sda:
Timing O_DIRECT disk reads: 7518 MB in 3.00 seconds = 2505.74 MB/sec
[root@10-255-254-24 ~]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 2360 MB in 3.03 seconds = 778.64 MB/sec

iowait is about 0.1

now the flow is half of the peak.

10

What does iostat -x show?

Elasticsearch tends to read and write a lot of small files rather than a few large ones, so a performance test using large sequential reads and/or writes may not represent how Elasticsearch used disk I/O very well.

11 
Linux 3.10.0-693.el7.x86_64 (10-255-254-24.privateip.tyc.edu.tw)        03/21/2018      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.84    0.92    0.48    0.02    0.00   97.74

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.04     0.29    1.59    8.04   312.97   140.40    94.15     0.01    1.03    4.91    0.26   0.28   0.27
scd0              0.00     0.00    0.00    0.00     0.01     0.00    72.41     0.00    0.38    0.38    0.00   0.38   0.00
dm-0              0.00     0.00    0.28    0.23    15.77     3.24    75.36     0.00    2.92    5.20    0.14   0.94   0.05
dm-1              0.00     0.00    0.05    0.22     0.22     0.87     8.15     0.00    2.49    4.92    1.94   0.16   0.00
dm-2              0.00     0.00    0.97    7.89   126.20   136.28    59.26     0.01    0.91    6.16    0.26   0.24   0.22
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.