Elasticsearch 5.6 - Query Return Wrong Data

Elastic Stack Elasticsearch
1

Hi ,
We have EasticSearch Index which store nested json data .
Its ES 5.6 we exceeded the default limitaton of number of fields and nested fields so our mapping contain :
"mapping": {
"nested_fields": {
"limit": "200"
},
"total_fields": {
"limit": "15000"
},
"depth": {
"limit": "100"
}
}
}

The query that brings wrong data is -

GET ltm/_search
{

"query": {

"bool": {

  "must": [

    {

      "term": {

        "modbus_rtu.function": 21

      }

    },

    {

      "term": {

        "general_info.protocol" : "modbus_rtu"

      }

    }

  ]

}

}

}

The record that we are looking to find is -

PUT ltm/insight/1
{
"frame": {
"protocols": "eth:ethertype:ip:tcp:mbrtu:modbus:modbus:modbus",
"encap_type": 1,
"len": 70
},
"eth": {
"destination_address": "cc:ef:48:cb:50:00",
"source_address": "80:c1:6e:6f:9a:c8",
"type": "0x00000800",
"type_meaning": "Internet Protocol version 4 (IPv4)"
},
"ip": {
"version": 4,
"hdr_len": 20,
"ds_field": "0x00000000",
"len": 56,
"id": "0x000052b7",
"flags": "0x00000002",
"frag_offset": 0,
"ttl": 64,
"proto": 6,
"proto_meaning": "TCP",
"checksum": "0x000099de",
"src": "0xc0a86f57",
"src_meaning": "192.168.111.87",
"dst": "0xc0a85d82",
"dst_meaning": "192.168.93.130",
"version_meaning": "IPv4"
},
"tcp": {
"srcport": 48891,
"dstport": 502,
"seq": 9,
"ack": 1,
"hdr_len": 20,
"flags": "0x00000019",
"window_size": 258,
"checksum": "0x0000ba29",
"urgent": 0,
"flags_meaning": "Flags: 0x019 (FIN, PSH, ACK)"
},
"mbrtu": {
"direction": "query",
"unit_id": 27,
"crc16": "0x00000143",
"crc16_status": 2,
"dest_uid": 27
},
"modbus_rtu": {
"byte_cnt": 0,
"function": 21,
"modbus_meaning": "Modbus",
"byte_cnt_meaning": "Byte Count: 0",
"func_code_meaning": " Function Code: Get Comm. Event Log (12)"
},
"general_info": {
"protocol": "modbus_rtu",
"direction": "client_to_server",
"malformed": "True",
"blackbox_name": "BBX_10.20.30.60",
"blackbox_id": 2,
"UTC_timestamp": "2018-06-06T08:36:35.743Z",
"local_timestamp": "2018-06-06T11:36:35.743Z",
"epoch": 1528274195.743346,
"archive": 0,
"read": 0,
"type": "insight"
}
}

The query return also wrong records with - "modbus_rtu.function":12

When we write the Json data to different index there is no problem .
Re-indexing the whole index to different one we get the same problem

POST _reindex
{
"source": {
"index": "ltm"
},
"dest": {
"index": "ltm_new"
}
}
Note - I have set the ltm_new index mapping to support more then 1500 fields , etc before re-indxing ,
So it seems that the problem occurrs when the problematic record is stored as part of other long Json Records .

We did not had such problem in ES 2.4 - I'm going to double check it .

Did You face this problem ? What can cause it ? How is it possible to solve it

Thanks

Alon

2

This is really strange issue . We have an engine in C++ that write the data using curl into elastic . Most of the fields , we have many > 1000 are OK . This specific field return wrong result data .
This issue exists in ES 5 and 2.4 .

I have tested it using Python API - Pull all data into 1 file containing many JSON and then re-index them to different index .Data was OK . No Query wrong results .

This is really mystery .

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.