Hello,
I have encounters a weird problem when adding entries to Elasticsearch through the Curl command.
The problem however seemingly appears in Kibana.
The "bug" is that for somereason kibana adds 1 hour to the timestamp.
Example I have an entry with the timestamp : 07:08:07
But in the discover tab I can not see the even unless my timefram contains 08:08:07
Here is an image of the problem:
I can mention as it might be of interest, that the events was upploaded to elasticsearch via the curl command. And that the file was originally a pcap file that converted to a json file through the tshark pcap conversion tool as seen here :
Here is the json entry for the specific entry that we are looking at:
{"index":{"_index":"packets-2019-12-02","_type":"doc"}}
{"timestamp":"1575270487730","layers":{"frame":
{"frame_frame_encap_type":"25","frame_frame_time":"2019-12-02T07:08:07.730535000Z","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1575270487.730535000","frame_frame_time_delta":"0.000160000","frame_frame_time_delta_displayed":"0.000160000","frame_frame_time_relative":"0.000160000","frame_frame_number":"2","frame_frame_len":"64","frame_frame_cap_len":"64","frame_frame_marked":false,"frame_frame_ignored":false,"frame_frame_protocols":"sll:ethertype:ip:sctp"},"sll":{"sll_sll_pkttype":"4","sll_sll_hatype":"1","sll_sll_halen":"6","sll_sll_src_eth":"fa:16:3e:c9:e0:11","sll_sll_unused":"00:00","sll_sll_etype":"0x00000800"},"ip":{"ip_ip_version":"4","ip_ip_hdr_len":"20","ip_ip_dsfield":"0x00000000","ip_ip_dsfield_dscp":"0","ip_ip_dsfield_ecn":"0","ip_ip_len":"48","ip_ip_id":"0x000054e9","ip_ip_flags":"0x00004000","ip_ip_flags_rb":false,"ip_ip_flags_df":true,"ip_ip_flags_mf":false,"ip_ip_frag_offset":"0","ip_ip_ttl":"64","ip_ip_proto":"132","ip_ip_checksum":"0x00006222","ip_ip_checksum_status":"2","ip_ip_src":"172.2.21.155","ip_ip_addr":["172.2.21.155","172.2.21.159"],"ip_ip_src_host":"172.2.21.155","ip_ip_host":["172.2.21.155","172.2.21.159"],"ip_ip_dst":"172.2.21.159","ip_ip_dst_host":"172.2.21.159"},"sctp":{"sctp_sctp_srcport":"3868","sctp_sctp_dstport":"6868","sctp_sctp_verification_tag":"0x00bb8ef2","sctp_sctp_assoc_index":"65535","sctp_sctp_port":["3868","6868"],"sctp_sctp_checksum":"0x4316605b","sctp_sctp_checksum_status":"2","text":"SACK chunk (Cumulative TSN: 12302063, a_rwnd: 8192, gaps: 0, duplicate TSNs: 0)","sctp_sctp_chunk_type":"3","sctp_sctp_chunk_bit_1":false,"sctp_sctp_chunk_bit_2":false,"sctp_sctp_chunk_flags":"0x00000000","sctp_sctp_sack_nounce_sum":"0","sctp_sctp_chunk_length":"16","sctp_sctp_sack_cumulative_tsn_ack":"12302063","sctp_sctp_ack":["12302060","12302061","12302062","12302063"],"sctp_sctp_ack_frame":["1","1","1","1"],"sctp_sctp_sack_rtt":["0.000160000","0.000160000","0.000160000","0.000160000"],"sctp_sctp_sack_a_rwnd":"8192","sctp_sctp_sack_number_of_gap_blocks":"0","sctp_sctp_sack_number_of_duplicated_tsns":"0"}}}
This is my curl command :
curl -s -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/test_one/_bulk" --data-binary "@test.json"; echo
Now I'm only showing one entry but I get this error with all of my entries that I add to elasticsearch with this method. And it is always 1 hour that it adds.
It might also be of interest, that as one can see in the images, that the entries does not contain a timestamp field but instead use the layers.frame.frame_frame_time field for its timestamp value.
After checking the visualized data from other files I can see that the same bug appears for them as well in kibana. Where the data in elasticsearch corresponds to the timestamp in the original files but in kibana they for some reason seem to say that the timestamp is the original timestamp + 1 hour for deciding time range in the discover, visualization and dashboard tab. These other files I have looked at a system log files sent to elasticsearch through filebeat -> logstash.