# I'm receiving just one log **URL:** \ **Category:** Kibana\ **Created:** [July 5, 2018, 9:32am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674 "2018-07-05T09:32:25Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 5, 2018, 9:32am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/1 "2018-07-05T09:32:25Z")
Hi, Please I don't why Kibana is receinving juste one log and when I refresh the timestamp changes ( It means that he is receing logs but just one and deleting other :o ) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b95171f250a65c2afe1e186d7f21dcf525c9b31d.png) ---
**Author:** ![Bill\_McConaghy](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@Bill\_McConaghy](https://discuss.elastic.co/u/Bill_McConaghy)\ **Post date:** [July 5, 2018, 12:34pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/2 "2018-07-05T12:34:34Z")
In that screenshot the time filter is set to the last 15 minutes. If you try picking last day do you see more data? If that does not fix your issue, I'll need a lot more details in order to help you find a solution. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 5, 2018, 2:23pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/3 "2018-07-05T14:23:33Z")
Event when I set the time to see all logs, I m stiil seing juste one log !! ---
**Author:** ![Bill\_McConaghy](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@Bill\_McConaghy](https://discuss.elastic.co/u/Bill_McConaghy)\ **Post date:** [July 5, 2018, 5:31pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/4 "2018-07-05T17:31:55Z")
Seems like maybe you have set up whatever is sending the logs to Elasticsearch to use the same document ID. This would result in the behavior you are seeing (only 1 doc). If you go to console in Kibana and do a search on the index, `GET yourindexname/_search`, what doc count comes back? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 5, 2018, 5:34pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/5 "2018-07-05T17:34:28Z")
Are you assigning your own ID in the ingest pipeline? Any chance there is something wrong with this logic so all documents gets indexed using the same ID? You should be able to look at the ID of the event you can see to see if this is the case. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 7:42am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/6 "2018-07-06T07:42:08Z")
> [@Bill\_McConaghy](#): > > `GET yourindexname/_search` Here is the result : ```auto { "error": { "root_cause": [ { "type": "index_not_found_exception", "reason": "no such index", "resource.type": "index_or_alias", "resource.id": "yourindexname", "index_uuid": "_na_", "index": "yourindexname" } ], "type": "index_not_found_exception", "reason": "no such index", "resource.type": "index_or_alias", "resource.id": "yourindexname", "index_uuid": "_na_", "index": "yourindexname" }, "status": 404 } ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 7:43am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/7 "2018-07-06T07:43:12Z")
I'm using fingerprint to avoid duplicate data, but it was working very well until yesterday !! > [@Christian\_Dahlqvist](#): > > ID ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 8:36am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/8 "2018-07-06T08:36:34Z")
@dadoonet I removed : ```auto fingerprint { method => "SHA1" key => "KEY" } ``` It works I'm receiving all logs but duplicated 😕 !! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 6, 2018, 10:10am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/9 "2018-07-06T10:10:50Z")
I did not say to remove it but to put it on the top of the filters. But I feel that if the timestamp of the event (when filebeat collected the logs) is different it will generate duplicates. When back to my keyboard, I'll try to give you some ideas to fix it. As I told you yesterday it's more important IMO that you fix the date filter problem you are having. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2018, 10:27am UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/10 "2018-07-06T10:27:08Z")
Typically you run the hash based on the content of the `message` field early on, so that the automatically assigned `@timestamp` field does not come into play. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 12:26pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/11 "2018-07-06T12:26:24Z")
I already put it on the top of the filter, it doesn't change anything, the logs were duplicated 😕 . Can you show me how to fix the date, please ? ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 12:27pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/12 "2018-07-06T12:27:16Z")
Do you know how to solve this, beacause i'm stuck !! do you need my conf of logstash ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2018, 12:36pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/13 "2018-07-06T12:36:59Z")
Did you specify to use just the message field as source for the hash: `source => "message"` ? ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 12:41pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/14 "2018-07-06T12:41:47Z")
This is my file.conf ```auto input { beats { port => "5044" type => "%{[fields] [log_type]}" } } filter { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:hostname} %{GREEDYDATA:data}"} } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } if [fields][log_type] == "fortigate" { kv { source => "data" } mutate { remove_field => ["data"] rename => {"dstip" => "dst_ip"} rename => {"dstport" => "dst_port"} rename => {"proto" => "protocol"} rename => {"dstintf" => "dst_interface"} rename => {"srcport" => "src_port"} rename => {"srcip" => "src_ip"} rename => {"srcintf" => "src_interface"} } if [protocol] == "6" { mutate { replace => {"protocol" => "tcp" }} } if [protocol] == "17" { mutate { replace => {"protocol" => "udp" }} } if [protocol] == "58" { mutate { replace => {"protocol" => "ipv6-icmp" }} } } if [fields][log_type] == "cisco_asa" { grok { match => { "data" => "%{CISCOTAG:cisco_tag}: %{GREEDYDATA:cisco_message}"} } grok { match => [ "cisco_message", "", "cisco_message", "%{CISCOFW106006_106007_106010}", "cisco_message", "%{CISCOFW106014}", "cisco_message", "%{CISCOFW106015}", "cisco_message", "%{CISCOFW106021}", "cisco_message", "%{CISCOFW106023}", "cisco_message", "%{CISCOFW106100}", "cisco_message", "%{CISCOFW110002}", "cisco_message", "%{CISCOFW302010}", "cisco_message", "%{CISCOFW302013_302014_302015_302016}", "cisco_message", "%{CISCOFW302020_302021}", "cisco_message", "%{CISCOFW305011}", "cisco_message", "%{CISCOFW313001_313004_313008}", "cisco_message", "%{CISCOFW313005}", "cisco_message", "%{CISCOFW402117}", "cisco_message", "%{CISCOFW402119}", "cisco_message", "%{CISCOFW419001}", "cisco_message", "%{CISCOFW419002}", "cisco_message", "%{CISCOFW500004}", "cisco_message", "%{CISCOFW602303_602304}", "cisco_message", "%{CISCOFW710001_710002_710003_710005_710006}", "cisco_message", "%{CISCOFW713172}", "cisco_message", "%{CISCOFW733100}", "cisco_message", "%{WORD:action} %{WORD:protocol} %{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}; %{GREEDYDATA:dnssec_validation}", "cisco_message", "%{CISCO_ACTION:action} %{WORD:protocol} %{CISCO_REASON:reason}.*(%{IP:src_ip}).*%{IP:dst_ip} on interface %{GREEDYDATA:interface}", "cisco_message", "Connection limit exceeded %{INT:inuse_connections}/%{INT:connection_limit} for input packet from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} on interface %{GREEDYDATA:interface}", "cisco_message", "TCP Intercept %{DATA:threat_detection} to %{IP:ext_nat_ip}/%{INT:ext_nat_port}.*(%{IP:int_nat_ip}/%{INT:int_nat_port}).*Average rate of %{INT:syn_avg_rate} SYNs/sec exceeded the threshold of %{INT:syn_threshold}.#%{INT}", "cisco_message", "Embryonic connection limit exceeded %{INT:econns}/%{INT:limit} for %{WORD:direction} packet from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} on interface %{GREEDYDATA:interface}" ] } mutate { remove_field => ["data"] remove_field => ["cisco_message"] lowercase => ["protocol"] } } else if [fields][log_type] == "paloalto" { csv { source => "data" columns => ["FUTURE_USE", "Receive Time", "Serial Number", "Type", "Subtype", "FUTURE_USE", "Generated Time", "src_ip", "dst_ip", "NAT Source IP", "NAT Source IP", "dst_mapped_ip", "Rule Name", "Source User", "Destination User", "Application", "Virtual System", "Source Zone", "Destination Zone", "src_interface", "dst_interface", "Log Forwarding Profile", "FUTURE_USE", "Session ID", "Repeat Count", "src_port", "dst_port", "src_mapped_ip", "NAT Destination Port", "Flags", "protocol", "action", "Bytes", "Bytes Sent", "Bytes Received", "Packets", "Start Time", "Elapsed Time", "Category", "FUTURE_USE", "Sequence Number", "Action Flags", "Source Location", "Destination Location", "FUTURE_USE", "Packets Sent", "Packets Received", "Session End Reason", "Device Group Hierarchy Level 1", "Device Group Hierarchy Level 2", "Device Group Hierarcherarchy Level 3", "Device Group Hierarchy Level 4", "Virtual System Name", "hostname", "Action Source"] } mutate { remove_field => ["data"] } } # Eviter la duplication des logs fingerprint { method => "SHA1" key => "KEY" } } output { elasticsearch { hosts => ["localhost:9200"] document_id => "%{fingerprint}" } } ``` What do I have to change please !! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2018, 12:46pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/15 "2018-07-06T12:46:29Z")
Try this: ```auto fingerprint { method => "SHA1" key => "KEY" source => "message" } ``` ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 12:54pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/16 "2018-07-06T12:54:39Z")
I still have the same problem 😕 !! I don't understand why !! ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9ef930ad2c63428262580ed11b329dd4df570455.png) Or is there an other way to avoid duplicated logs ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2018, 12:57pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/17 "2018-07-06T12:57:00Z")
That screen shot shows a single document, not duplicates. Not sure I understand. ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 12:57pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/18 "2018-07-06T12:57:45Z")
Yes I know !! When I use : fingerprint : I have one log When I remove it : I receive all logs duplicated !! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2018, 12:58pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/19 "2018-07-06T12:58:54Z")
Can you show us the full event you have? Are you removing the `message` field somewhere in your config (did not see it in what you posted)? ---
**Author:** ![asalma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asalma/32/32199_2.png) [@asalma](https://discuss.elastic.co/u/asalma)\ **Post date:** [July 6, 2018, 1:04pm UTC](https://discuss.elastic.co/t/im-receiving-just-one-log/138674/20 "2018-07-06T13:04:12Z")
I didn't get you, do you mean I have to remove this line ? ```auto mutate { remove_field => ["data"] } ``` I also have this WARN when testing the conf : ```auto [root@frghcslnetv10 conf.d]# /usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash -f /etc/logstash/conf.d/firewalls.conf Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties [2018-07-06T15:03:25,517][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-07-06T15:03:35,257][INFO][logstash.runner] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash ``` [Next page](https://discuss.elastic.co/t/im-receiving-just-one-log/138674.md?page=2)