# Logs are overwritten in the specified index under the same \_id **URL:** \ **Category:** Logstash\ **Created:** [May 22, 2019, 10:59am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208 "2019-05-22T10:59:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![madhanbaskar](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Post date:** [May 22, 2019, 10:59am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/1 "2019-05-22T10:59:08Z")
Hi There, I'm using Logstash - 6.5.1 and elasticsearch - 6.5.1. Below is my Filebeat.yml filebeat.prospectors: - type: log paths: - type: log paths: - type: log paths: Below is my logstash config file - input { beats { port =\> 5044 tags =\> ["ApacheAccessLogs", "ApacheErrorLogs", "MysqlErrorLogs"] } } filter { if "ApacheAccessLogs" in [tags] { grok { match =\> [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}", "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra\_fields}" ] overwrite =\> ["message"] } mutate { convert =\> ["response", "integer"] convert =\> ["bytes", "integer"] convert =\> ["responsetime", "float"] } geoip { source =\> "clientip" target =\> "geoip" add\_tag =\> ["apache-geoip"] } date { match =\> ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"] remove\_field =\> ["timestamp"] } useragent { source =\> "agent" } } if "ApacheErrorLogs" in [tags] { grok { match =\> { "message" =\> ["[%{APACHE\_TIME:[apache2][error][timestamp]}] [%{LOGLEVEL:[apache2][error][level]}]( [client %{IPORHOST:[apache2][error][client]}])? %{GREEDYDATA:[apache2][error][message]}", "[%{APACHE\_TIME:[apache2][error][timestamp]}] [%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}] [pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?]( [client %{IPORHOST:[apache2][error][client]}])? %{GREEDYDATA:[apache2][error][message1]}" ] } pattern\_definitions =\> { "APACHE\_TIME" =\> "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" } remove\_field =\> "message" } mutate { rename =\> { "[apache2][error][message1]" =\> "[apache2][error][message]" } } date { match =\> ["[apache2][error][timestamp]", "EEE MMM dd H:m:s YYYY", "EEE MMM dd H:m:s.SSSSSS YYYY" ] remove\_field =\> "[apache2][error][timestamp]" } } if "MysqlErrorLogs" in [tags] { grok { match =\> { "message" =\> ["%{LOCALDATETIME:[mysql][error][timestamp]} ([%{DATA:[mysql][error][level]}] )?%{GREEDYDATA:[mysql][error][message]}", "%{TIMESTAMP\_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread\_id]} [%{DATA:[mysql][error][level]}] %{GREEDYDATA:[mysql][error][message1]}", "%{GREEDYDATA:[mysql][error][message2]}"] } pattern\_definitions =\> { "LOCALDATETIME" =\> "[0-9]+ %{TIME}" } remove\_field =\> "message" } mutate { rename =\> { "[mysql][error][message1]" =\> "[mysql][error][message]" } } mutate { rename =\> { "[mysql][error][message2]" =\> "[mysql][error][message]" } } date { match =\> ["[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s" ] remove\_field =\> "[apache2][access][time]" } } } output { if "ApacheAccessLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_type =\> "apacheaccess" } } if "ApacheErrorLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_id =\> "apacheerror" } } if "MysqlErrorLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_type =\> "sqlerror" } } stdout { codec =\> rubydebug } } The data is sent to elastic search but only 3 records are getting created for each document\_id in the same index. Only 3 records are created and every new logs incoming are overwritten onto the same document\_id and the old one is lost. Can you guys please help me out? @magnusbaeck ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2019, 11:39am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/2 "2019-05-22T11:39:12Z")
You are specifying multiple document types for the same index which would cause errors for recent Elasticsearch versions. You also have a fixed document id specified for one output which will cause the same document to be updated repeatedly. ---
**Author:** ![madhanbaskar](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Post date:** [May 22, 2019, 11:43am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/3 "2019-05-22T11:43:39Z")
@Christian_Dahlqvist - What is the best way to split data, which field will help me out instead of document\_type or document\_id? Also my exact output block is - output { if "ApacheAccessLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_id =\> "apacheaccess" } } if "ApacheErrorLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_id =\> "apacheerror" } } if "MysqlErrorLogs" in [tags] { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "apache" document\_id =\> "sqlerror" } } stdout { codec =\> rubydebug } } I'm using only the document\_id! How can I write my output block inorder to avoid overwriting? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2019, 1:06pm UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/4 "2019-05-22T13:06:23Z")
You can not use document id that was as it is a unique identifier for each document. Remove it and let Elasticsearch assign it. ---
**Author:** ![madhanbaskar](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Post date:** [May 22, 2019, 1:28pm UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/5 "2019-05-22T13:28:45Z")
@Christian_Dahlqvist : Thats Right... But what if there are 2 fields of the same name from 2 different sources? That will clash right? I need to put all the data into one index & I should have another field which helps me to segragate data of one source from other 2 sources.. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 22, 2019, 2:13pm UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/6 "2019-05-22T14:13:45Z")
> [@madhanbaskar](#): > > I need to put all the data into one index What is driving this requirement? You can run queries against multiple indexes. You are likely to be better off keeping different document types in different indexes. You should read about [why](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html) document types are being removed from elasticsearch. ---
**Author:** ![madhanbaskar](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Post date:** [May 23, 2019, 7:38am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/7 "2019-05-23T07:38:17Z")
@Badger : Ok I shall use separate Index for each source. Thanks! Specifying individual index works well!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2019, 7:38am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208/8 "2019-06-20T07:38:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.