# Array datatype limit size?

**URL:** <https://discuss.elastic.co/t/array-datatype-limit-size/158654>\
**Category:** Logstash\
**Created:** [November 28, 2018, 9:27pm UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654 "2018-11-28T21:27:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [November 28, 2018, 9:27pm UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/1 "2018-11-28T21:27:33Z")

</div>

Hi all,  
I'm trying to collect information about login of accounts on a website. Currently I'm able to save, for instance, the last 10 login attempts using the `array` datatype.  
To do that I use Filebeat that sends messages to Logstash to grok strings.  
Is there a way to limit the array's size?

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [November 30, 2018, 5:56pm UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/2 "2018-11-30T17:56:00Z")

</div>

This is the logstash.conf used right now:

```
input { 
	beats {
		client_inactivity_timeout => 600
		port => 5044
	}
}

filter {
 grok {
        break_on_match => false
		patterns_dir => [".\patterns"]
        match => [
            "message", ".*\s%{TT:timestamp}.*login\=\"%{USERNAME:username}\""
        ]
    }
 date {
   match => ["timestamp", "yyyyMMdd HHmmss"]
 }

mutate {
    add_field => {
	 "logins" => ["%{@timestamp}"]
	}
	remove_field => ["timestamp"]
}
}

output {
  elasticsearch { 		
   hosts => ["localhost:9200"]
   index => "ftaudit"
   document_id => "%{[username]}_%{[beat][name]}"
  }
 }

```

unfortunately, when add\_field is performed, it will overwrite the previous value. can anyone help me?  
@magnusbaeck maybe you could know the answer?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 30, 2018, 6:15pm UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/3 "2018-11-30T18:15:20Z")

</div>

Please don't ping people who are not yet involved in the thread.  
I moved your question to #logstash.

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [November 30, 2018, 6:52pm UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/4 "2018-11-30T18:52:04Z")

</div>

sorry @dadoonet.  
at the end I modified the output section of my logstash.conf in this way:

```
 output {
  elasticsearch { 		
   hosts => ["localhost:9200"]
   index => "ftaudit"
   document_id => "%{[username]}_%{[beat][name]}"
   doc_as_upsert => true
   action => "update"
   script => 'ctx._source.logins += ";""%{@timestamp}"'
  }
 }

```

but I'm facing this error:

> [2018-11-30T19:37:46,744][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["update", {:\_id=\>"NoTmanaGe\_CE", :\_index=\>"ftaudit", :\_type=\>"doc", :\_routing=\>nil, :\_retry\_on\_conflict=\>1}, #LogStash::Event:0x28acefed], :response=\>{"update"=\>{"\_index"=\>"ftaudit", "\_type"=\>"doc", "\_id"=\>"NoTmanaGe\_CE\_dxfbft03", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [logins] of type [date]", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: "2018-11-30T16:42:10.000Z2018-11-30T16:41:30.000Z" is malformed at "2018-11-30T16:41:30.000Z""}}}}}

so basically, in my index there is only the first `document_id` of each record. I'm not understanding how to manage my `login` field, composed by several `date`

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [December 5, 2018, 8:28am UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/5 "2018-12-05T08:28:50Z")

</div>

just an update. I changed the output section of `logstash.conf` in this way:

```
output {
  elasticsearch { 		
   hosts => ["localhost:9200"]
   index => "ftaudit"
   document_id => "%{[username]}_%{[beat][name]}"
   doc_as_upsert => true
   action => "update"
   script => 'ctx._source.logins.add(%{@timestamp}); if(ctx._source.logins.length > 10) { ctx._source.logins.remove(0); }'
  }

```

it seems to be the correct way but I'm facing this error:

> [2018-12-04T17:48:29,014][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["update", {:\_id=\>"NoTmanaGe\_CE\_dxfbft03", :\_index=\>"ftaudit", :\_type=\>"doc", :\_routing=\>nil, :\_retry\_on\_conflict=\>1}, #LogStash::Event:0x3ec54e10], :response=\>{"update"=\>{"\_index"=\>"ftaudit", "\_type"=\>"doc", "\_id"=\>"NoTmanaGe\_CE\_ftp", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"failed to execute script", "caused\_by"=\>{"type"=\>"script\_exception", "reason"=\>"compile error", "script\_stack"=\>["... ource.list.add(2018-11-30T16:52:41.000Z)", " ^---- HERE"], "script"=\>"ctx.\_source.list.add(2018-11-30T16:52:41.000Z)", "lang"=\>"painless", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"invalid sequence of tokens near ['T16'].", "caused\_by"=\>{"type"=\>"no\_viable\_alt\_exception", "reason"=\>nil}}}}}}}

here it is also the mapping of my index:

```
{
  "ftaudit" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "software" : {
            "properties" : {
              "message" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          },
          "beat" : {
            "properties" : {
              "hostname" : {
                "type" : "keyword"
              },
              "name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "version" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          },
          "client" : {
            "properties" : {
              "ip" : {
                "type" : "keyword"
              },
              "port" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          },
          "filebeat_source" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "host" : {
            "properties" : {
              "name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          },
          "logins" : {
            "type" : "date"
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "offset" : {
            "type" : "long"
          },
          "server" : {
            "properties" : {
              "ip" : {
                "type" : "keyword"
              },
              "port" : {
                "type" : "keyword"
              }
            }
          },
          "tags" : {
            "type" : "keyword"
          },
          "username" : {
            "type" : "keyword"
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2019, 8:28am UTC](https://discuss.elastic.co/t/array-datatype-limit-size/158654/6 "2019-01-02T08:28:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.