Document count is same but index size is growing every logstash run

Elastic Stack Elasticsearch
1 
input {
	jdbc {
		jdbc_connection_string => "jdbc:mysql://******.rds.amazonaws.com:3306/"
		jdbc_user => "user"
		jdbc_password => "pass"
		jdbc_driver_library => "****\mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar"
		jdbc_driver_class => "com.mysql.jdbc.Driver"
		statement => "SELECT id,title,url FROM tableName"
		schedule => "*/2 * * * *"
	}
}
filter {
  json {
    source => "texts"
    target => "texts"
  }
  mutate { remove_field => [ "@version", "@timestamp" ] }
}
output {
    stdout {
            codec => json_lines
    }
    amazon_es {
      hosts => ["*****.es.amazonaws.com"]
      document_id => "%{id}"
      index => "texts"
      region => "****"
      aws_access_key_id => '*****'
      aws_secret_access_key => '*****'
  }  
}

Here is the logstash.conf file .
how can I solve this problem? The index size is growing every two minutes.

The first run had 333 documents. The size is approximately 200kb.

Now also the size is 1.6mb
Ba%C5%9Fl%C4%B1ks%C4%B1z-2

2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Here I guess this is because the number of segments has increased a bit.
You can try to run https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html if this is really a problem for you.

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, APM, Logs UI, Infra UI, SIEM, Maps UI and what is coming next :slight_smile: ...

3

Not to answering your question but
your schedule seems wrong it should be */2 * * * * ( five field)

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.