Excessive time to recreate the index

Hi!

I am working with Kibana and I have some problems with my index.

I use two index one is allways correct and never lost data.
These index are made by a consult to PosgreSQL this only take 100ms but on kibana the data disappear for 10 min aprox until it recreate the index.

We have checked the logs and it doesn´t seem nothing unusual.

Additional information:

  • Kibana version: 6.8.23
  • Elasticsearch version: 6.8.23
  • Lucene version: 7.7.3
  • Logstash version: 6.8.23
  • jdbc version: 42.3.3

Thank you in advance for your help.

Hi @hrodriguez. I don't have enough information from you. Can you share your Logstash configuration?

Of course.

On logstash.yml :

# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
path.data: /var/lib/logstash

# ------------ Debugging Settings --------------
#
# Options for log.level:
#   * fatal
#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
# log.level: info
path.logs: /var/log/logstash

And the .conf that has the problem is:


input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://localhost:5432/my_ddbb"
jdbc_user => "postgres"
jdbc_password => "****"
jdbc_driver_class => "org.postgresql.Driver"
clean_run => false
use_column_value => true
tracking_column => "column"
schedule => "*/30 * * * *"
statement_filepath => "/opt/elastic/indexes/my_select.sql"
}
}
output {
elasticsearch {
hosts => ["http://my_host"]
index => "my_index"
document_id => "my_index%{column}"
doc_as_upsert => true
#user => “es_user”
#password => “es_password”
}
}

If you need further information do not hesitate to ask for it.

Thanks. If I understand correctly, your schedule is running every 30 minutes. Is that not what is causing the issue?

I doubt it, because I have tried with different schedules, without it and had the same problem. It seems that when the other index runs this one too.

Hi @hrodriguez. I'm not sure if this is an issue with Kibana or Logstash? Does the data exist in Elasticsearch, but not appear in Kibana? If the data is not getting added to Elasticsearch we may want to move this to the Logstash forum for a wider audience.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.