I have the numbers but I'm not sure about the field "@timestamp", it represents the "ingestion time", but from where? The ingestion into Logstash or Elasticsearch?
Indexing in Elasticsearch
Once Logstash is done, data are shipped to Elasticsearch, but how do I know how much time does it takes?
The easiest way to get those numbers is to use the stack monitoing as this can show you how much time each part of your logstash pipeline is taking.
The value of the @timestamp field depends entirely on your pipelin, this field is normally used to get the event time and could be not related to processing time, for example, if you have a date filter applied on a field in your document and do not specify the target for this filter, logstash will store the parsed value in the @timestamp field.
If you are not using the date filter to change the value of the @timestamp field, then the value of this field will be the time when the event exited the input part of your pipeline.
I'm using both, the ingest @timestamp and the one in the logs.
The stack monitoring has enough details your are right, there are also information directly on the index (tab details) I did not see it at the beginning (everything were at 0), maybe because I restarted the pipeline or those were old data...
Now I have another question, not exactly link to the topic:
I noticed (thanks to the stack monitoring) that the JVM size is around 16 GB (that's normal, 50% of my ram), but when the pipeline is processing new data (from logs files), the JVM Heap never use more than 9GB, with an average around 6GB.
It is normal / advice to keep it so low? Or should I configure it somewhere, to allow it to use more power?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.