Below is the log message that have captured to the ELK from the server , i have configured the filebeat on july 24th ,the below date July 24th is the time and in message we have a logs of the year 2018.
My requirement is, i have to show the previous year logs by selecting the time range of 2018 year. is that possible?
and the same should be selected for the machine learning anomaly detection. means we have to manipulate and show the evidence saying that we are maintaining logs of the previous year.
Message:
Jul 24, 2020 @ 15:57:13.888 24-Jan-2018 17:22:35.917 INFO [pool-1-thread-1] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [config.properties]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
If you're hoping to use general UI around Kibana like various date pickers to automatically select things a year before what the user selected, that's going to be tricky.
One thing you could try is to before ingesting your data, create a field that's one year off from your actual date and set this field as your @timestamp for a new index.
Otherwise, if you only need this in one specific visualization, you can do this inside of Canvas.
First add time filter controls. Then to your other visualizations, you can edit their scripts to offset the time. (You can find an example on how to do that in this other Discuss question).
How to achieve the below, using the below if i try to select the time range of older than 6 months, can i see the log/graph results.
One thing you could try is to before ingesting your data, create a field that's one year off from your actual date and set this field as your @timestamp for a new index.
Will it be possible to create anomaly job for the machine learning and can select the time range of older than 6 months to visualize the data(ingested data by today)
In this case, we are trying to ingest the 6 months of old data now (today). means we can see the data of the time stamp by selecting today.
i think we cannot visualize the anomaly detection of the timestamp older than 6 months.
From the below screenshot, I have the results for the month of July.
in similar way, my requirement is to get the results for the previous months(jan to june) , will that possible? if i inject the previous months data now, i have to see the results for the older months.
One can ingest data today that has old timestamps in the log lines and if that timestamp is parsed properly it will go into elasticsearch as if you ingested it way back then
One can then create an ML job on that "old" data and see anomalies in it. Even if the data is years old.
However, one thing you may need to be careful of is that if you have an ILM Policy that, for example, is set to delete data older than 6 months, then as soon as you ingest that old data, the ILM policy will delete it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
