Hello all,
I have apache logs coming into elasticsearch(aws). The logs are being created in epoch 'format'. I am using td-agent for the transportation. I tried different methods to transform the date to humand readable with subsecond precision and then send to elasticsearch but I am not being successful. I would like to know is there a way elasticsearch can do this job(in human readable with subsecond precision)?
Whenever I change the logs of apache from seconds to milli or micro, elasticsearch creates new index with first few digits of epoch time for each log.
Could someone guide me what can be done in this case?
Thank you
When you say aws, do you mean their hosted service? Also, what version of Elasticsearch?
Hi Warkolm,
Yes, aws managed elasticsearch service with version 7.9.
Elasticsearch will always store time as UTC in epoch.
So when you say you want it in human readable format, you need to take that epoch and then convert it once outside Elasticsearch. Kibana will do that for you automatically, otherwise you need to code the change.
Thank you for your time.
Sorry, I didn't explain myself better in the beginning. At the moment I get time normally like 'Aug 17, 2021 @ 01:19:03.000'.
I want to have it in subsecond. When I change logging in apache from seconds to milli or microseconds, elasticsearch creates new index for each log like 'apache-access-51627800.51'. My normal indices are created as 'apache-access-2021-33'(weekly indices).
I wanted to confirm if by changing mappings etc, i could achieve this(I have tried but in vain).
So now it's clear that I need to change it before ingestion to elasticsearch.
Oh right. Well that's probably something in your agent then, because Elasticsearch will create the index name based on the request from the agent.
Can you manually convert the timestamp into a readable format that contain milliseconds? Standard epoch is by default only measured in seconds so it is not clear that the original data contains the precision needed.
I am not familiar with td-agent, but if you were using Logstash you would need to convert the epoch timestamp using a date filter. This variable would in turn be used to generate the date string for the index name. If td-agent is not able to do this type of processing you could avoid incorrect index generation by using rollover.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.