Need to replace default @timestamp with userdefined timestamp

Elastic Stack Kibana
1

Hi,

I want to remove the default @timestamp entry from log messages and need to change the default log search timestamp a different one.

Kibana Version 5.5.2

2

This is typically something you need to do when you ingest the data. What does your ingest pipeline look like?

3

Hi,

Setup : fluentd, elasticsearch and kibana are using to collect and display the logs from client nodes.
Logs includes each node name,timestamp at which the logs are generated in nodes and error message.
The logs displaying with kibana should be listed based on the timestamp sending from client node.

I have written fluentd code for sending and receiving logs. But i haven't done anything in kibana end.. its showing default items.

Thanks..

4

I suspect you may need to alter your fluentd processing to ensure the events are written into Elasticsearch with the right data and format. It may be easier if you show what the events look like and what you expect, although I have never used fluentd, so may not be able to help.

5

The conf in fluentd are wring the required data only(if writing to a file) but when displaying in kibana there is one more additional value @timestamp and this is the default value in kibana i think. i need to remove this entry.

6

I am not sure I understand. Can you please provide an example of an event and what you want it to look like?

7

Example: Current o/p displaying in kibana as below:
2017-08-29T14:40:40+03:00 { "log_id": 24, "server_name": "node1", "time_stamp": "2017-08-29 11:40:01 +0300", "issue_category": "File systems issues / not mounted properly", "issue": "filesystem is full", "arguments": "/tmp usage crossed threshold by utilizing 15 ", "@timestamp": "2017-08-29T14:40:40+03:00" }

Here @timestamp is default time stamp in kibana
time_stamp is the time stamp from node when generating log.

I need only time_stamp (generated from nodes), need to remove @timestamp . Also the searching in kibana is now using default @timestamp(first entry in example) this also need to replace with time_stamp .

8

@fazi347:
If you don't alter @timestamp, then @timestamp is the date of insertion into elastic.

You can always put time_stamp into @timestamp and remove your time_stamp.

Other option is to leave it as is and just set kibana to use your time_stamp for display.

I'm using date field for this. Just go into kibana settings and set the timelion:es.timefield to your time_stamp

9

Hi,
thanks ..

Could you please explain how to put time_stamp into @timestamp.

10 
date {
		locale => "en"
		match => ["time_stamp", "dd.MM.YYYY HH:mm:ss"]
		timezone => "Europe/Ljubljana"
		target => "@timestamp"
		remove_field = > ['time_stamp']
	}

If you want both timestamps leave out the remove part.

11

Thank you.. will check this.

12

Please let me know were should i add this date entry..?
in fluentd/logstash conf file or kibana conf file or elasticsearch ?

Thanks..

13

In your pipeline, that your using in LS.

Kibana is just for display & monitor.

14

thank you..

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.