# Timestamp replaced

**URL:** <https://discuss.elastic.co/t/timestamp-replaced/120054>\
**Category:** Logstash\
**Created:** [February 15, 2018, 6:31pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054 "2018-02-15T18:31:33Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 6:31pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/1 "2018-02-15T18:31:33Z")

</div>

Hi,

There is one attribute in my json file called "ts" for each item in the file.  
{"name" : "David", "ts" : 2018-2-15 13:20:00 +0000", .....}

And if I directly import this json to ES, the "@timestamp" will be the exact time when I import the date, not the time that this specific event happened ("ts" in the json). So I am wondering if there is some way to replace the default "@timestamp" of ES by the specific date attribute that is already in the json file("ts")?

Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2018, 6:49pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/2 "2018-02-15T18:49:39Z")

</div>

Yes, use a [date](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) filter.

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 9:27pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/3 "2018-02-15T21:27:52Z")

</div>

Could you explain a lil bit more about how to do that?

Is it the date filter inside of json filter? And how does that work?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2018, 9:29pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/4 "2018-02-15T21:29:17Z")

</div>

You posted this in the logstash forum. Are you using logstash to push the json into elasticsearch?

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 9:33pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/5 "2018-02-15T21:33:53Z")

</div>

Yes, I used logstash

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 9:35pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/6 "2018-02-15T21:35:07Z")

</div>

and this is my current config

```
input {
	file {
		path => "/Users/apple/Desktop/Logstash_trial_3/event.log.20180214"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => json

	}
}

filter {
	json{
		source => "message"
		#target =>"doc"
		remove_field => ["message"]
	}
	mutate{
		rename => ["_id" , "ID"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "realrollingdata"
		document_type => "eventdata"
	}
	stdout {}
	
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2018, 9:47pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/7 "2018-02-15T21:47:24Z")

</div>

Thanks for posting that. I would suggest you get rid of the elasticsearch output until you get the parsing working, and modify the stdout output to use rubydebug

```auto
output { stdout { codec => rubydebug } }

```

You should either use a json codec on the input or a json filter to do the parsing. You cannot do both. If your input is valid json (i.e. the timestamp starts with a quote, missing in your initial post) then I would use a codec. In that case your events will look like this

```
"@timestamp" => 2018-02-15T21:39:39.855Z,
            "ts" => "2018-2-15 13:20:00 +0000",
          "name" => "David",
```

So remove the json filter, because it is failing every time (there is no message field when you use a json codec on the input). Then add a date filter. Something like

```auto
date { match => ["ts", "yyyy-M-d HH:mm:ss Z"] }

```

Then your events will look more like

```
"ts" => "2018-2-15 13:21:00 +0000",
    "@timestamp" => 2018-02-15T13:21:00.000Z,
          "name" => "Goliath",
```

That "yyyy-M-d HH:mm:ss Z" pattern may be wrong. For example, you may need dd rather than d. Read the doc and look at your data to find out.

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 10:09pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/8 "2018-02-15T22:09:04Z")

</div>

This is so helpful, I am gonna try it. Thank you so much.

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 10:19pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/9 "2018-02-15T22:19:43Z")

</div>

This is my updated config,

```
input {
	file {
		path => "/Users/apple/Desktop/Logstash_trial/Logstash_trial_3/event.log.20180214"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => json

	}
}

filter {

	mutate{
		rename => ["_id" , "ID"]
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss.SSS ZZ"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "realrollingdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } }
	
}

```

But when I run the logstash to import the data, it threw the error that I did not meet before.

 ![14](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf1043a31e60530ba07bbff68c7c2a0d6306ed1c.png)

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 15, 2018, 10:25pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/10 "2018-02-15T22:25:17Z")

</div>

And if I commented off "stdout{ codec =\> rubydebug } }", the logstash will be stuck in Pipelines running.

 ![17](https://us1.discourse-cdn.com/elastic/original/3X/2/1/214c0cd896d4a847432b24456e06a55ba9db7396.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2018, 10:43pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/11 "2018-02-15T22:43:54Z")

</div>

> [@Pororo](#):
>
> stdout { codec =\> rubydebug } }

That has an imbalance of }, which is what the original error was complaining about.

If you comment it out then logstash will sit there and tail all the files that match your path, waiting for new log entries to be written to them, so that it can process them and write them to your elasticsearch output. Uncomment the stdout and remove the extra } and you will see it doing this. You can even append an extra line to the file whilst it is sitting there with pipelines running and see it wake up and process it.

Also the example date you gave ("2018-2-15 13:21:00 +0000") does not have milliseconds in it. So I would expect a \_dateparsefailure if you keep the .SSS in the pattern.

Also, please do not post images of text, they are a pain to read. Just post the text.

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 17, 2018, 7:50am UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/12 "2018-02-17T07:50:50Z")

</div>

Thank you for explaining it patiently.

Here is my latest config

```
input {
	file {
		path => ""C:\Users\User\Desktop\rollingdata\event.log.20180214""
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => json

	}
}

filter {

	mutate{
		rename => ["_id" , "ID"]
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss ZZ"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "realrollingdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}

```

And it threw the error like this.

`[2018-02-17T02:29:46,489][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 3, column 13 (byte 31) after input {\r\n\tfile {\r\n\t\tpath => \"\"", :backtrace=>["/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:42:in`compile\_imperative'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:12:in`block in compile\_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:11:in`compile\_sources'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:169:in`initialize'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/pipeline\_action/create.rb:40:in `execute'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:315:in`block in converge\_state'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:312:in`block in converge\_state'", "org/jruby/RubyArray.java:1734:in `each'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:299:in`converge\_state'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:90:in`execute'", "/Users/HanBit/Desktop/logstash-6.2.1/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/Users/HanBit/Desktop/logstash-6.2.1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in`block in initialize'"]}`

Meanwhile, if I removed the content in stdout, it works fine.

Btw, do I need to terminate the importing data thru logstash by myself once it complete importing all the files. Cuz for now, my interface of logstash is like stilling running and waiting for the new files. I thought it will terminate by itself once it has done importing.

Thanks!!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 17, 2018, 1:54pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/13 "2018-02-17T13:54:49Z")

</div>

> [@Pororo](#):
>
> Expected one of #, {, } at line 3, column 13 (byte 31) after input {\r\n\tfile {\r\n\t\tpath =\> ""

That seems reasonable. You have two sets of double quotes around the path. You only need one.

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 17, 2018, 8:45pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/14 "2018-02-17T20:45:06Z")

</div>

I have fixed the quote issues.

And I just found some problems through the process of ingesting data files.

Normally, it shows the status like this.

`2018-02-14T14:51:44.021Z Hans-MacBook-Pro.local %{message}`

But I got some abnormal ingesting.

```
2018-02-14T14:51:44.021Z Hans-MacBook-Pro.local %{message}
2018-02-14T14:54:13.295Z Hans-MacBook-Pro.local %{message}
2018-02-17T07:59:49.059Z Hans-MacBook-Pro.local "result.statusMessage":"Success",
2018-02-13T22:24:40.685Z Hans-MacBook-Pro.local %{message}
2018-02-14T14:53:37.384Z Hans-MacBook-Pro.local %{message}
2018-02-17T07:59:49.019Z Hans-MacBook-Pro.local {"jurHash":"112256955",
2018-02-17T07:59:49.060Z Hans-MacBook-Pro.local "durationMillis":3,"requestId":"1518411024","extUserId":"mocktest3","result.status":"0000","wsdlVersion":"1_8","operation":"createUser","_id":"car4be-w2-tc.1518411025323.3904474"}
2018-02-13T22:24:50.373Z Hans-MacBook-Pro.local %{message}
2018-02-14T01:05:26.438Z Hans-MacBook-Pro.local %{message}
2018-02-14T01:05:27.258Z Hans-MacBook-Pro.local %{message}
2018-02-14T14:42:56.489Z Hans-MacBook-Pro.local %{message}
2018-02-14T14:45:48.741Z Hans-MacBook-Pro.local %{message}
2018-02-17T07:59:49.055Z Hans-MacBook-Pro.local "txnId":"vipusDEE529597C6FD8AF",
2018-02-12T04:50:33.424Z Hans-MacBook-Pro.local %{message}
2018-02-13T22:24:50.453Z Hans-MacBook-Pro.local %{message}

```

And before ingesting, the error info is

```
[2018-02-17T02:59:49,008][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input within/between Object entries
 at [Source: (String)"{"jurHash":"112256955","; line: 1, column: 47]>, :data=>"{\"jurHash\":\"112256955\","}
[2018-02-17T02:59:49,054][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"\"txnId\":\"vipusDEE529597C6FD8AF\","}
[2018-02-17T02:59:49,057][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"\"ts\":\"2018-02-12 04:50:25.323 +0000\","}
[2018-02-17T02:59:49,058][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"\"result.statusMessage\":\"Success\","}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 17, 2018, 9:07pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/15 "2018-02-17T21:07:20Z")

</div>

With a json codec there is no event field called message. It is going to be easier to tell what is happening if you remove the code on the input, and go back to using

```auto
json { source => "message" }

```

(Do not delete the message field.) The output you have is stdout with a plain codec. Does rubydebug work for you now?

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 17, 2018, 10:22pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/16 "2018-02-17T22:22:38Z")

</div>

I tried to remove the code on the input and using json{ source =\> "message" },  
but it caused the same problems as before.

Yes, rubydebug works for me now.

```
input {
	file {
		path => "C:\Users\User\Desktop\rollingdata\event.log.20180214"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		#codec => json

	}
}

filter {
	json{
		source => "message"
	}

	mutate{
		rename => ["_id" , "ID"]
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss ZZ"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "realrollingdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}
```

---

<div class="post-metadata">

**Author:** ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)\
**Post date:** [February 17, 2018, 10:29pm UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/17 "2018-02-17T22:29:08Z")

</div>

![19%20PM](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc883c44a7439320b13dbf832cc9025113ff170c.png)

It seems the whole message for one item is accidentally splited by logstash.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2018, 12:03am UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/18 "2018-02-18T00:03:40Z")

</div>

> [@Pororo](#):
>
> {"jurHash":"112256955",

The logs suggest that is a complete line, but it is not valid JSON, so both the codec and the filter will tail to parse it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 18, 2018, 12:04am UTC](https://discuss.elastic.co/t/timestamp-replaced/120054/19 "2018-03-18T00:04:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.