Logstash change default @timestamp value by field from mongo

Elastic Stack Logstash
1

Hello, I would like get @timestamp to be my mongo field "created_at", i do

input {
		jdbc {
			 jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mongojdbc1.8.jar"
			 jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
			 jdbc_connection_string => "jdbc:mongodb://0.0.0:27018/toto"
			 jdbc_validate_connection => true
			 jdbc_user => ""
			 jdbc_password => ""
			 schedule => "/15 * * * * *"
			 tracking_column => "created_at"
       tracking_column_type => "timestamp"
			 statement => "db.getCollection('people').aggregate([
			 		{ $project: {
'_id': {$toString: '$_id'},
'name' : 1,
'gender' : 1,
'birthDate' : 1,
'type' : 1,
'origin' : 1,
'created_at' : 1,
'updated_at' : 1
						}
					}
					]);"
			 }
}

filter{
			mutate {
			    convert => { "created_at" => "string" }
			  }
			date {
				match => ["created_at", "yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'"]
				target => "@timestamp"
			}
}



output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "elastic"
		password => "changeme"
		index => "pdn_test_people_tt"
		document_id => "%{_id}"
	}
	stdout { codec => rubydebug }
}

No result @timestamp is alway date Now.
Thanks for your help

2

What does a document look like on the JSON tab of an expanded event in Kibana? Specifically the created_at field.

3

the stdOut from logstash

{
     "@timestamp" => 2020-03-31T18:53:15.696Z,
       "document" => {
               "name" => "test-kibana",
         "created_at" => 2020-03-30T13:59:09.608Z,
          "birthDate" => "1983",
               "type" => "Adulte",
             "gender" => "homme",
             "origin" => "Lors d'une intervention extérieure",
                "_id" => "5e81fb2ded73c60028bf2f54"
     },       "@version" => "1"
}

and the json in Kibana discover

{
  "_index": "pdn_test_people_tt",
  "_type": "_doc",
  "_id": "%{_id}",
  "_version": 57996,
  "_score": null,
  "_source": {
    "document": {
      "origin": "Lors d'une intervention extérieure",
      "name": "test-kibana",
      "birthDate": "1983",
      "_id": "5e81fb2ded73c60028bf2f54",
      "gender": "homme",
      "type": "Adulte",
      "created_at": "2020-03-30T13:59:09.608Z"
    },
    "@version": "1",
    "@timestamp": "2020-03-31T19:01:00.686Z"
  },
  "fields": {
    "@timestamp": [
      "2020-03-31T19:01:00.686Z"
    ],
    "document.created_at": [
      "2020-03-30T13:59:09.608Z"
    ]
  },
  "sort": [
    1585681260686
  ]
}

thanks

4

and i have only 3 hits on 1 years , i'm waiting for 108 hit ???

5

You do not have a field called [created_at]

You have a field called [document][created_at]

6

thanks your 're right .

who add this, logstash or my mongo request ?
is this the reason why i only have one doc in ES ?

thanks

7

Having never used Mongo I am unable to answer your additional questions.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.