# Import old postgres data to elasticsearch

**URL:** <https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090>\
**Category:** Logstash\
**Created:** [July 30, 2017, 7:50pm UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090 "2017-07-30T19:50:47Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [July 30, 2017, 7:50pm UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/1 "2017-07-30T19:50:47Z")

</div>

I have a lot of data in my postgres database( on a remote). This is the data of the past 1 year, and I want to push it to elasticsearch now.

The data has a time field in it in this format `2016-09-07 19:26:36.817039+00`.

I want this to be the timefield(`@timestamp`) in elasticsearch. So that I can view it in kibana, and see some visualizations over the last year.

I need help on how do I push all this data efficiently. I cannot get that how do I get all this data from postgres.

I know we can inject data via jdbc plugin, but I think I cannot create my `@timestamp` field with that.

I also know about zombodb but not sure if that also gives me feature to give my own timefield.

Also, the data is in bulk, so I am looking for an efficient solution

I need help on how I can do this. So, suggestions are welcome.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 30, 2017, 8:15pm UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/2 "2017-07-30T20:15:12Z")

</div>

WDYM by jdbc "plugin"?

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [July 31, 2017, 5:08am UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/3 "2017-07-31T05:08:49Z")

</div>

> **[INSERT INTO LOGSTASH SELECT DATA FROM DATABASE
	  	 | Elastic](https://www.elastic.co/blog/logstash-jdbc-input-plugin)**
>
> Ever want to search your database entities from Elasticsearch? Now you can use Logstash to do just that! In this blog we introduce the JDBC input, which has been created to import data ...

Read this.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 31, 2017, 6:18am UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/4 "2017-07-31T06:18:53Z")

</div>

So it's a logstash question. I moved your thread to #logstash.

Not sure why:

- you think you need a `@timestamp` field
- you think you can't get it from the db

Logstash date filter plugin should probably help here.

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [July 31, 2017, 7:21am UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/5 "2017-07-31T07:21:23Z")

</div>

Actually I got a simple(not the best) solution.

I came to know that elasticsearch reads python timestamp.

I used python client for elasticsearch, and psycopg2 library to fetch data from postgres.

There were around 40 coulmns and 55000 rows of data. It took almost 3 minutes for fetching and writing the data. The solution was simple and straight forward.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 28, 2017, 7:22am UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090/6 "2017-08-28T07:22:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
