Not able to get expected output from elasticsearch through logstash pipeline

[somkhil]

HI
I am running logstash pipeline to pull the data from elasticsearch and output it into csv file.
At the very first run it generated output properly. However during subsequent runs (after loading more data/fields into elasticsearch) it is not giving proper output.

Two questions :

1. Is there any setting to clear out earlier offsets/history so that logstash can consider every request as fresh one and pull all the data from elasticsearch based on the specified query.

2. How do I specify document type in the input section of logstash config file. Index name can be specified with "index" attribute. Is there any attribute for specifying document type within index.

input {
elasticsearch {
hosts => ["localhost:9200"]
index => "xyz"

Thanks.

[magnusbaeck (Magnus Bäck)]

Is there any setting to clear out earlier offsets/history so that logstash can consider every request as fresh one and pull all the data from elasticsearch based on the specified query.

The elasticsearch input doesn't store any state so it pulls all data each time.

How do I specify document type in the input section of logstash config file. Index name can be specified with "index" attribute. Is there any attribute for specifying document type within index.

Use the query option?

[somkhil]

The elasticsearch input doesn't store any state so it pulls all data each time.

Actually I am seeing different behavior. When I ran logstash pipeline initially with 10 fields in the elasticsearch doc. I got all the docs with those 10 fields populated properly in the output csv.

After adding around 30 new fields in the same elasticsearch documents, I am getting most of the earlier fields as empty. Does logstash tries to get the delta somehow.

Use the query option?
Thanks. I got this working by adding filter on "_type" field.

[magnusbaeck (Magnus Bäck)]

Does logstash tries to get the delta somehow.

No, it doesn't.