# Elastic search logstash

**URL:** <https://discuss.elastic.co/t/elastic-search-logstash/120568>\
**Category:** Logstash\
**Created:** [February 20, 2018, 7:04am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568 "2018-02-20T07:04:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 7:04am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/1 "2018-02-20T07:04:59Z")

</div>

Hi all,

I want to use logstash for real time syncing of data in both mongodb and elasticsearch.

i am using mongoosastic for real time syncing and it is working fine but i am not able to trnasfer the existing mongodb documents into elasticsearch. I am able to send the existing mongodb documents into elasticsearch using logstash. so i want to use logstash for real time syncing instead of mongoosastic.

Please help me how i configure logstash in my nodejs and how to call the logstash function from nodejs itself instead of through command line.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 10:10am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/2 "2018-02-20T10:10:19Z")

</div>

Logstash isn't a library that can be called from NodeJS.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 10:50am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/3 "2018-02-20T10:50:51Z")

</div>

ok that means i have to run the command

.\bin\logstatsh -f "path of the config file" from the command line.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 10:52am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/4 "2018-02-20T10:52:49Z")

</div>

hello sir,

when i am loading the file using logstash after modifying some field, all the remaining entry is also getting injected 2nd time in elasticsearch. How to not allow the duplicate entry into elasticsearch.

Here is my config file:

input {  
file {  
path =\> "C:\Users\Downloads\node-todo-api\output.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
csv {

```
  separator => ","
   
   columns => ["name", "age", "rollNo"]
   
}

  mutate { convert => ["age", "integer"] }
  mutate { convert => ["rollNo", "integer"] }

```

}

output {

```
elasticsearch { 
    hosts => "localhost:9200" 
    index => "friends"
    document_type => "college"
}

 stdout {}

```

}

Please help me I am new to the ES and Logstash.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 11:50am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/5 "2018-02-20T11:50:32Z")

</div>

> ok that means i have to run the command
> 
> .\bin\logstatsh -f "path of the config file" from the command line.

Yes, or you can use a wildcard in your file input (e.g. `*.csv`) and just copy new files to that directory. Logstash will pick up new files within seconds.

> How to not allow the duplicate entry into elasticsearch.

Remove `sincedb_path => "/dev/null"`.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 1:05pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/6 "2018-02-20T13:05:53Z")

</div>

> [@magnusbaeck](#):
>
> sincedb\_path =\> "/dev/null".

Hello sir,

when i am removing sincedb\_path =\> "/dev/null".

the documents is not getting indexed in Elasticsearch

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 1:20pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/7 "2018-02-20T13:20:10Z")

</div>

Under which exact circumstances? Have you read the file input documentation and what it says about `sincedb_path`?

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 2:08pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/8 "2018-02-20T14:08:13Z")

</div>

Hello sir,

i use sincedb\_path so that i can index the data again.

Because i have a csv file i want to index it into elasticsearch and trying to perform update operation if the data is already in elasticsearch and add new record if that record is not present into elasticsearch.

Please Help !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 2:51pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/9 "2018-02-20T14:51:41Z")

</div>

> Because i have a csv file i want to index it into elasticsearch and trying to perform update operation if the data is already in elasticsearch and add new record if that record is not present into elasticsearch.

Then you can't rely on Elasticsearch's automatic assignment of an id to each document. Instead, pick one or more fields from the input data that'll constitute a key to the document and define the document id in the `document_id` option of the elasticsearch output. If you reprocess the same file or another file where a line has the same key as an existing document it'll get overwritten. You may have to change the elasticsearch output's `action` option.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 3:14pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/10 "2018-02-20T15:14:03Z")

</div>

Hello sir,

with the configuration of output plugin

output {

```
elasticsearch { 
    hosts => "localhost:9200" 
    index => "candidate"
    document_type => "stud"
    document_id => "%{name}%{age}"
}

 stdout {}

```

}

the problem of updating and adding the new record is solved.

Thanks for the help

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 3:16pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/11 "2018-02-20T15:16:31Z")

</div>

hello sir,

my csv file conatins these data

"name","age","rollNo"  
"ajeet kumar",23,614  
"rishu",23,6213  
"abhishek",23,6180  
"shaurya",25,689

when i am running the config file as an input to the logstash, it is not doing the index of the last  
record. in this case , ("shaurya" , 25, 689) is not getting indexed into elasticsearch.

Please Help!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 3:48pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/12 "2018-02-20T15:48:49Z")

</div>

Does the file end with a newline character?

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 4:01pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/13 "2018-02-20T16:01:19Z")

</div>

yes sir, file is ending with newline character.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 4:28pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/14 "2018-02-20T16:28:10Z")

</div>

hello sir,

This is the code i am using to create the csv file.

var json2csv = require('json2csv');  
var fs = require('fs');  
var fields = ['name', 'age', 'rollNo'];

var friends = [{  
"name": "ajeet kumar",  
"age": 23,  
"rollNo": 614  
},  
{  
"name": "rishu",  
"age": 23,  
"rollNo": 213  
},  
{  
"name": "abhishek",  
"age": 23,  
"rollNo": 6180  
},  
{  
"name": "shaurya",  
"age": 22,  
"rollNo": 689  
},{  
"name": "HOD",  
"age": 24,  
"rollNo": 45  
}]

// console.log(friends)

var csv = json2csv({ data: friends, fields: fields });  
fs.writeFile('output\_test1.csv', csv, function(err) {  
if (err) throw err;  
console.log('file saved');  
});

But while indexing the last json object is not getting indexed.

Please Help!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 6:34pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/15 "2018-02-20T18:34:17Z")

</div>

I just tested with your script and the resulting file does _not_ end with a newline character.

```nohighlight
$ hexdump -C < output_test1.csv      
00000000 22 6e 61 6d 65 22 2c 22 61 67 65 22 2c 22 72 6f |"name","age","ro|
00000010 6c 6c 4e 6f 22 0a 22 61 6a 65 65 74 20 6b 75 6d |llNo"."ajeet kum|
00000020 61 72 22 2c 32 33 2c 36 31 34 0a 22 72 69 73 68 |ar",23,614."rish|
00000030 75 22 2c 32 33 2c 32 31 33 0a 22 61 62 68 69 73 |u",23,213."abhis|
00000040 68 65 6b 22 2c 32 33 2c 36 31 38 30 0a 22 73 68 |hek",23,6180."sh|
00000050 61 75 72 79 61 22 2c 32 32 2c 36 38 39 0a 22 48 |aurya",22,689."H|
00000060 4f 44 22 2c 32 34 2c 34 35 |OD",24,45|
00000069

```

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 6:43pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/16 "2018-02-20T18:43:53Z")

</div>

ok sir, is it required to end with new line character ?

or can i add a dummy record at the last entry of json object, to solve it ?

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 20, 2018, 7:02pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/17 "2018-02-20T19:02:39Z")

</div>

Hello sir,

Thanks for the help  
I modified the array of json object to

```
var friends = [{
  "name": "ajeet kumar",
  "age": 23,
  "rollNo": 600
},
{
    "name": "rishu",
    "age": 23,
    "rollNo": 6213
},
{
    "name": "abhishek",
    "age": 23,
    "rollNo": 6180
},
{
    "name": "shaurya",
    "age": 22,
    "rollNo": 689
},{
    "name": "HOD",
    "age": 25,
    "rollNo": 450
},
{
    "name": "sush",
    "age": 25,
    "rollNo": 580
},
{
    "name": "chandu",
    "age": 24,
    "rollNo": 215
},
"/n"
]

```

then i am now able to get all the input field into elasticsearch.

Is it any other way to do this without adding ("/n") at the end.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 8:27pm UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/18 "2018-02-20T20:27:38Z")

</div>

> ok sir, is it required to end with new line character ?

Yes.

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 21, 2018, 8:53am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/19 "2018-02-21T08:53:24Z")

</div>

Thanks for Help sir !

---

<div class="post-metadata">

**Author:** ![Ajit\_Kumar1](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@Ajit\_Kumar1](https://discuss.elastic.co/u/Ajit_Kumar1)\
**Post date:** [February 21, 2018, 9:06am UTC](https://discuss.elastic.co/t/elastic-search-logstash/120568/20 "2018-02-21T09:06:21Z")

</div>

Hello sir,

```
  when i am adding a new data into the csv file , then the logstash is automatically reading that new input, but when i am modifing any field of csv file , in this case logstash is not automatically reading the file .

```

Please Help!

[Next page](https://discuss.elastic.co/t/elastic-search-logstash/120568.md?page=2)