Elastic search logstash


(Ajit Kumar) #1

Hi all,

I want to use logstash for real time syncing of data in both mongodb and elasticsearch.

i am using mongoosastic for real time syncing and it is working fine but i am not able to trnasfer the existing mongodb documents into elasticsearch. I am able to send the existing mongodb documents into elasticsearch using logstash. so i want to use logstash for real time syncing instead of mongoosastic.

Please help me how i configure logstash in my nodejs and how to call the logstash function from nodejs itself instead of through command line.


(Magnus Bäck) #2

Logstash isn't a library that can be called from NodeJS.


(Ajit Kumar) #3

ok that means i have to run the command

.\bin\logstatsh -f "path of the config file" from the command line.


(Ajit Kumar) #4

hello sir,

when i am loading the file using logstash after modifying some field, all the remaining entry is also getting injected 2nd time in elasticsearch. How to not allow the duplicate entry into elasticsearch.

Here is my config file:

input {
file {
path => "C:\Users\Downloads\node-todo-api\output.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
csv {

  separator => ","
   
   columns => ["name", "age", "rollNo" ]
   
}

  mutate { convert => [ "age", "integer"] }
  mutate { convert => [ "rollNo", "integer"] }

}

output {

elasticsearch { 
    hosts => "localhost:9200" 
    index => "friends"
    document_type => "college"
}

 stdout {}

}

Please help me I am new to the ES and Logstash.


(Magnus Bäck) #5

ok that means i have to run the command

.\bin\logstatsh -f "path of the config file" from the command line.

Yes, or you can use a wildcard in your file input (e.g. *.csv) and just copy new files to that directory. Logstash will pick up new files within seconds.

How to not allow the duplicate entry into elasticsearch.

Remove sincedb_path => "/dev/null".


(Ajit Kumar) #6

Hello sir,

when i am removing sincedb_path => "/dev/null".

the documents is not getting indexed in Elasticsearch


(Magnus Bäck) #7

Under which exact circumstances? Have you read the file input documentation and what it says about sincedb_path?


(Ajit Kumar) #8

Hello sir,

i use sincedb_path so that i can index the data again.

Because i have a csv file i want to index it into elasticsearch and trying to perform update operation if the data is already in elasticsearch and add new record if that record is not present into elasticsearch.

Please Help !


(Magnus Bäck) #9

Because i have a csv file i want to index it into elasticsearch and trying to perform update operation if the data is already in elasticsearch and add new record if that record is not present into elasticsearch.

Then you can't rely on Elasticsearch's automatic assignment of an id to each document. Instead, pick one or more fields from the input data that'll constitute a key to the document and define the document id in the document_id option of the elasticsearch output. If you reprocess the same file or another file where a line has the same key as an existing document it'll get overwritten. You may have to change the elasticsearch output's action option.


(Ajit Kumar) #10

Hello sir,

with the configuration of output plugin

output {

elasticsearch { 
    hosts => "localhost:9200" 
    index => "candidate"
    document_type => "stud"
    document_id => "%{name}%{age}"
}

 stdout {}

}

the problem of updating and adding the new record is solved.

Thanks for the help


(Ajit Kumar) #11

hello sir,

my csv file conatins these data

"name","age","rollNo"
"ajeet kumar",23,614
"rishu",23,6213
"abhishek",23,6180
"shaurya",25,689

when i am running the config file as an input to the logstash, it is not doing the index of the last
record. in this case , ("shaurya" , 25, 689) is not getting indexed into elasticsearch.

Please Help!


(Magnus Bäck) #12

Does the file end with a newline character?


(Ajit Kumar) #13

yes sir, file is ending with newline character.


(Ajit Kumar) #14

hello sir,

This is the code i am using to create the csv file.

var json2csv = require('json2csv');
var fs = require('fs');
var fields = ['name', 'age', 'rollNo'];

var friends = [{
"name": "ajeet kumar",
"age": 23,
"rollNo": 614
},
{
"name": "rishu",
"age": 23,
"rollNo": 213
},
{
"name": "abhishek",
"age": 23,
"rollNo": 6180
},
{
"name": "shaurya",
"age": 22,
"rollNo": 689
},{
"name": "HOD",
"age": 24,
"rollNo": 45
}]

// console.log(friends)

var csv = json2csv({ data: friends, fields: fields });
fs.writeFile('output_test1.csv', csv, function(err) {
if (err) throw err;
console.log('file saved');
});

But while indexing the last json object is not getting indexed.

Please Help!


(Magnus Bäck) #15

I just tested with your script and the resulting file does not end with a newline character.

$ hexdump -C < output_test1.csv      
00000000  22 6e 61 6d 65 22 2c 22  61 67 65 22 2c 22 72 6f  |"name","age","ro|
00000010  6c 6c 4e 6f 22 0a 22 61  6a 65 65 74 20 6b 75 6d  |llNo"."ajeet kum|
00000020  61 72 22 2c 32 33 2c 36  31 34 0a 22 72 69 73 68  |ar",23,614."rish|
00000030  75 22 2c 32 33 2c 32 31  33 0a 22 61 62 68 69 73  |u",23,213."abhis|
00000040  68 65 6b 22 2c 32 33 2c  36 31 38 30 0a 22 73 68  |hek",23,6180."sh|
00000050  61 75 72 79 61 22 2c 32  32 2c 36 38 39 0a 22 48  |aurya",22,689."H|
00000060  4f 44 22 2c 32 34 2c 34  35                       |OD",24,45|
00000069

(Ajit Kumar) #16

ok sir, is it required to end with new line character ?

or can i add a dummy record at the last entry of json object, to solve it ?


(Ajit Kumar) #17

Hello sir,

Thanks for the help
I modified the array of json object to

var friends = [{
  "name": "ajeet kumar",
  "age": 23,
  "rollNo": 600
},
{
    "name": "rishu",
    "age": 23,
    "rollNo": 6213
},
{
    "name": "abhishek",
    "age": 23,
    "rollNo": 6180
},
{
    "name": "shaurya",
    "age": 22,
    "rollNo": 689
},{
    "name": "HOD",
    "age": 25,
    "rollNo": 450
},
{
    "name": "sush",
    "age": 25,
    "rollNo": 580
},
{
    "name": "chandu",
    "age": 24,
    "rollNo": 215
},
"/n"
]

then i am now able to get all the input field into elasticsearch.

Is it any other way to do this without adding ("/n") at the end.

Thanks


(Magnus Bäck) #18

ok sir, is it required to end with new line character ?

Yes.


(Ajit Kumar) #19

Thanks for Help sir !


(Ajit Kumar) #20

Hello sir,

  when i am adding a new data into the csv file , then the logstash is automatically reading that new input, but when i am modifing any field of csv file , in this case logstash is not automatically reading the file .

Please Help!