How to insert Data to Elasticsearch using Logstash?

[amruth]

Hi, I am trying to insert data to Elasticsearch using Logstash. I created an index 'twitter' and my config file is as follows,

input{
file{
path=>"C:/Users/Sample.log"
codec=>json
}
}
output{
stdout{
codec=>rubydebug
}

elasticsearch {  
hosts => "13.10.256.10:9200"
 index => "twitter"
}

}

I could see the logs in command prompt but when I checked elasticsearch with the API 'http://13.10.256.10:9200/_cat/indices?v', it's showing docs.count as 0. What wrong am I doing? Do I need to create any other fields other than index? Please help me on this issue.

[dadoonet (David Pilato)]

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I moved your question to #logstash.
May be share the output of logstash and of your curl command?

[Christian_Dahlqvist]

The file input plugin by default tails the file, so if it was created before Logstash was started you will need to set start_position to beginning and also delete the sincedb file, as this will have recorded the current file as processed.

[amruth]

Hi David,

Thanks for your quick reply. Please see my logstash output and output of elasticsearch.

Logstash:

{
"CheckTS" => 1496264400085,
"path" => "C:/Users/Sample.log",
"@timestamp" => 2017-06-13T13:49:01.335Z,
"VersionTag" => 0,
"@version" => "1",
"host" => "localhost",
"location" => "abc",
"Component" => "mongodb",
"Application" => "Test"
}
{
"CheckTS" => 1496264400085,
"path" => "C:/Users/Sample.log",
"@timestamp" => 2017-06-13T13:49:01.335Z,
"VersionTag" => 0,
"@version" => "1",
"host" => "localhost",
"location" => "abc",
"Component" => "sql",
"Application" => "Test1"
}

Elasticsearch:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open twitter BcDQCs7uRWunz39AavLAVg 3 2 0 0 477b 477b

[amruth]

Hi Christian, I don't have problem with start_position, because I am able to see all my results in the command prompt output. The only problem is I can't see when I check it in the elasticsearch index.

[amruth]

I got it solved. Logstash is running from 2 different folders and that made the mess.