# Custom log file not getting pushed to Elasticsearch via logstash

**URL:** https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469
**Category:** Logstash
**Created:** [May 11, 2017, 7:25pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469 "2017-05-11T19:25:06Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![amit.karir](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@amit.karir](https://discuss.elastic.co/u/amit.karir)
#### Post date: [May 11, 2017, 7:25pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/1 "2017-05-11T19:25:06Z")

</div>

Hi,

I am trying to push a custom log file generated by my program into elasticsearch(5.3.2) via logstash(5.3.2).

I dont get any error but index is also not getting created... Am I missing something? I am pretty new to ELK stack and trying out things of my own by reading the ELK documentation.

Here is the output:  
C:\Data\ELK\logstash-5.3.2\logstash-5.3.2\bin\>logstash -f AE-log.conf  
Could not find log4j2 configuration at path /Data/ELK/logstash-5.3.2/logstash-5.3.2/config/log4j2.properties. Using default config which logs to conso  
le  
13:06:15.598 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[http:  
[//elastic:xxxxxx@localhost:9200/](https://elastic:xxxxxx@localhost:9200/)]}}  
13:06:15.604 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:  
healthcheck\_url=\>[http://elastic:xxxxxx@localhost:9200/](http://elastic:xxxxxx@localhost:9200/), :path=\>"/"}  
13:06:15.827 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>#\<URI::HTTP:0x48d8acfb URL:htt  
p://elastic:xxxxxx@localhost:9200/\>}  
13:06:15.829 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=\>nil}  
13:06:16.209 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash  
-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_te  
mplates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"  
=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword"}}}}}], "properti  
es"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "p  
roperties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
13:06:16.221 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :h  
osts=\>[#\<URI::Generic:0x327ae81d URL://localhost:9200\>]}  
13:06:16.499 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "  
pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
13:06:17.440 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started  
13:06:17.568 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9600}

Here is the config file:

input {

file {  
path =\> ["C:\Data\ELK\input\AElogs\AssistEdge\_SE.log"]  
start\_position =\> "beginning"  
type =\> "log"

}

}

filter {

kv {  
value\_split =\> "~"  
field\_split =\> " "   
}

grok {

```
match => {"message" => "%{NUMBER:id} %{WORD:level} %{NUMBER:priority} %{WORD:srcmodule} %{WORD:method} %{WORD:message} %{WORD:description} %{WORD:userid} %{IP:client}" }

 }

```

}

output {  
elasticsearch {

hosts =\> ["localhost:9200"]  
user =\> "elastic"  
password =\> "XXXXXX"  
action =\> "index"  
index =\> "assistedge\_se"  
}  
stdout { }  
}

Here is the log file content for reference:

instid~1 level~Info priority~1 srcmodule~Utilities.Logging method~Logging message~Loaded logUserID from app.config. Value is : True description~NA userid~L3\john.kam ipaddress~10.1.99.121   
instid~1 level~Info priority~1 srcmodule~Utilities.Loggnig method~Logging message~Loaded logErrorDetails from app.config. Value is : True description~NA userid~L3\john.kam ipaddress~10.1.99.121

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 12, 2017, 5:47am UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/2 "2017-05-12T05:47:37Z")

</div>

Logstash is probably tailing the file. Set `sincedb_path` to "nul" or delete the sincedb file.

---

<div class="post-metadata">

### Author: ![amit.karir](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@amit.karir](https://discuss.elastic.co/u/amit.karir)
#### Post date: [May 12, 2017, 3:13pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/3 "2017-05-12T15:13:55Z")

</div>

Thanks Magnus.

I did that but didn't resulted into desired results.

Hence, I changed input to **stdin** and started sending individual log file record into logstash.

See below the stdin and the output I got...

You'll notice that now I am getting a grokparsefailure failure:

instid~1 level~Info priority~10 srcmodule~Utilities.CommonServices.AppListGenerator method~GenerateApplicationListConfig message~Completed generation of applicationlist.xml configuration file description~NA userid~LEVEL3\karir.amit ipaddress~10.1.47.180  
{  
"ipaddress" =\> "10.1.47.180",  
"method" =\> "GenerateApplicationListConfig",  
"level" =\> "Info",  
"srcmodule" =\> "Utilities.CommonServices.AppListGenerator",  
"description" =\> "NA",  
"message" =\> "Completed generation of applicationlist.xml configuration file",  
"priority" =\> "10",  
"userid" =\> "LEVEL3\karir.amit",  
**"tags" =\> [**  
**[0] "\_grokparsefailure"**  
**],**  
"instid" =\> "1",  
"@timestamp" =\> 2017-05-12T15:03:15.394Z,  
"@version" =\> "1",  
"host" =\> "US-HPELNVV5J"  
}

Here is the updated config file, not sure what am I doing wrong here..

input {

stdin{ }  
}

filter {

kv {  
value\_split =\> '~'  
field\_split =\> ' '   
}

grok {

match =\> {'message' =\> '%{NUMBER:instid} %{WORD:level} %{NUMBER:priority} %{WORD:srcmodule} %{WORD:method} %{WORD:message} %{WORD:description} %{WORD:userid} %{IP:ipaddress}' }

```
}

```

}

output {  
stdout {codec =\> rubydebug }

}

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 12, 2017, 4:55pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/4 "2017-05-12T16:55:05Z")

</div>

What are you trying to accomplish with the grok filter? The preceding kv filter already extracts the fields for you.

---

<div class="post-metadata">

### Author: ![amit.karir](https://avatars.discourse-cdn.com/v4/letter/a/b2d939/32.png) [@amit.karir](https://discuss.elastic.co/u/amit.karir)
#### Post date: [May 15, 2017, 8:21pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/5 "2017-05-15T20:21:23Z")

</div>

Thanks Magnus.. I am able to make it work now.. thanks for your valuable input.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 12, 2017, 8:25pm UTC](https://discuss.elastic.co/t/custom-log-file-not-getting-pushed-to-elasticsearch-via-logstash/85469/6 "2017-06-12T20:25:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.