# Filebeat cannot able to index into elasticsearch
**URL:** https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927
**Category:** Beats
**Tags:** filebeat
**Created:** [March 1, 2017, 7:47am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927 "2017-03-01T07:47:30Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 1, 2017, 7:47am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/1 "2017-03-01T07:47:31Z")
Filebeat cannot able to index into elasticsearch
please find the error below, ES ping not happening. Please suggest
2017-03-01T13:14:44+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:14:45+05:30 DBG Ping request failed with: Get [http://10.209.68.81:9201](http://10.209.68.81:9201): dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 ERR Connecting error publishing events (retrying): Get [http://10.209.68.81:9201](http://10.209.68.81:9201): dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 DBG send fail
2017-03-01T13:14:46+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 1, 2017, 7:51am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/2 "2017-03-01T07:51:38Z")
Also find below error too
2017-03-01T13:20:28+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:20:28+05:30 DBG Ping request failed with: 401 Unauthorized
2017-03-01T13:20:28+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2017-03-01T13:20:28+05:30 DBG send fail
2017-03-01T13:20:30+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-01T13:20:30+05:30 DBG Run prospector
2017-03-01T13:20:30+05:30 DBG Start next scan
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 1, 2017, 8:18am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/3 "2017-03-01T08:18:19Z")
Moving your question to #beats:filebeat
I'd say that probably elasticsearch is not running at 10.209.68.81:9201
BTW I'd use 10.209.68.81:9200 instead
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 1, 2017, 8:24am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/4 "2017-03-01T08:24:32Z")
will the ES port number matters?
Because i have added custom port number in ES.yml file where ES is working fine, But i couldnt able to index filebeat into ES.
Please help
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 1, 2017, 5:49pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/5 "2017-03-01T17:49:40Z")
> [@Sujith](#):
>
> 2017-03-01T13:20:28+05:30 DBG Ping request failed with: 401 Unauthorized
you getting `401 Unauthorized` from elasticsearch. did you correctly configure the username and password?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 2, 2017, 8:01am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/6 "2017-03-02T08:01:24Z")
Yes Steffens, I ahve configured proper username and password.
By in case if i have not done properly, let me know what needs to be done please ?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 2, 2017, 12:22pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/7 "2017-03-02T12:22:04Z")
Hi Steffens, Please can you let us know what needs to be done for 401 unauthorized error.?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 2, 2017, 1:05pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/8 "2017-03-02T13:05:43Z")
are you using xpack or some proxy asking for authentication? Can you share your config (redact the password please)? Which filebeat version are you using?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 2, 2017, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/9 "2017-03-02T13:13:54Z")
Im using x pack 5.1.1
filebeat version 5.1.1
May i know which config file i need to share please?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 2, 2017, 1:32pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/10 "2017-03-02T13:32:55Z")
Please find the below filebeat config file
filebeat.prospectors:
- input\_type: log
paths:
- /var/log/\*.log
- D:\Team\logs\*.log
document\_type: log
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["[bngwidap107.aonnet.aon.net:9200](http://bngwidap107.aonnet.aon.net:9200)"]
logging.level: debug
logging.selectors: ["\*"]
please let us know if anything needed.
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 2, 2017, 2:15pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/11 "2017-03-02T14:15:57Z")
Hi steffens, Please let us know what misght be causing the error.?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 3, 2017, 7:20am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/12 "2017-03-03T07:20:02Z")
please can someone help me in this issue, we are awaiting for your response.
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 3, 2017, 11:39am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/13 "2017-03-03T11:39:03Z")
Hi steffen, Let us know what is the issue causing for us.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 3, 2017, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/14 "2017-03-03T12:03:52Z")
Please be patient as this forum is manned by volunteers. As you have secured your cluster with X-Pack, you will need to [configure Beats to take this into account](https://www.elastic.co/guide/en/x-pack/current/beats.html) as well.
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 3, 2017, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/15 "2017-03-03T12:53:14Z")
Hi Christian,
Yes we have configured with beats security since then same issue we are facing.
For now i have made xpack.security.enabled: false in elasticsearch config and tried to index filebeat into elasticsearch, still filebeat not indexing.
Please find filebeat log below where no error appersa i seems to be.
2017-03-03T18:18:55+05:30 DBG Prospector states cleaned up. Before: 18, After: 18
2017-03-03T18:18:56+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:01+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:05+05:30 DBG Run prospector
2017-03-03T18:19:05+05:30 DBG Start next scan
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_12.49.43.log, offset: 903525
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut\_17.03.03\_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.05.33.log, offset: 1045061
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut\_17.03.03\_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.16.09.log, offset: 931302
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut\_17.03.03\_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.47.27.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut\_17.03.03\_13.47.27.log, offset: 1039071
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut\_17.03.03\_13.47.27.log
also please find elasticsearch log below for reference
[2017-03-03T17:54:20,639][ERROR][o.e.x.m.AgentService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:194) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.1.1.jar:5.1.1]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_101]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default\_local]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default\_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
[2017-03-03T17:54:20,658][INFO][o.e.c.m.MetaDataMappingService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update\_mapping [wineventlog]
[2017-03-03T17:54:21,921][INFO][o.e.c.m.MetaDataMappingService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update\_mapping [wineventlog]
[2017-03-03T17:54:21,926][INFO][o.e.c.m.MetaDataMappingService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update\_mapping [wineventlog]
[2017-03-03T17:54:24,334][INFO][o.e.c.m.MetaDataMappingService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update\_mapping [wineventlog]
[2017-03-03T17:54:44,172][ERROR][o.e.x.m.c.c.ClusterStateCollector] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] collector [cluster-state-collector] timed out when collecting data
[2017-03-03T17:54:45,479][INFO][o.e.c.r.a.AllocationService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2017-03-03T17:54:45,636][INFO][o.e.c.m.MetaDataMappingService] [[bngwidap107.aonnet.aon.net](http://bngwidap107.aonnet.aon.net)] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update\_mapping [wineventlog]
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 3, 2017, 5:30pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/16 "2017-03-03T17:30:59Z")
Please format logs and config files with `` button.
You config is kind of incomplete... by redacting username/password I didn't mean to drop them.
Plus, how did you create your user for writing to ES?
No idea if/how logs from ES are related to your problem.
when posting logs, read them first. The filebeat log says nothing about faild send-attempts. But files not being updated... did you change any logs? Have you tried to delete the registry file?
I adapted the configuration from Christians link a little to create me a `beat_user` for writing to `filebeat-*`, `metricbeat-*` and `packetbeat-*` index (if you did just copy the samples as is, you would have no credentials for filebeat):
```auto
POST _xpack/security/role/beat_writer
{
"cluster": ["manage_index_templates", "monitor"],
"indices": [
{
"names": ["filebeat-*", "metricbeat-*", "packetbeat-*"],
"privileges": ["read","write","create_index"]
}
]
}
POST /_xpack/security/user/beat_user
{
"password" : "changeme",
"roles" : ["beat_writer"],
"full_name" : "Internal Beat User"
}
```
And the beats output configuration:
```auto
output.elasticsearch:
hosts: ["localhost:9200"]
username: "beat_user"
password: "changeme"
```
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 6, 2017, 11:33am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/17 "2017-03-06T11:33:45Z")
Hi Steffens,
Thanks for the above input.
Please can you let me know how to format logs and config files with \ button.
also i have configured elastic user in filebeat output.elaticsearch. please let me is that the proper configuration which user writing to ES.?
As you mentioned samples above, i have created an beat\_writer role and beat\_user, still i couldnt able to index filebeat into ES.
Please do help.
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 9, 2017, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/18 "2017-03-09T13:59:04Z")
Hi steffens,
continously im getting the below error in filebeat logs.
2017-03-09T18:43:24+05:30 DBG send completed
2017-03-09T18:43:24+05:30 DBG output worker: publish 50 events
2017-03-09T18:43:24+05:30 DBG PublishEvents: 50 events have been published to elasticsearch in 1.9989ms.
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_expression","[resource.id](http://resource.id)":"filebeat-2017.03.09","index\_uuid":"_na_","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_expression","[resource.id](http://resource.id)":"filebeat-2017.03.09","index\_uuid":"_na_","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index\_not\_found\_exception","reason":"no such index","resource.type":"index\_expression","[resource.id](http://resource.id)":"filebeat-2017.03.09","index\_uuid":"_na_","index":"filebeat-2017.03.09"}
please let us know how do we index filebeat into elasticsearch.?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 10, 2017, 11:48pm UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/19 "2017-03-10T23:48:15Z")
Could you quickly try to ingest with the admin user to see if it is an user access issue or not?
---
### Author:  [@Sujith](https://discuss.elastic.co/u/Sujith)
#### Post date: [March 11, 2017, 11:03am UTC](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927/20 "2017-03-11T11:03:03Z")
Ruflin, May i know what admin user you are talking about?
[Next page](https://discuss.elastic.co/t/filebeat-cannot-able-to-index-into-elasticsearch/76927.md?page=2)