# Not getting data into Elasticsearch

**URL:** <https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848>\
**Category:** Elasticsearch\
**Created:** [June 27, 2019, 1:37pm UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848 "2019-06-27T13:37:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![markov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@markov](https://discuss.elastic.co/u/markov)\
**Post date:** [June 27, 2019, 1:37pm UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/1 "2019-06-27T13:37:21Z")

</div>

Hi all,

i'm sending logs using filebeat but i don't receive anything in kibana  
x-pack is enabled

I created a logstash writer role

```
> POST _xpack/security/role/logstash_writer
> {
> "cluster": ["manage_index_templates", "monitor"],
> "indices": [
> {
> "names": ["filebeat-*"],
> "privileges": ["write","delete","create_index"]
> }
> ]
> }

```

Then a logstash internal user:

```
> POST _xpack/security/user/logstash_internal
> {
> "password" : "changeme",
> "roles" : ["logstash_writer"],
> "full_name" : "Internal Logstash User"
> }

```

I then run the following command

```
> systemctl status -l logstash`

> Jun 27 15:29:10 sd-131865 logstash[23755]: [2019-06-27T15:29:10,956][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"security_exception", "reason"=>"action [indices:admin/create] is unauthorized for user [logstash_internal]"})
> Jun 27 15:29:10 sd-131865 logstash[23755]: [2019-06-27T15:29:10,958][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}

```

what am i doing wrong!

---

<div class="post-metadata">

**Author:** ![markov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@markov](https://discuss.elastic.co/u/markov)\
**Post date:** [June 27, 2019, 2:02pm UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/2 "2019-06-27T14:02:44Z")

</div>

This is my `.conf` files  
02-beats-input.conf

```auto
input {
  beats {
    port => 5044
 }
}

```

10-syslog-filter.conf

```auto
filter {
     if [source] =~ "gid" {
        grok {
            match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}\|(\[%{BASE10NUM:nbr}\])\|%{IPORHOST:ClientIP}\|%{USERNAME:User};%{DATA:Node}\|%{URIPATH:Url}\|%{NOTS$
        }
     }else if [source] =~ "server" {

        grok {
            match =>{ "message" => [

                "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<Classname>[^\]]+)\] %{GREEDYDATA:Message}",
                "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<Classname>[^\]]+)\] %{WORD:n} \| %{NUMBER:number} \| %{WORD:b} \| %{DATA:Url} \| %{GREEDYDAT$
                "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<Classname>[^\]]+)\] %{GREEDYDATA:Message}"] }
        }
      }
  date {match => ["timestamp" , "MMM dd yyyy HH:mm:ss","MMM d yyyy HH:mm:ss", "ISO8601"]
      target => "@timestamp"}

  mutate {
        remove_field => ["[beat][name]", "[beat][version]", "[beat][hostname]", "[host][architecture]", "[host][containerized]", "[host][id]", "[host][name]", "[host]$
  }
}

```

30-elasticsearch-output.conf

```auto
output {
  elasticsearch {
    hosts => ["myIpAdress:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    user => "logstash_internal"
    password => "admin10@"
    document_type => "%{[@metadata][type]}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 27, 2019, 4:24pm UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/3 "2019-06-27T16:24:01Z")

</div>

You should try to add a stdout output to see if anything is produced.

Some thoughts: why using logstash in that case? Elasticsearch ingest feature should be ok to do the filter part you want.  
So connect filebeat to elasticsearch directly and define the pipeline to use.

---

<div class="post-metadata">

**Author:** ![markov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@markov](https://discuss.elastic.co/u/markov)\
**Post date:** [June 28, 2019, 10:28am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/4 "2019-06-28T10:28:16Z")

</div>

> [@dadoonet](#):
>
> stdout output

stdout added, but receiving the operating system logs in elastic

---

<div class="post-metadata">

**Author:** ![markov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@markov](https://discuss.elastic.co/u/markov)\
**Post date:** [June 28, 2019, 10:58am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/6 "2019-06-28T10:58:13Z")

</div>

bad, i have my own log files to send with the path defined in filebeat

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2019, 10:58am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/7 "2019-06-28T10:58:37Z")

</div>

What is a typical document you can see on stdout?

---

<div class="post-metadata">

**Author:** ![markov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@markov](https://discuss.elastic.co/u/markov)\
**Post date:** [June 28, 2019, 11:02am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/8 "2019-06-28T11:02:09Z")

</div>

sorry this is the first time using the stdout; how can i see those documents

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2019, 11:15am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/9 "2019-06-28T11:15:45Z")

</div>

In the console.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 26, 2019, 11:15am UTC](https://discuss.elastic.co/t/not-getting-data-into-elasticsearch/187848/10 "2019-07-26T11:15:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.