# ElasticSearch output of Filebeat is empty, displays http error 400 Bad Request

**URL:** <https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407>\
**Category:** Beats\
**Tags:** elastic-stack-monitoring, filebeat\
**Created:** [February 2, 2024, 1:46pm UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407 "2024-02-02T13:46:07Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [February 15, 2024, 12:42pm UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/21 "2024-02-15T12:42:38Z")

</div>

> [@NotTheRealV](#):
>
> `Skipping directory`

Ok thanks, can you also share the contents of the nginx.yml file you customized?

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 16, 2024, 5:33am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/22 "2024-02-16T05:33:19Z")

</div>

Sure.

```auto
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["C:/Program Files/nginx-1.25.3/logs"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["C:/Program Files/nginx-1.25.3/logs"]

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [February 16, 2024, 12:17pm UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/23 "2024-02-16T12:17:44Z")

</div>

> [@NotTheRealV](#):
>
> `var.paths: ["C:/Program Files/nginx-1.25.3/logs"]`

These settings need to point to files either directly or via wildcards, you currently have them pointing at directories.

```auto
- module: nginx
  access:
    enabled: true
    var.paths: ["/path/to/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/path/to/log/nginx/error.log*"]

```

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 16, 2024, 2:20pm UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/24 "2024-02-16T14:20:49Z")

</div>

I made the changes to my `nginx.yml` file. Now its `"C:/Program Files/nginx-1.25.3/logs/error.log*"` and unfortunately the output remains the same.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [February 16, 2024, 2:28pm UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/25 "2024-02-16T14:28:28Z")

</div>

Now the we've fixed the paths can you run `filebeat -e -d "*"` and share the output again?

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 19, 2024, 5:17am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/26 "2024-02-19T05:17:23Z")

</div>

Sure. Here are the changes in the message fields from the last run.

```auto
message : Start next scan   
message : Check file for harvesting: C:\\Program Files\\nginx-1.25.3\\logs\\error.log   
message : Check file for harvesting: C:\\Program Files\\nginx-1.25.3\\logs\\access.log   
message : Update existing file for harvesting: C:\\Program Files\\nginx-1.25.3\\logs\\access.log offset: 802   
message : Update existing file for harvesting: C:\\Program Files\\nginx-1.25.3\\logs\\error.log offset: 772   
message : File didn't change: C:\\Program Files\\nginx-1.25.3\\logs\\access.log   
message : File didn't change: C:\\Program Files\\nginx-1.25.3\\logs\\error.log   
message : input states cleaned up. Before: 1 After: 1 Pending: 0   
message : input states cleaned up. Before: 1 After: 1 Pending: 0   
message : Registry file updated. 2 active states.   
message : Start next scan   
message : File scan complete   
message : Run input 

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [February 19, 2024, 11:32am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/27 "2024-02-19T11:32:46Z")

</div>

It looks like it believes it has read the files and forwarded logs, did you check the index in elasticsearch to see if logs are present?

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 22, 2024, 4:18am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/28 "2024-02-22T04:18:14Z")

</div>

Using the console in DevTools, I queried the `filebeat-*` index and all the logs are present. Here's a sample of the output when I ran this query:

```auto
GET filebeat-*/_search
{
  "query": {
    "match_all": {}
  }
}

```

Output:

```auto
"event": {
            "ingested": "2024-02-16T14:18:59.560536200Z",
            "original": "127.0.0.1 - - [12/Feb/2024:10:48:12 +0530] \"GET / HTTP/1.1\" 200 615 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36\"",
            "timezone": "+05:30",
            "created": "2024-02-16T14:18:46.548Z",
            "kind": "event",
            "module": "nginx",
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "dataset": "nginx.access",
            "outcome": "success"
          }

```

So it looks like Filebeat has loaded the logs into Elasticsearch, but Kibana cannot display them, as I'm still getting the "400 Bad Request" on the Kibana console window.

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 26, 2024, 6:24am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/29 "2024-02-26T06:24:50Z")

</div>

So I did some digging around in my Elasticsearch and I found the following things:

1. When I ran `GET _cat/indices` in the console I can see the Filebeat datastream as `.ds-filebeat-8.12.0-2024.02.06-000001`. However, I cannot delete the datastream and when I run `GET /_data_stream/.ds-filebeat-8.12.0-2024.02.06-000001`, the output is empty: `{ "data_streams": []}`

2. When I go under Search -\> Content -\> Indices, I can find the Filebeat datastream under the hidden indices, and it contains all the `nginx` logs/documents.

I'm not sure if this helps make progress with the issue at hand, but I guess this confirms that Filebeat is sending the log files to Elasticsearch, and Elasticsearch is storing them. It's just Kibana's dashboards that cannot display them.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 26, 2024, 6:52am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/30 "2024-02-26T06:52:05Z")

</div>

Hi @NotTheRealV Progress, but I am a little confused...

When you say console, do you Mean Kibana -\> Dev Tools?

> [@NotTheRealV](#):
>
> `GET /_data_stream/.ds-filebeat-8.12.0-2024.02.06-000001`

That is not a proper command not sure what you are expecting to get out of that.

I think you are trying to run which show the backing indices...  
`GET /_data_stream/filebeat-8.12.0`

When you go to Discover and Use the filebeat-\* Data view do you see your data?  
When you open a document does it look fully parsed?

Also can you run

`GET filebeat-*/_search`

and show a full document of an access log, not just a snippet?

And for sure you ran `setup` right? before your started sending the logs?

> [@NotTheRealV](#):
>
> It's just Kibana's dashboards that cannot display them.

Exactly Which Dashboards? Can you show? What do they look like, do you have the correct time range?

It is possible that you have non-standard ngnix logs and that is why they are not displaying ...

You could share a couple of the raw log lines and we might be able to check.

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 27, 2024, 7:40am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/31 "2024-02-27T07:40:57Z")

</div>

> [@stephenb](#):
>
> When you say console, do you Mean Kibana -\> Dev Tools?

Yes, I meant Kibana -\> Dev Tools -\> Console Tab

> [@stephenb](#):
>
> > [@NotTheRealV](#):
> >
> > `GET /_data_stream/.ds-filebeat-8.12.0-2024.02.06-000001`
> 
> That is not a proper command not sure what you are expecting to get out of that.
> 
> I think you are trying to run which show the backing indices...  
> `GET /_data_stream/filebeat-8.12.0`

Well I ran - `GET /_data_stream/.ds-filebeat-8.12.0-2024.02.06-000001` because I was facing some errors when I tried `GET filebeat-*/_search` the first time, so I tried  
viewing all the indices. I saw that this .`ds-filebeat-8.12.0-2024.02.06-000001` was a datastream, so I searched the Elasticsearch documentation and found this command: `GET /_data_stream`.  
I ran that to get this result:

```auto
{
"data_streams": []
}

```

I assumed the result meant that the index was empty. However now when I run `GET filebeat-*/_search` it works fine and all the documents are visible. The errors I saw were probably due to some errors I did when I was running Filebeats the other day.

> [@stephenb](#):
>
> When you go to Discover and Use the filebeat-\* Data view do you see your data?  
> When you open a document does it look fully parsed?

As I mentioned before, on my Discover page, the filebeat-\* data view just shows empty fields as seen in this image:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/b/5b9d121da807359964575379f7e897cb2834b20e.png)

> [@stephenb](#):
>
> Exactly Which Dashboards? Can you show? What do they look like, do you have the correct time range?

I was viewing the Dashboards -\> Editing [Filebeat Nginx] Access and error logs ECS.

I've refreshed the time range multiple times and the Discover/Dashboard view does not change.

> [@stephenb](#):
>
> Also can you run
> 
> `GET filebeat-*/_search`
> 
> and show a full document of an access log, not just a snippet?

> [@stephenb](#):
>
> You could share a couple of the raw log lines and we might be able to check.

Sure, here are some of the raw log lines from the nginx/access.log and a full document of an access log in `filebeat-*`:  
Log Lines from nginx/access.log:

```auto
127.0.0.1 - - [12/Feb/2024:10:48:12 +0530] "GET / HTTP/1.1" 200 615 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
127.0.0.1 - - [12/Feb/2024:10:48:12 +0530] "GET /favicon.ico HTTP/1.1" 404 555 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"

```

Full document of access log from `filebeat-*`:

```auto
{
        "_index": ".ds-filebeat-8.12.0-2024.02.06-000001",
        "_id": "0MJIso0BmY7W8rCq6hbx",
        "_score": 1,
        "_source": {
          "agent": {
            "name": "$LAPTOP NAME",
            "id": "$ID",
            "ephemeral_id": "$EPHEMERAL_ID",
            "type": "filebeat",
            "version": "8.12.0"
          },
          "nginx": {
            "access": {
              "remote_ip_list": [
                "127.0.0.1"
              ]
            }
          },
          "log": {
            "file": {
              "path": """C:\Program Files\nginx-1.25.3\logs\access.log"""
            },
            "offset": 0
          },
          "source": {
            "address": "127.0.0.1",
            "ip": "127.0.0.1"
          },
          "fileset": {
            "name": "access"
          },
          "url": {
            "path": "/",
            "original": "/"
          },
          "input": {
            "type": "log"
          },
          "@timestamp": "2024-02-12T05:18:12.000Z",
          "ecs": {
            "version": "1.12.0"
          },
          "_tmp": {},
          "related": {
            "ip": [
              "127.0.0.1"
            ]
          },
          "service": {
            "type": "nginx"
          },
          "host": {
            "hostname": "$LAPTOP_NAME",
            "os": {
              $OS_DETAILS
            },
            "ip": [
             $IP_DETAILS
            ],
            "name": "$LAPTOP_NAME",
            "id": "$ID",
            "mac": [
             $MAC_ADDR_DETAILS
            ],
            "architecture": "x86_64"
          },
          "http": {
            "request": {
              "method": "GET"
            },
            "response": {
              "status_code": 200,
              "body": {
                "bytes": 615
              }
            },
            "version": "1.1"
          },
          "event": {
            "ingested": "2024-02-16T14:18:59.560536200Z",
            "original": "127.0.0.1 - - [12/Feb/2024:10:48:12 +0530] \"GET / HTTP/1.1\" 200 615 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36\"",
            "timezone": "+05:30",
            "created": "2024-02-16T14:18:46.548Z",
            "kind": "event",
            "module": "nginx",
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "dataset": "nginx.access",
            "outcome": "success"
          },
          "user_agent": {
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "name": "Chrome",
            "device": {
              "name": "Other"
            },
            "version": "121.0.0.0"
          }
        }
      }

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 27, 2024, 8:54am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/32 "2024-02-27T08:54:41Z")

</div>

> [@NotTheRealV](#):
>
> `"@timestamp": "2024-02-12T05:18:12.000Z`

Perhaps On discover set time range last 90 days .... Your data is several weeks old and so you have to make the time range where your data is.

If that works try the same on the dashboards

---

<div class="post-metadata">

**Author:** ![NotTheRealV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nottherealv/32/127589_2.png) [@NotTheRealV](https://discuss.elastic.co/u/NotTheRealV)\
**Post date:** [February 28, 2024, 4:53am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/33 "2024-02-28T04:53:08Z")

</div>

Yes, that worked. All the documents are present. I tried it with the RabbitMQ module as well, and changing the time range worked. In the past when I changed the time range I didn't understand that the documents are stored by their timestamp, not when the filebeat index went active. Sorry for the confusion, and thank you very much for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2024, 6:54am UTC](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407/34 "2024-03-27T06:54:00Z")

</div>

Closed on March 27, 2024, 6:54am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/elasticsearch-output-of-filebeat-is-empty-displays-http-error-400-bad-request/352407.md?page=1)
