# Parsing problem when streaming a log file

**URL:** <https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268>\
**Category:** Kibana\
**Created:** [November 29, 2023, 5:49pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268 "2023-11-29T17:49:37Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [November 29, 2023, 5:49pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/1 "2023-11-29T17:49:37Z")

</div>

Hey everyone,

I followed the _Stream any log file_ guide, and have set up a local agent that listens to my log file. But every time I add a new log (manually to test) the parsing just isn't there when it gets indexed in Kibana. The whole line is just falls under the _message_ field like so:

 ![help_1.PNG](https://us1.discourse-cdn.com/elastic/original/3X/5/0/5035da499b2d9594ba3c2d249cd36a23c318899b.jpeg)

But when I use the API and make a POST call to the same data stream, it parses correctly:

```auto
</>
POST logs-generic-default/_doc
{"log_time":"2023-11-28T09:50:33.026Z","project":"Public_Documentation_Mappings","last_activity":"2023-11-25T09:50:33.026Z"}
</>

```

I already created the _project_, _last\_activity_ and _log\_time_ fields.

It is the same for the filebeat (where is parses correctly for each field) but when I try to copy a log line and paste it to my own file that Kibana is also listening to, the entire line just falls under the _message_ field.

Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2023, 3:30am UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/2 "2023-11-30T03:30:23Z")

</div>

Hi @Kyps Welcome to the community.

1st we discourage screen shots of text, they are hard to read, can not be copied, searched debugged etc..

Also you should look at Kibana -\> Discover to look at your logs

What version are you on?

> [@Kyps](#):
>
> I followed the _Stream any log file_ guide,

Did you look at the next step...

> **[Parse and organize logs | Elastic Observability \[8.11\] | Elastic](https://www.elastic.co/guide/en/observability/current/logs-parse.html)**

When you read a log file the entire content of the log line ends up in the `message` field...

If you want to parse it you will need and ingest pipeline...

If your `message` field is JSON you can use the JSON processor in an ingest pipeline

> **[JSON processor | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html)**

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 2:42pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/3 "2023-12-01T14:42:54Z")

</div>

Hey Stephen,

Thank you for the answer.

Regarding the pipeline, I have created I pipeline (i think), I did it with the Console in the Dev Tools like this:

```auto
PUT _ingest/pipeline/logs-generic-default
{
  "description": "Extracts the log time, project name and project last activity",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": """{"log_time":"%{log_time}","project":"%{project}","last_activity":"%{last_activity}"}"""
      }
    }
  ]
}
POST _ingest/pipeline/logs-generic-default/_simulate
{
  "docs": [
    {
      "_source": {
        "message": """{"log_time":"2023-11-28T09:50:33.026Z","project":"Public_Documentation_Mappings","last_activity":"2023-11-25T09:50:33.026Z"}"""
      }
    }
  ]
}

PUT _index_template/logs-generic-default-template
{
  "index_patterns": ["logs-generic-*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.default_pipeline":"logs-generic-default"
    }
  },
  "composed_of": [
    "logs-mappings",
    "logs-settings",
    "logs@custom",
    "ecs@dynamic_templates"
  ],
  "ignore_missing_component_templates": ["logs@custom"]
}

```

The \_simulate POST call works as expected,, but my logs don't seem to go through the pipeline before getting indexed, maybe it has something to do with my elastic-agent.yml? which looks like this:

```auto
outputs:
  default:
    type: elasticsearch
    hosts: <my-host>
    #api_key: 'your-api-key'
    username: <my-user>
    password: <my-pass>
    pipeline: logs-generic-default # is this right to apply the pipeline?
inputs:
  - id: logs-generic-default
    type: filestream
    streams:
      - id: logs-generic-default
        data_stream.dataset: logs-generic-default
        paths:
          - C:\Program Files\Elastic\Agent\data\elastic-agent-03ef9d\logs\myapp.log

```

But I'll try to use the JSON processor, but Im not sure how to apply the ingest pipeline, like where would I add this code (sorry for the image, this is from the JSON processor guide you linked):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a3189c853a50c3639b208dd356acc37bff1c3a20.png)

Thanks again, Stephen!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 4:15pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/4 "2023-12-01T16:15:17Z")

</div>

Perhaps this will help..

There are a lot of parameters on the `json` so look carefully

there are pros and cons to putting the fields at root so you might want to put them under a different Field

```auto

PUT _ingest/pipeline/logs-generic-default
{
  "description": "Extracts the log time, project name and project last activity",
  "processors": [
    {
      "json": {
        "field": "message",
        "add_to_root": true
      }
    }
  ]
}

POST _ingest/pipeline/logs-generic-default/_simulate
{
  "docs": [
    {
      "_source": {
        "message": """{"log_time":"2023-11-28T09:50:33.026Z","project":"Public_Documentation_Mappings","last_activity":"2023-11-25T09:50:33.026Z"}"""
      }
    }
  ]
}

# result

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "project": "Public_Documentation_Mappings",
          "last_activity": "2023-11-25T09:50:33.026Z",
          "message": """{"log_time":"2023-11-28T09:50:33.026Z","project":"Public_Documentation_Mappings","last_activity":"2023-11-25T09:50:33.026Z"}""",
          "log_time": "2023-11-28T09:50:33.026Z"
        },
        "_ingest": {
          "timestamp": "2023-12-01T16:14:14.270428604Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 4:39pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/5 "2023-12-01T16:39:37Z")

</div>

Hey Stephen,

Yeah, I got stuck on this step, I get the same results as you with my other pipeline using the disect processor. So back to my initial problem, why is it that that my logs don't get parsed (this is from my myapp.log which I have in my path in the elastic-agent.yml):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/6/96c31a91ace6d580d2193f1cb3ecbc04eb7baafd.png)

There is no _project_, _last\_activity_ and _log\_time_ field.  
Only:  
"message": """{"log\_time":"2023-11-28T09:50:33.026Z","project":"Public\_Documentation\_Mappings","last\_activity":"2023-11-25T09:50:33.026Z"}"""

It's like it never goes through the pipeline before getting indexed  
I can show you **View details** from the stream section for each log that it reads from _myapp.log_

Thanks for your time Stephen.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 4:48pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/6 "2023-12-01T16:48:25Z")

</div>

See [here](https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme)

> ## Elastic data stream naming scheme
> 
> The Elastic data stream naming scheme is made for time series data and consists of splitting datasets into different data streams using the following naming convention.
> 
> - **type** : Generic type describing the data
> - **dataset** : Describes the data ingested and its structure
> - **namespace** : User-configurable arbitrary grouping
> 
> These three parts are combined by a “-” and result in data streams like `logs-nginx.access-production`. In all three parts, the “-” character is not allowed. This means all data streams are named in the following way:

so  
` data_stream.dataset: logs-generic-default`

is not allowed... you are making assumptions

My suggestion is to following the instructions exactly get it working and **then** start changing names etc.. putting those `-` s in definitely part of the problem

**And with that it looks like the [first page](https://www.elastic.co/guide/en/observability/8.11/logs-stream.html) and the second are NOT aligned UGH!!**

So looking at the next page you should set this in your agent

` data_stream.dataset: example`

> `index_pattern` – Needs to match your log data stream. Naming conventions for data streams are `<type>-<dataset>-<namespace>` . In this example, your logs data stream is named `logs-example-*` . Data that matches this pattern will go through your pipeline.

Which then will be aligned with the template and everything on [this](https://www.elastic.co/guide/en/observability/8.11/logs-parse.html) page

That now aligns with

```auto
PUT _index_template/logs-example-default-template
{
  "index_patterns": ["logs-example-*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.default_pipeline":"logs-example-default"
    }
  },
  "composed_of": [
    "logs-mappings",
    "logs-settings",
    "logs@custom",
    "ecs@dynamic_templates"
  ],
  "ignore_missing_component_templates": ["logs@custom"]
}

```

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:23pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/7 "2023-12-01T17:23:57Z")

</div>

Hey Stephen,

Yeah, Ill change it back to _generic_, I changed it because I was just experimenting with everything to apply the pipeline... forgot to change it back.  
But I can say with confidence that even with the default elastic-agent.yml (following the _Stream any log file_) it still didn't parse correctly. The only thing I added was the credentials and path.

As of right now I changed my .yml file to look like this:

```auto
outputs:
  default:
    type: elasticsearch
    hosts: '<host>:<port>'
    #api_key: 'your-api-key'
    username: <user>
    password: <pass>
inputs:
  - id: your-log-id
    type: filestream
    streams:
      - id: your-log-stream-id
        data_stream.dataset: generic
        paths:
          - C:\Program Files\Elastic\Agent\data\elastic-agent-03ef9d\logs\myapp.log
        

```

Regarding this:  
`index_pattern` – Needs to match your log data stream. Naming conventions for data streams are `<type>-<dataset>-<namespace>` . In this example, your logs data stream is named `logs-example-*` . Data that matches this pattern will go through your pipeline.

Is it the id in the inputs -\> streams - id ?

I really appreciate the time Stephen.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:25pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/8 "2023-12-01T17:25:58Z")

</div>

> [@Kyps](#):
>
> ` data_stream.dataset: generic`

No set it to `example` if you want it to work with the 2nd page... I just found that

` data_stream.dataset: example`

All the code on the Parsing Page expects the `dataset` to be `example`

I reported this to our docs people... that is not good..

So follow the first and second page but use in the agent.yml

` data_stream.dataset: example`

then try the json parser I gave you in the ingest pipeline.

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:30pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/9 "2023-12-01T17:30:59Z")

</div>

Hey Stephen,

I changed data\_stream.dataset to _example_ (the listener restarted) and then I added a new log to myapp.log and although it still doesn't parse correctly, it looks like the dataset is still _generic_?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a98b12c7e7d46649d71192c71cb41137a077bfd0.png)

What do you think?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:35pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/10 "2023-12-01T17:35:52Z")

</div>

> [@Kyps](#):
>
> ```auto
> inputs:
> - id: your-log-id
> type: filestream
> streams:
> - id: your-log-stream-id
> data_stream.dataset: example
> paths:
> - C:\Program Files\Elastic\Agent\data\elastic-agent-03ef9d\logs\myapp.log
>         
> 
> ```

Did you restart the agent?

Does not look like it

 ![Screenshot 2023-12-01 at 9.35.29 AM](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a6011158feceb852eb8b611649db80994c08aee.png)

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:37pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/11 "2023-12-01T17:37:36Z")

</div>

I run this in Powershell

Stop-Service "Elastic Agent"  
Start-Service "Elastic Agent"

In the root of the Agent, this folder:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/d/adb299a8388b6252f35e020308d05a87f8aaa7d8.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:38pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/12 "2023-12-01T17:38:42Z")

</div>

And you saved the file

`C:\Program Files\Elastic\Agent\elastic-agent.yml`

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:39pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/13 "2023-12-01T17:39:06Z")

</div>

Yes, sir!

Edit: is there a cache?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:42pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/14 "2023-12-01T17:42:51Z")

</div>

not sure what to tell you....

Uninstall and reinstall... and start over

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:45pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/15 "2023-12-01T17:45:19Z")

</div>

Alright, will do.

FYI: even with the logs that were coming in _elastic-agent-20231201-3.ndjson_ that is the default filebeat ("Non-zero metrics in the last 30s" messages) with no custom field names, when I copied those logs into my myapp.log it still didn't parse correctly. That is the the entire log was in the _message_ field

But I'll reinstall and start over.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:53pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/16 "2023-12-01T17:53:36Z")

</div>

OK Need to slow down a bit...

Get everything aligned....

The docs have some issues, sorry about that...

> [@Kyps](#):
>
> FYI: even with the logs that were coming in _elastic-agent-20231201-3.ndjson_ that is the default filebeat ("Non-zero metrics in the last 30s" messages) with no custom field names, when I copied those logs into my myapp.log it still didn't parse correctly. That is the the entire log was in the _message_ field

Of course because the ingest pipeline is not getting executed because the template is not getting applied which defines the pipelei etc...etc...etc...etc.. because the dateset is wrong it is all related...

Let me work though this I will get back... it is all close just an issue or 2

Please Verify you are using standalone or are you do you have a Fleet Server?

If so there is easier ways to do this... Standalone is fine I just want to know what you have

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:54pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/17 "2023-12-01T17:54:43Z")

</div>

Alright,

Thanks a lot for the help Stephen!

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:56pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/18 "2023-12-01T17:56:39Z")

</div>

I'm certain I pressed 'n' during the setup when it ask for something 'fleet', so Standalone.

I followed this in the _Stream any log file_ guide:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d75f059b2270e80c788f63c8c57d89963a1d531.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 5:58pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/19 "2023-12-01T17:58:46Z")

</div>

Right but you are NOT doing Fleet Managed... (i.e. you did not install a Fleet Server)  
Looks like that is correct .. no fleet server .. ok no problem give me 20 mins...

This should not be this hard... sorry... I will be doing on Linux but should translate

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 5:59pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/20 "2023-12-01T17:59:23Z")

</div>

No I did not install any Fleet Server

[Next page](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268.md?page=2)