# Parsing problem when streaming a log file

**URL:** <https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268>\
**Category:** Kibana\
**Created:** [November 29, 2023, 5:49pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268 "2023-11-29T17:49:37Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 6:18pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/21 "2023-12-01T18:18:31Z")

</div>

So did you actually delete the data stream and all before you restarted...???

Stack Management Data Streams etc...

I am going through the example right now

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 6:33pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/22 "2023-12-01T18:33:07Z")

</div>

I don't think I did,

All I did was stop the service and start it up again

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 6:33pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/23 "2023-12-01T18:33:52Z")

</div>

While I am going through the rest of this... try this... note the corrected syntax on the dataset

> [@Kyps](#):
>
> ```auto
> inputs:
> - id: your-log-id
> type: filestream
> streams:
> - id: your-log-stream-id
> data_stream:
> dataset: example
> paths:
> - C:\Program Files\Elastic\Agent\data\elastic-agent-03ef9d\logs\myapp.log
>         
> 
> ```

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 6:37pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/24 "2023-12-01T18:37:35Z")

</div>

It changed!

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/b/dbd4d26af84d32bb2c61400da3ac76dadab17634.png)

The entire log is still in the message field though 😕

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/9/49ec4feee910012473d1ed2ec3e99d021d3a7dc9.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 6:45pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/25 "2023-12-01T18:45:34Z")

</div>

Yeah because it is still writing to the old index...

I got it working... you were close...

Steps Stop The Agent

Go to Kibana -\> Index Management and Delete the logs-example-default and logs-generic-default data streams

then Put these notice the names above you using generic etc..

```auto
PUT _index_template/logs-example-default-template
{
  "index_patterns": ["logs-example-*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.default_pipeline":"logs-example-default"
    }
  },
  "composed_of": [
    "logs-mappings",
    "logs-settings",
    "logs@custom",
    "ecs@dynamic_templates"
  ],
  "ignore_missing_component_templates": ["logs@custom"]
}

PUT _ingest/pipeline/logs-example-default
{
  "description": "Extracts the timestamp and log level",
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "message_details"
      }
    }
  ]
}

```

Agent File

```auto
outputs:
  default:
    type: elasticsearch
    hosts: '<host:port>
    api_key: '<apikey>'
inputs:
  - id: your-log-id
    type: filestream
    streams:
      - id: your-log-stream-id
        data_stream:
          dataset: example
        paths:
          # - /var/log/*.log
          - /opt/Elastic/Agent/elastic-agent-*.ndjson

```

start the agent

---

<div class="post-metadata">

**Author:** ![Kyps](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyps/32/128241_2.png) [@Kyps](https://discuss.elastic.co/u/Kyps)\
**Post date:** [December 1, 2023, 6:56pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/26 "2023-12-01T18:56:35Z")

</div>

I can't thank you enough, Stephen

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/7/67c4555ba8c5f6748db566f0d47a14f7cc3ed941.png)

I really appreciate the time you had to put up with me.

Thank you again!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 6:59pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/27 "2023-12-01T18:59:02Z")

</div>

No it was fine... 1 little glitch in the docs!  
Thanks for hanging in!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2023, 6:59pm UTC](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268/28 "2023-12-29T18:59:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/parsing-problem-when-streaming-a-log-file/348268.md?page=1)