# Display text file in Kibana with filebeat and logstash

**URL:** <https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692>\
**Category:** Logstash\
**Created:** [July 10, 2019, 8:37am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692 "2019-07-10T08:37:10Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 10, 2019, 8:37am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/1 "2019-07-10T08:37:11Z")

</div>

Hi guys,

I run ELK Stack (one machine) with filebeat (second machine) as log shipper.

I've got txt file with 4228 rows where each row is one log in follow form:

Jul 4 13:56:17 vMMR mmr-core[29839]: GtsAwegAOMTbez\_1562241377271986.mt npdbProfiling-end: pid[29839] table[npdbcz] operation[SELECT] duration[6.27 ms] error sql[SELECT carrier,validity,now()\>validity as now\_valid FROM npdbcz WHERE `range` IN ('606339842','60633984','6063398','606339','60633','6063') ORDER BY now\_valid DESC,validity DESC]

How can I send the txt file with these logs to the elasticsearch and see them in Kibana? Which configuration files should I edit and how? Is it possible to send each row in the file as one log message in Kibana?

Any help is really appreciated !!

Thank you 😊

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 10, 2019, 12:56pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/2 "2019-07-10T12:56:27Z")

</div>

I'm trying to use logstash for load .txt file into elasticsearch. Here is my config file loadfile.conf:

```
input {
 file {
  path => "/home/vladuser/logs.txt"
  start_position => "beginning"
 }
}

output {
  elasticsearch {
   hosts => ["127.0.0.1:9200"]
   index => "ssh_auth-2019.07"
  }
}

```

I tried change hosts to localhost:9200 but it doesn't work either.

Unfortunately when I'm trying to run `sudo bin/logstash -f /etc/logstash/conf.d/loadfile.conf` I end up with this message:

**[INFO] 2019-07-10 12:49:43.061 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9601}**

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 10, 2019, 1:51pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/3 "2019-07-10T13:51:37Z")

</div>

Try adding

```
sincedb_path => "/dev/null"

```

to your file input and restarting logstash.

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 10, 2019, 4:00pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/4 "2019-07-10T16:00:00Z")

</div>

Thank you for your help!

Now it goes further but for each row in the .txt file it shows this WARN:

> [WARN] 2019-07-10 15:55:52.020 [[main]\>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"ssh\_auth-2019.07", :\_type=\>"\_doc", :routing=\>nil}, #LogStash::Event:0x64e46377], :response=\>{"index"=\>{"\_index"=\>"ssh\_auth-2019.07", "\_type"=\>"\_doc", "\_id"=\>"aYKZ3GsBKh5qbaH2ItLV", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}

The index where I want to send it "ssh\_auth-2019.07" is already created and connected with Filebeat. Could output part in logstash config file create new index for the .txt file?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 10, 2019, 4:09pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/5 "2019-07-10T16:09:31Z")

</div>

Does [this](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/4) help?

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 10, 2019, 5:06pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/6 "2019-07-10T17:06:53Z")

</div>

I added this filter

```
filter {
  mutate {
rename {
  "[host]" => "[host][name]"
}
  }
}

```

And when I run `sudo bin/logstash -f /etc/logstash/conf.d/loadfile.conf` it returns me another ERROR:

> [ERROR] 2019-07-10 17:01:17.788 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 14, column 12 (byte 156) after filter {\n mutate {\n rename ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}

I have no idea what it means especially that it expects # after filter ???

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 10, 2019, 5:27pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/7 "2019-07-10T17:27:30Z")

</div>

> [@Vladpov](#):
>
> rename {

That should be

```
rename => {

```

The error message is telling you that it expects to find either a comment or =\> after

```
filter {
  mutate {
     rename

```

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 10, 2019, 6:44pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/8 "2019-07-10T18:44:26Z")

</div>

OK, don't know if it is right solution but I somehow created new index (my\_index) which I found in Index Patterns in Kibana. Then without any filters I change output index for my\_index.

After I ran `sudo bin/logstash -f /etc/logstash/conf.d/loadfile.conf` I ended up with the same output as in beginning:

**[INFO] 2019-07-10 12:49:43.061 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9601}**

Although this output showed up I can see the rows from my file as logs in Kibana interface in Discover bar.

 ![Sn%C3%ADmek%20obrazovky%20(8)](https://us1.discourse-cdn.com/elastic/original/3X/4/4/4461bf6e8b5b2c6831f532cb48bffdf75fc116cc.png)

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 12, 2019, 9:53am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/9 "2019-07-12T09:53:01Z")

</div>

@Badger Could you please help 🙏

I need to parse my data from the txt file so I can get duration of the translation I can querying later. As you can see my file contents these rows:

`> Jul 4 13:56:17 vMMR mmr-core[29839]: GtsAwegAOMTbez_1562241377271986.mt npdbProfiling-end: pid[29839] table[npdbcz] operation[SELECT] duration[6.27 ms] error [] sql[SELECT carrier,validity,now()>validity as now_valid FROM npdbcz WHERE`range`IN ('606339842','60633984','6063398','606339','60633','6063') ORDER BY now_valid DESC,validity DESC]`

I'd like to separate it to the fields as it is : pid, table, operation, **duration**... the most important for me is duration.

Do you have any idea how to manage it? I'm a beginner and don't understand grok so much but if you have any hints I'd be really thankful!!

`

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 12, 2019, 2:50pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/10 "2019-07-12T14:50:31Z")

</div>

I've got this in my grok debugger so far:

```
%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:type} %{SPACE} %{DATA:file_id} %{DATA:file_name} %{DATA:syslog_pid} .*table\s*\[%{WORD:table}\] .*operation\s*\[%{WORD:operation}\] %{GREEDYDATA:rest}

{
  "rest": "duration[6.27 ms] error [] sql[SELECT carrier,validity,now()>validity as now_valid FROM npdbcz WHERE `range` IN ('606339842','60633984','6063398','606339','60633','6063') ORDER BY now_valid DESC,validity",
  "syslog_pid": "pid[29839]",
  "file_name": "npdbProfiling-end:",
  "type": "mmr-core[29839]:",
  "hostname": "vMMR",
  "syslog_timestamp": "Jul 4 13:56:17",
  "file_id": "GtsAwegAOMTbez_1562241377271986.mt",
  "operation": "SELECT",
  "table": "npdbcz"
}

```

I have problem to get parsed duration 😅 When I try to skip the word duration as I did it before with operation or table grok shows me an error that it doesn't match ☹

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 12, 2019, 5:14pm UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/11 "2019-07-12T17:14:47Z")

</div>

[and] have to be escaped because they are used to delimit character groups in a regexp. Try

```
grok { match => { "rest" => "^duration\[%{NUMBER:d:float} ms\]" } }
```

---

<div class="post-metadata">

**Author:** ![Vladpov](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@Vladpov](https://discuss.elastic.co/u/Vladpov)\
**Post date:** [July 13, 2019, 10:29am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/12 "2019-07-13T10:29:44Z")

</div>

Thank you very much! It works nicely 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2019, 10:29am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692/13 "2019-08-10T10:29:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.