# Logstash

**URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=397

[Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md)

**Page:** 398

---

## [Single line files](https://discuss.elastic.co/t/single-line-files/221637)

<div class="topic-metadata">

**Author:** [@lacerated](https://discuss.elastic.co/u/lacerated)\
**Replies:** 0\
**Last updated:** [March 2, 2020, 7:50am UTC](https://discuss.elastic.co/t/single-line-files/221637 "2020-03-02T07:50:20Z")

</div>

i'm trying to make a pipeline that sends xml documents to elasticsearch. Problem is that each document is in its own separate file as a single line without \\n in the end. Any way to tell logstash not to wait for \\n but …

---

## [Logstash port is not opening](https://discuss.elastic.co/t/logstash-port-is-not-opening/221564)

<div class="topic-metadata">

**Author:** [@tbhaxor](https://discuss.elastic.co/u/tbhaxor)\
**Replies:** 4\
**Last updated:** [March 2, 2020, 5:58am UTC](https://discuss.elastic.co/t/logstash-port-is-not-opening/221564 "2020-03-02T05:58:28Z")

</div>

I have configured the logstash to bind 0.0.0.0:9600, after giving logstash service some time, the output of netstat -tunlpa | grep 9600 is empty # journalctl -xe | grep logstash | tail -30 Mar 01 17:38:09 soc logstash\[1…

---

## [Corona Virus data](https://discuss.elastic.co/t/corona-virus-data/221535)

<div class="topic-metadata">

**Author:** [@Rob\_wylde](https://discuss.elastic.co/u/Rob_wylde)\
**Replies:** 1\
**Last updated:** [March 1, 2020, 2:33pm UTC](https://discuss.elastic.co/t/corona-virus-data/221535 "2020-03-01T14:33:04Z")

</div>

So i'd like to try and generate a better dashboard within kibana to display the data that has been made available by john hopkins below are the link to that data. Time series and Github repo Header looks like this Pro…

---

## [Logstash filter to match jdbc identifier with yaml file](https://discuss.elastic.co/t/logstash-filter-to-match-jdbc-identifier-with-yaml-file/220921)

<div class="topic-metadata">

**Author:** [@ads](https://discuss.elastic.co/u/ads)\
**Replies:** 4\
**Last updated:** [February 29, 2020, 6:14pm UTC](https://discuss.elastic.co/t/logstash-filter-to-match-jdbc-identifier-with-yaml-file/220921 "2020-02-29T18:14:59Z")

</div>

Hi, I am using ES and logstash version 7.6.0. I am trying to plot a set of UK postcodes into maps but am not having much success. I have a set of dates and postcodes from a jdbc call. This works using the following quer…

---

## [Pipeline was Ignoring Logstash](https://discuss.elastic.co/t/pipeline-was-ignoring-logstash/221343)

<div class="topic-metadata">

**Author:** [@muhjaury](https://discuss.elastic.co/u/muhjaury)\
**Replies:** 5\
**Last updated:** [February 29, 2020, 3:38pm UTC](https://discuss.elastic.co/t/pipeline-was-ignoring-logstash/221343 "2020-02-29T15:38:43Z")

</div>

I'm newbie for using Logstash and Elasticsearch. I wanted to sync my MongoDB data into Elasticsearch using Logstash Plugin (logstash-input-mongodb). In my mongodata.conf is input { uri =\> 'mongodb://127.0.0.1:27017/f…

---

## [Java error when too many conditions in Logstash](https://discuss.elastic.co/t/java-error-when-too-many-conditions-in-logstash/221466)

<div class="topic-metadata">

**Author:** [@syunusic](https://discuss.elastic.co/u/syunusic)\
**Replies:** 3\
**Last updated:** [February 29, 2020, 3:25pm UTC](https://discuss.elastic.co/t/java-error-when-too-many-conditions-in-logstash/221466 "2020-02-29T15:25:35Z")

</div>

I have a source (CSV) that has one field called EVENTNO. In that field are numbers (INT). In another document I have what EVENTNO number means (e.g. "1" = "The machine is off", "2" = "The machine is on", and so on). Tha…

---

## [Leap year problem on Logstash date filter](https://discuss.elastic.co/t/leap-year-problem-on-logstash-date-filter/221512)

<div class="topic-metadata">

**Author:** [@taro\_x](https://discuss.elastic.co/u/taro_x)\
**Replies:** 0\
**Last updated:** [February 29, 2020, 8:27am UTC](https://discuss.elastic.co/t/leap-year-problem-on-logstash-date-filter/221512 "2020-02-29T08:27:54Z")

</div>

Logs in the following formats are now failing: "timestamp\_str": "Feb 29 00:00:00" filter config: filter { date { match =\> \["timestamp\_str", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"\] timezone =\> "Asia/Tokyo…

---

## [Logstash with TCP Input but Error](https://discuss.elastic.co/t/logstash-with-tcp-input-but-error/221504)

<div class="topic-metadata">

**Author:** [@kopilim96](https://discuss.elastic.co/u/kopilim96)\
**Replies:** 0\
**Last updated:** [February 29, 2020, 3:43am UTC](https://discuss.elastic.co/t/logstash-with-tcp-input-but-error/221504 "2020-02-29T03:43:21Z")

</div>

I am using Logstash version 7.5.2 and below is my conf file for tcp input test input { tcp { port =\> "5666" host =\> "localhost" type =\> "syslog" } } filter { grok { patterns\_dir =\> \["/master\_pattern"\] match =\> …

---

## [Output: specify document id \*or\* allow elasticsearch to pick](https://discuss.elastic.co/t/output-specify-document-id-or-allow-elasticsearch-to-pick/221486)

<div class="topic-metadata">

**Author:** [@Supermathie](https://discuss.elastic.co/u/Supermathie)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 8:26pm UTC](https://discuss.elastic.co/t/output-specify-document-id-or-allow-elasticsearch-to-pick/221486 "2020-02-28T20:26:29Z")

</div>

Sometimes on injest we have an id we want to keep as the document id: if \[trace\_id\] { mutate { copy =\> { "trace\_id" =\> "\[@metadata\]\[document\_id\]" } remove\_field =\> "trace\_id" } } so …

---

## [Logstash date to datetime format](https://discuss.elastic.co/t/logstash-date-to-datetime-format/221446)

<div class="topic-metadata">

**Author:** [@michal\_mastro](https://discuss.elastic.co/u/michal_mastro)\
**Replies:** 2\
**Last updated:** [February 28, 2020, 7:55pm UTC](https://discuss.elastic.co/t/logstash-date-to-datetime-format/221446 "2020-02-28T19:55:44Z")

</div>

Hi, Im using logstash to get data from mysql and pass them to elasticsearch. Unfortunately date format is default parsed to zulu time. In need get data in datetime format like yyyy-mm-dd H:i:s without t and z. Below …

---

## [Data Dropped by Logstash](https://discuss.elastic.co/t/data-dropped-by-logstash/221465)

<div class="topic-metadata">

**Author:** [@kunal.1709](https://discuss.elastic.co/u/kunal.1709)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 5:02pm UTC](https://discuss.elastic.co/t/data-dropped-by-logstash/221465 "2020-02-28T17:02:34Z")

</div>

Hello Everyone, While debugging the issue with TCP input with logstash, I have introduced Rsyslog as an intermediate between SIEM and Logstash as a temporary workaround. Please refer to below link - SIEM -\> Rsyslog (…

---

## [Unable to split message received on TCP port into individual records](https://discuss.elastic.co/t/unable-to-split-message-received-on-tcp-port-into-individual-records/218836)

<div class="topic-metadata">

**Author:** [@kunal.1709](https://discuss.elastic.co/u/kunal.1709)\
**Replies:** 3\
**Last updated:** [February 28, 2020, 4:25pm UTC](https://discuss.elastic.co/t/unable-to-split-message-received-on-tcp-port-into-individual-records/218836 "2020-02-28T16:25:51Z")

</div>

Hi everyone, I'm sending some network logs from RSA Netwitness SIEM to a logstash server over TCP on port 5515. Currently logstash is not able to separate the incoming data into separate records, its considering the en…

---

## [Loosing zero in a log line](https://discuss.elastic.co/t/loosing-zero-in-a-log-line/221262)

<div class="topic-metadata">

**Author:** [@AdamJade](https://discuss.elastic.co/u/AdamJade)\
**Replies:** 2\
**Last updated:** [February 28, 2020, 3:21pm UTC](https://discuss.elastic.co/t/loosing-zero-in-a-log-line/221262 "2020-02-28T15:21:42Z")

</div>

Hi there Here is my log line \[2020-02-11 09:55:39.595\] \[Value1\] \[Value2\] \[Value3\] \[Value4\] \[Value5\] \[Value6\] \[Value7\] # !! Détail code retour: 150 my grok grok { match =\> { "message" =\> "\[%{TIMESTAMP\_ISO8601:logDat…

---

## [Unable to install logstash=1:7.5.2-1](https://discuss.elastic.co/t/unable-to-install-logstash-1-7-5-2-1/221454)

<div class="topic-metadata">

**Author:** [@Chandrashekar467](https://discuss.elastic.co/u/Chandrashekar467)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 3:14pm UTC](https://discuss.elastic.co/t/unable-to-install-logstash-1-7-5-2-1/221454 "2020-02-28T15:14:12Z")

</div>

Hi all, I am unable to install logstash=1:7.5.2-1,getting below error. Installing new version of config file /etc/logstash/logstash.yml ... /etc/logstash/startup.options is not a file path To manually specify a start…

---

## [Best way to send a log from filebeat to his respective logstash conf](https://discuss.elastic.co/t/best-way-to-send-a-log-from-filebeat-to-his-respective-logstash-conf/221449)

<div class="topic-metadata">

**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 2:37pm UTC](https://discuss.elastic.co/t/best-way-to-send-a-log-from-filebeat-to-his-respective-logstash-conf/221449 "2020-02-28T14:37:18Z")

</div>

Filebeat consume many logs with different type of data, ex: -cpu.log -filesystem.log -memory.log -uptime.log Then this data is sended to logstash, and I have multiple logstash pipelines configurations, what will be …

---

## [Logstash dateparse failure error](https://discuss.elastic.co/t/logstash-dateparse-failure-error/221258)

<div class="topic-metadata">

**Author:** [@BIs1](https://discuss.elastic.co/u/BIs1)\
**Replies:** 3\
**Last updated:** [February 28, 2020, 12:35pm UTC](https://discuss.elastic.co/t/logstash-dateparse-failure-error/221258 "2020-02-28T12:35:14Z")

</div>

Hi, I am trying to use grok filter to change some indexes and format. Also trying to change the date format but comes with error "dateparsefailure". Please help me identify the mistake I made in following grok pattern. …

---

## [How to config properly to send logs from Spring Boot to Logstash and Logstash send to Elasticsearch](https://discuss.elastic.co/t/how-to-config-properly-to-send-logs-from-spring-boot-to-logstash-and-logstash-send-to-elasticsearch/221422)

<div class="topic-metadata">

**Author:** [@franzisk](https://discuss.elastic.co/u/franzisk)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 11:56am UTC](https://discuss.elastic.co/t/how-to-config-properly-to-send-logs-from-spring-boot-to-logstash-and-logstash-send-to-elasticsearch/221422 "2020-02-28T11:56:14Z")

</div>

Hi there, I have this scenario where I have a Spring Boot application using Logback appenders to send logs to Logstash: \<?xml version="1.0" encoding="UTF-8"?\> \<!-- CONSOLE OUTPUT --\> \<appender name="STDOUT" clas…

---

## [Extracting fields from a linked xml file](https://discuss.elastic.co/t/extracting-fields-from-a-linked-xml-file/221402)

<div class="topic-metadata">

**Author:** [@djb000m](https://discuss.elastic.co/u/djb000m)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 10:33am UTC](https://discuss.elastic.co/t/extracting-fields-from-a-linked-xml-file/221402 "2020-02-28T10:33:48Z")

</div>

Hi, I am indexing xml files using Logstash, which is working fine. Does anyone know if it is possible to grab values from an xml file linked to from within the xml file being processed by Logstash? Specifically, my da…

---

## [Logstash can't communicate to es and attempted to resurrect dead es instance](https://discuss.elastic.co/t/logstash-cant-communicate-to-es-and-attempted-to-resurrect-dead-es-instance/221385)

<div class="topic-metadata">

**Author:** [@zeghitsama](https://discuss.elastic.co/u/zeghitsama)\
**Replies:** 1\
**Last updated:** [February 28, 2020, 9:16am UTC](https://discuss.elastic.co/t/logstash-cant-communicate-to-es-and-attempted-to-resurrect-dead-es-instance/221385 "2020-02-28T09:16:19Z")

</div>

Hi, I'm getting error on logstash that want to communicate to elasticsearch. I've already set my username and password on conf.d. Here's my detail on my conf file. input { beats { id =\> "01-beat…

---

## [How to increase the number of threads at Logstash output pipeline](https://discuss.elastic.co/t/how-to-increase-the-number-of-threads-at-logstash-output-pipeline/221391)

<div class="topic-metadata">

**Author:** [@suvarna](https://discuss.elastic.co/u/suvarna)\
**Replies:** 1\
**Last updated:** [February 28, 2020, 9:00am UTC](https://discuss.elastic.co/t/how-to-increase-the-number-of-threads-at-logstash-output-pipeline/221391 "2020-02-28T09:00:15Z")

</div>

Hi The pipeline.workers setting determines how many threads to run for filter and output processing. If you we want to set number of threads at input side or output side, what configuration we have to add in yml.

---

## [Tuning Logstash for optimal throughput for ELK pipeline](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007)

<div class="topic-metadata">

**Author:** [@kumarvikash1](https://discuss.elastic.co/u/kumarvikash1)\
**Replies:** 3\
**Last updated:** [February 28, 2020, 8:21am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007 "2020-02-28T08:21:55Z")

</div>

We have Kafka-\>ELK pipeline and logstash is one of the key component in between Kafka and Elastic Search. Looks like Logstash is not able to process data at the same rate as injecting into kafka. We have below logstash …

---

## [Cannot access @metadata of dead\_letter\_queue in 7.6.0](https://discuss.elastic.co/t/cannot-access-metadata-of-dead-letter-queue-in-7-6-0/221240)

<div class="topic-metadata">

**Author:** [@roflmao](https://discuss.elastic.co/u/roflmao)\
**Replies:** 1\
**Last updated:** [February 28, 2020, 7:52am UTC](https://discuss.elastic.co/t/cannot-access-metadata-of-dead-letter-queue-in-7-6-0/221240 "2020-02-28T07:52:05Z")

</div>

dead\_letter\_queue { path =\> "/usr/share/logstash/data/dead\_letter\_queue/" pipeline\_id =\> "main" commit\_offsets =\> true } Trying to do stdout { codec =\> rubycodec { metadata =\> true } fails logstash from starting: …

---

## [Field merge](https://discuss.elastic.co/t/field-merge/221165)

<div class="topic-metadata">

**Author:** [@111291](https://discuss.elastic.co/u/111291)\
**Replies:** 9\
**Last updated:** [February 28, 2020, 7:24am UTC](https://discuss.elastic.co/t/field-merge/221165 "2020-02-28T07:24:24Z")

</div>

Hello, Please tell me what is my mistake? grok { match =\> { "message" =\> \["%{INT:Min}:%{BASE10NUM:Sec}-%{INT:Duration},(%{WORD:Event}|%{SPACE:Event}),%{INT:Level}"\] } match =\> { "log.file.path" =\> \["%{INT:T…

---

## [Expected one of \\"\\\\\\\\'\\", any character, \\"'\\"](https://discuss.elastic.co/t/expected-one-of-any-character/221249)

<div class="topic-metadata">

**Author:** [@ghl1l0](https://discuss.elastic.co/u/ghl1l0)\
**Replies:** 5\
**Last updated:** [February 28, 2020, 7:12am UTC](https://discuss.elastic.co/t/expected-one-of-any-character/221249 "2020-02-28T07:12:11Z")

</div>

hello elastic community i need your help, i got this error when i have add the function rename on mutate this is the full logs error : \[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::P…

---

## [Prepare-offline-pack subcommand ERROR](https://discuss.elastic.co/t/prepare-offline-pack-subcommand-error/221362)

<div class="topic-metadata">

**Author:** [@lznfs](https://discuss.elastic.co/u/lznfs)\
**Replies:** 0\
**Last updated:** [February 28, 2020, 4:37am UTC](https://discuss.elastic.co/t/prepare-offline-pack-subcommand-error/221362 "2020-02-28T04:37:46Z")

</div>

HI,when i install logstash offline plugins,I get this error： my server can not connect internet,so,i build my offline plugin pack in my another server that can connect internet. os:rhel7.4 logstash:7.6 \[root@logstash…

---

## [Index not rolling over since upgrading to 7.6](https://discuss.elastic.co/t/index-not-rolling-over-since-upgrading-to-7-6/221145)

<div class="topic-metadata">

**Author:** [@martine](https://discuss.elastic.co/u/martine)\
**Replies:** 4\
**Last updated:** [February 28, 2020, 1:15am UTC](https://discuss.elastic.co/t/index-not-rolling-over-since-upgrading-to-7-6/221145 "2020-02-28T01:15:58Z")

</div>

On earlier versions of Logstah and Elastic search a new index would be created every day with the naming in the form of logstash-%{+YYYY.MM.dd}. After upgrading 7.6 from 6.8 indexes started be created with errors like: …

---

## [Check if logstash is working](https://discuss.elastic.co/t/check-if-logstash-is-working/221281)

<div class="topic-metadata">

**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\
**Replies:** 1\
**Last updated:** [February 27, 2020, 9:47pm UTC](https://discuss.elastic.co/t/check-if-logstash-is-working/221281 "2020-02-27T21:47:11Z")

</div>

I have a filebeat running and I can see in the logs that it is parsing my log files. The check config worked and it stated that everything is ok. I goto logstash and I have a filebeat.conf in the conf.d folder. Inside…

---

## [Logstash EXEC input](https://discuss.elastic.co/t/logstash-exec-input/221295)

<div class="topic-metadata">

**Author:** [@RunningSmurf](https://discuss.elastic.co/u/RunningSmurf)\
**Replies:** 3\
**Last updated:** [February 27, 2020, 7:00pm UTC](https://discuss.elastic.co/t/logstash-exec-input/221295 "2020-02-27T19:00:00Z")

</div>

Hello, I have a simple EXEC input in logstash, as follows: input { exec { command =\> "ls" interval =\> 30 } } output { elasticsearch { hosts =\> \["\<es\_host\>:9200"\] } } However, I keep getting the following err…

---

## [Logstash with docker-compose exists after some time running](https://discuss.elastic.co/t/logstash-with-docker-compose-exists-after-some-time-running/221284)

<div class="topic-metadata">

**Author:** [@franzisk](https://discuss.elastic.co/u/franzisk)\
**Replies:** 0\
**Last updated:** [February 27, 2020, 4:52pm UTC](https://discuss.elastic.co/t/logstash-with-docker-compose-exists-after-some-time-running/221284 "2020-02-27T16:52:59Z")

</div>

I am running Logstash with ELK stack but after some time running it just stops, besides, when connecting with Logback from Spring Bot application it always gives me: \> -WARN in net.logstash.logback.appender.Logstas…

---

## [Trying to parse when LOGLEVEL is ERROR](https://discuss.elastic.co/t/trying-to-parse-when-loglevel-is-error/220378)

<div class="topic-metadata">

**Author:** [@dbviz](https://discuss.elastic.co/u/dbviz)\
**Replies:** 4\
**Last updated:** [February 27, 2020, 10:44am UTC](https://discuss.elastic.co/t/trying-to-parse-when-loglevel-is-error/220378 "2020-02-27T10:44:36Z")

</div>

Hello, Im trying to parse only LOGLEVEL field using grok but it doesn't work. My logstash.log comes from : \[2020-02-20T15:55:46,198\]\[ERROR\]\[logstash.inputs.jdbc \] Java::ComSybaseJdbc3Jdbc::SybSQLException: Differe…

[Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=396)

[Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=398)