# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=517 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 518 --- ## [Logstash 7.0.1 - CSV upload with subheaders](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399)
**Author:** [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Replies:** 5\ **Last updated:** [July 2, 2019, 11:17pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399 "2019-07-02T23:17:34Z")
Hello, I am trying to upload a csv file using Logstash 7.0.1 that contains subheaders. For example: "netflow": { "flow\_seq\_num": 55729, "flowset\_id": 259, "flow\_end\_msec": 1562002327000, "version… --- ## [Elastic Stack 7.0 date\_time\_parse\_exception](https://discuss.elastic.co/t/elastic-stack-7-0-date-time-parse-exception/188542)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 7\ **Last updated:** [July 2, 2019, 8:13pm UTC](https://discuss.elastic.co/t/elastic-stack-7-0-date-time-parse-exception/188542 "2019-07-02T20:13:16Z")
Not sure what happened, but overnight I started getting this logged constantly in my Logstash-plain.log. No changes have been made in the last week or two since upgrading everything to version 7. About 3k entries in th… --- ## [Warnning after 7.2](https://discuss.elastic.co/t/warnning-after-7-2/188566)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 7:29pm UTC](https://discuss.elastic.co/t/warnning-after-7-2/188566 "2019-07-02T19:29:25Z")
/usr/share/logstash/bin/logstash -f usages.conf WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-… --- ## [Multiple templates from logstash](https://discuss.elastic.co/t/multiple-templates-from-logstash/188576)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 0\ **Last updated:** [July 2, 2019, 7:28pm UTC](https://discuss.elastic.co/t/multiple-templates-from-logstash/188576 "2019-07-02T19:28:58Z")
It seems in a Logstash pipeline, we can only specify a template. How can I push more than one template from Logstash to Elasticsearch? I don't want to dynamically put the template. --- ## [Sending logging data to ES using tcp input plugin, no use of file](https://discuss.elastic.co/t/sending-logging-data-to-es-using-tcp-input-plugin-no-use-of-file/188159)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 7\ **Last updated:** [July 2, 2019, 5:56pm UTC](https://discuss.elastic.co/t/sending-logging-data-to-es-using-tcp-input-plugin-no-use-of-file/188159 "2019-07-02T17:56:48Z")
Hi All We get very good log file in LS, but i want to mute this file logging and send the logging data directly to ES I am using following configuration, but it is not working. It does not show anything in console and … --- ## [Let CSV columns be fields](https://discuss.elastic.co/t/let-csv-columns-be-fields/188403)
**Author:** [@Adolfo\_Herrera](https://discuss.elastic.co/u/Adolfo_Herrera)\ **Replies:** 4\ **Last updated:** [July 2, 2019, 4:10pm UTC](https://discuss.elastic.co/t/let-csv-columns-be-fields/188403 "2019-07-02T16:10:28Z")
Hello I'm trying to get the names of the columns from my csv file to become field names. However, they all seem to be staying in the message field input { beats { port =\> 5044 } } filter { csv { … --- ## [How to grok a multiline message?](https://discuss.elastic.co/t/how-to-grok-a-multiline-message/188351)
**Author:** [@TimWard](https://discuss.elastic.co/u/TimWard)\ **Replies:** 7\ **Last updated:** [July 2, 2019, 4:01pm UTC](https://discuss.elastic.co/t/how-to-grok-a-multiline-message/188351 "2019-07-02T16:01:02Z")
(I've come across a number of similar questions, but either they don't match what I'm doing or they don't have answers.) I've got Java log lines, to the end of which might be appended " | {\}". Parsing the mai… --- ## [Use filebeat file output as logstash input](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904)
**Author:** [@maddin2016](https://discuss.elastic.co/u/maddin2016)\ **Replies:** 5\ **Last updated:** [July 2, 2019, 3:57pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904 "2019-07-02T15:57:35Z")
I want to use the file output from filebeat as file input in logstash because there is no direct connection between the both systems. Here is my logstash config input { file { path =\> "/home/martin/logs/filebeat\*"… --- ## [Change pipeline worker node to provide sequence of logs in kibana](https://discuss.elastic.co/t/change-pipeline-worker-node-to-provide-sequence-of-logs-in-kibana/188536)
**Author:** [@mrashid](https://discuss.elastic.co/u/mrashid)\ **Replies:** 3\ **Last updated:** [July 2, 2019, 3:52pm UTC](https://discuss.elastic.co/t/change-pipeline-worker-node-to-provide-sequence-of-logs-in-kibana/188536 "2019-07-02T15:52:09Z")
I tried to configure the pipeline worker node by changing from the default of 1. This is to provide solution to the issue where logs are not in sequence in kibana. I have followed suggestion from the below already logge… --- ## [Logstash 7.1.1 not writing last\_run in last\_run\_metadata\_path](https://discuss.elastic.co/t/logstash-7-1-1-not-writing-last-run-in-last-run-metadata-path/188496)
**Author:** [@Giovanni\_Gasparri](https://discuss.elastic.co/u/Giovanni_Gasparri)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 3:23pm UTC](https://discuss.elastic.co/t/logstash-7-1-1-not-writing-last-run-in-last-run-metadata-path/188496 "2019-07-02T15:23:08Z")
Hello guys! I'm experiencing a problem with logstash 7.1.1 running as a deamon on debian with multiple pipelines. Despite the fact that record\_last\_run is true and last\_run\_metadata\_path is specified, no file is saved. … --- ## [Not able to use since\_db properly](https://discuss.elastic.co/t/not-able-to-use-since-db-properly/188530)
**Author:** [@Pranay\_Garg](https://discuss.elastic.co/u/Pranay_Garg)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 3:13pm UTC](https://discuss.elastic.co/t/not-able-to-use-since-db-properly/188530 "2019-07-02T15:13:02Z")
I am trying to insert data in elasticsearch using logstash but when i update a file and again process it the events get duplicated when i only want the newly added entries to get indexed, for this i am trying to use sinc… --- ## [Grok pattern for access.log file](https://discuss.elastic.co/t/grok-pattern-for-access-log-file/188532)
**Author:** [@AkilaSachin](https://discuss.elastic.co/u/AkilaSachin)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 3:06pm UTC](https://discuss.elastic.co/t/grok-pattern-for-access-log-file/188532 "2019-07-02T15:06:11Z")
I need a grok pattern for below type access.log. 172.26.76.68 - - \[02/Jul/2019:19:23:16 +0530\] POST /abc/apiname HTTP/1.1 200 2049 7 172.26.76.68 - - \[02/Jul/2019:19:23:15 +0530\] GET /abc/apiname/id/123456789/idType/NI… --- ## [Logstash pluguin to march multiline logs?](https://discuss.elastic.co/t/logstash-pluguin-to-march-multiline-logs/186399)
**Author:** [@gerard.ramos](https://discuss.elastic.co/u/gerard.ramos)\ **Replies:** 9\ **Last updated:** [July 2, 2019, 2:44pm UTC](https://discuss.elastic.co/t/logstash-pluguin-to-march-multiline-logs/186399 "2019-07-02T14:44:01Z")
Hello everyone, I was trying to send logs to Elasticsearch, this logs are in a file and they are multiline like this: ERROR 2019-06-18 05:00:15,919 \[pool-1-thread-147\] somelongtexthere moretext moretext some text some… --- ## [Logstash writes SQL queries instead of logs](https://discuss.elastic.co/t/logstash-writes-sql-queries-instead-of-logs/188515)
**Author:** [@MaxON](https://discuss.elastic.co/u/MaxON)\ **Replies:** 3\ **Last updated:** [July 2, 2019, 2:11pm UTC](https://discuss.elastic.co/t/logstash-writes-sql-queries-instead-of-logs/188515 "2019-07-02T14:11:19Z")
Hi everyone! Not long ago Logstash began writting very strange logs: \[2019-07-02T13:25:01,101\]\[INFO \]\[logstash.inputs.jdbc \] (0.891000s) select id, FORMAT(datetimestamp,'yyyy-MM-ddTHH:mm:sszzz') as datetimestamp, ev… --- ## [Problem Parsing Date field](https://discuss.elastic.co/t/problem-parsing-date-field/188427)
**Author:** [@Pranay\_Garg](https://discuss.elastic.co/u/Pranay_Garg)\ **Replies:** 4\ **Last updated:** [July 2, 2019, 2:10pm UTC](https://discuss.elastic.co/t/problem-parsing-date-field/188427 "2019-07-02T14:10:19Z")
I am trying to match date format Mon Jul 04 22:48:23 2019 using date filter in logstash but it is showing as string in kibana. Can Someone point out the mistake. Thanks date { match =\> \["selling\_date" , "EEE MMM d HH… --- ## [Storage Size PROBLEM while Insert complex nested json documents from Postgres to Elasticsearch via Logstash](https://discuss.elastic.co/t/storage-size-problem-while-insert-complex-nested-json-documents-from-postgres-to-elasticsearch-via-logstash/188318)
**Author:** [@MARCO\_RAMBALDI](https://discuss.elastic.co/u/MARCO_RAMBALDI)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 1:26pm UTC](https://discuss.elastic.co/t/storage-size-problem-while-insert-complex-nested-json-documents-from-postgres-to-elasticsearch-via-logstash/188318 "2019-07-02T13:26:08Z")
Hi, I configured a pipepline to transfer complex nested json documents from postgres to elasticsearch via logstash. This is the configuration file: file.conf input { jdbc { jdbc\_connection\_string =\> "jdbc:postgre… --- ## [Logstash Helm Chart](https://discuss.elastic.co/t/logstash-helm-chart/188521)
**Author:** [@georgec](https://discuss.elastic.co/u/georgec)\ **Replies:** 0\ **Last updated:** [July 2, 2019, 1:25pm UTC](https://discuss.elastic.co/t/logstash-helm-chart/188521 "2019-07-02T13:25:29Z")
Hey all, Does anyone (particularly anyone who works at Elastic) know if the logstash charts will be integrated into the new Elastic maintained setup (https://github.com/elastic/helm-charts) ? There are a few mods I was… --- ## [How to know that logstash has finished processing all its input](https://discuss.elastic.co/t/how-to-know-that-logstash-has-finished-processing-all-its-input/187603)
**Author:** [@Stuart\_Moore](https://discuss.elastic.co/u/Stuart_Moore)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 1:14pm UTC](https://discuss.elastic.co/t/how-to-know-that-logstash-has-finished-processing-all-its-input/187603 "2019-07-02T13:14:33Z")
I have a relatively complicated logstash pipeline setup, with some pipelines feeding into others, splitting events, making http calls to external services, and sometimes feeding an event back into the pipeline it came fr… --- ## [Error logging: is it possible to customise mapper\_parsing\_exception logs?](https://discuss.elastic.co/t/error-logging-is-it-possible-to-customise-mapper-parsing-exception-logs/188514)
**Author:** [@rozling](https://discuss.elastic.co/u/rozling)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 1:11pm UTC](https://discuss.elastic.co/t/error-logging-is-it-possible-to-customise-mapper-parsing-exception-logs/188514 "2019-07-02T13:11:24Z")
I have a central index which many services write logs to. However when I get unexpected data (e.g. a date instead of a boolean) the log looks like this: \[2019-07-02T11:36:32,836\]\[WARN \]\[logstash.outputs.elasticsearch\] … --- ## [Logs not in sequence as in console output](https://discuss.elastic.co/t/logs-not-in-sequence-as-in-console-output/188437)
**Author:** [@mrashid](https://discuss.elastic.co/u/mrashid)\ **Replies:** 6\ **Last updated:** [July 2, 2019, 1:08pm UTC](https://discuss.elastic.co/t/logs-not-in-sequence-as-in-console-output/188437 "2019-07-02T13:08:49Z")
I am sending logs from Jenkins to ELK. Because of the jade timestamp and not having enough precision to nano or milli seconds , I have introduced a sequence number against every line of log that enters logstash . Even … --- ## [Send multiple fields with the syslog output](https://discuss.elastic.co/t/send-multiple-fields-with-the-syslog-output/188463)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 12:49pm UTC](https://discuss.elastic.co/t/send-multiple-fields-with-the-syslog-output/188463 "2019-07-02T12:49:48Z")
Hi, We need to send two fields from a document to a remote TCP destination. We tried to use the syslog output plugin and have not managed to solve this. We can succesfully send one field by configuring the output like … --- ## [Not getting logs entries on the kibana - after Xpack Configuration | ELK v7.1](https://discuss.elastic.co/t/not-getting-logs-entries-on-the-kibana-after-xpack-configuration-elk-v7-1/188485)
**Author:** [@Ranjith\_SG1](https://discuss.elastic.co/u/Ranjith_SG1)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 11:06am UTC](https://discuss.elastic.co/t/not-getting-logs-entries-on-the-kibana-after-xpack-configuration-elk-v7-1/188485 "2019-07-02T11:06:17Z")
We are not getting log entries in the - kibana after configuring xpack in version 7.1 . below is the logsatsh conf file . Let me know is anything wrong in the configuration . - User -elastic( role - super user) logstash… --- ## [How to parse logs to nested json field then store in Elasticsearch?](https://discuss.elastic.co/t/how-to-parse-logs-to-nested-json-field-then-store-in-elasticsearch/188477)
**Author:** [@LinJIngYun](https://discuss.elastic.co/u/LinJIngYun)\ **Replies:** 0\ **Last updated:** [July 2, 2019, 10:28am UTC](https://discuss.elastic.co/t/how-to-parse-logs-to-nested-json-field-then-store-in-elasticsearch/188477 "2019-07-02T10:28:51Z")
I want to parse AWS Elastic Loadbalancer Logs in to pattern that I can query for find the response time of each request does it fail or not, does request URI have fail from 404 or 500 ? or it it success from 200 somewhat… --- ## [Windows event fields parsing](https://discuss.elastic.co/t/windows-event-fields-parsing/187858)
**Author:** [@pinguin](https://discuss.elastic.co/u/pinguin)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 8:39am UTC](https://discuss.elastic.co/t/windows-event-fields-parsing/187858 "2019-07-02T08:39:44Z")
hello, we are trying to send windows event logs using winlogbeat 7.XX version , we are creating severals of dashboards based on the correlation rules of event built on top of event ,sysmon and security logs ex : event.i… --- ## [Dissecting google logs textPayLoad](https://discuss.elastic.co/t/dissecting-google-logs-textpayload/188300)
**Author:** [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Replies:** 22\ **Last updated:** [July 2, 2019, 6:22am UTC](https://discuss.elastic.co/t/dissecting-google-logs-textpayload/188300 "2019-07-02T06:22:42Z")
Hi here is my logstash config for the dissect filter { dissect { mapping =\> { "textPayload" =\> "%{something1} \[%{something2} %{+something2}\] %{something3} %{something4} %{something5} %{something6} %{… --- ## [Jenkins ELK Integration without using logstash plugin](https://discuss.elastic.co/t/jenkins-elk-integration-without-using-logstash-plugin/188409)
**Author:** [@mrashid](https://discuss.elastic.co/u/mrashid)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 4:56am UTC](https://discuss.elastic.co/t/jenkins-elk-integration-without-using-logstash-plugin/188409 "2019-07-02T04:56:31Z")
I am using file-beat to ship logs from Jenkins job host location by configuring filebeat.yml as follows : /builds/v2018/jenkins/jobs/ELK\_TEST/builds/\*/log The problem is that sometimes i get a source field in Kibana wh… --- ## [Can you do a "pull" --FROM-- logstash?](https://discuss.elastic.co/t/can-you-do-a-pull-from-logstash/188404)
**Author:** [@preddy75](https://discuss.elastic.co/u/preddy75)\ **Replies:** 1\ **Last updated:** [July 2, 2019, 3:38am UTC](https://discuss.elastic.co/t/can-you-do-a-pull-from-logstash/188404 "2019-07-02T03:38:21Z")
Key issue is the pull. Is there an output plugin that allows an application to PULL logs from logstash? Looks like all plugins are push from what I have seen so far. --- ## [Logstash date filter parsing date](https://discuss.elastic.co/t/logstash-date-filter-parsing-date/187176)
**Author:** [@cbeprem](https://discuss.elastic.co/u/cbeprem)\ **Replies:** 4\ **Last updated:** [July 2, 2019, 3:35am UTC](https://discuss.elastic.co/t/logstash-date-filter-parsing-date/187176 "2019-07-02T03:35:06Z")
I am using the date filter but it is not working . Kindly help me out. Below is the config file. I am using a csv in s3 to ingest data via logstash. I am using ELK 7 filter { csv { separator =\> "," columns =\> \[ "D… --- ## [Aggregate exception occurred](https://discuss.elastic.co/t/aggregate-exception-occurred/188400)
**Author:** [@rohith1570](https://discuss.elastic.co/u/rohith1570)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 2:49am UTC](https://discuss.elastic.co/t/aggregate-exception-occurred/188400 "2019-07-02T02:49:59Z")
Hi, I'm trying to generate a total file size for a day based on user Input Filter Conf if \[sum\_daily\_filesize\] == "daily" { if \[user.id\] != "" and \[user.id\] != 'null' { aggregate { task\_id =\> "%{user.id}\_%{+d}\_%{+MM… --- ## [Calculations using ruby filter in logstash is giving ruby exception error](https://discuss.elastic.co/t/calculations-using-ruby-filter-in-logstash-is-giving-ruby-exception-error/188423)
**Author:** [@Pranay\_Garg](https://discuss.elastic.co/u/Pranay_Garg)\ **Replies:** 2\ **Last updated:** [July 2, 2019, 1:10am UTC](https://discuss.elastic.co/t/calculations-using-ruby-filter-in-logstash-is-giving-ruby-exception-error/188423 "2019-07-02T01:10:11Z")
I want to store multiplication of two fields in another field but when i run it gives ruby exception error. I am new to logstash and any help is appreciated. Thanks in advance for help. mutate { add\_field =\> { "tota… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=516) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=518)