# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=223 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 224 --- ## [Logstash read file in what order?](https://discuss.elastic.co/t/logstash-read-file-in-what-order/274549)
**Author:** [@theirfan](https://discuss.elastic.co/u/theirfan)\ **Replies:** 21\ **Last updated:** [June 3, 2021, 2:52am UTC](https://discuss.elastic.co/t/logstash-read-file-in-what-order/274549 "2021-06-03T02:52:37Z")
Hi All, Would like to know how logstash "input" reads files from a directory. I have files which are created based on size limit and each new log creation is appended with the creation date in it's name. Ex: file-010… --- ## [Logstash, syslog, ECS and Kibana Logs / SIEM](https://discuss.elastic.co/t/logstash-syslog-ecs-and-kibana-logs-siem/274789)
**Author:** [@egli](https://discuss.elastic.co/u/egli)\ **Replies:** 4\ **Last updated:** [June 2, 2021, 11:08pm UTC](https://discuss.elastic.co/t/logstash-syslog-ecs-and-kibana-logs-siem/274789 "2021-06-02T23:08:42Z")
Hi, I have logstash working as a central syslog server using syslog\_pri plugin and sending the events to elasticsearch. I would like to ingest those syslog events and adhere to ECS so it is possible to use Kibana Logs o… --- ## [Dateparsefailure error logstash](https://discuss.elastic.co/t/dateparsefailure-error-logstash/274806)
**Author:** [@byakko1234](https://discuss.elastic.co/u/byakko1234)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 10:57pm UTC](https://discuss.elastic.co/t/dateparsefailure-error-logstash/274806 "2021-06-02T22:57:30Z")
Hi everyone. I´m trying to use the date filter without any luck. I want to create a new field that is the combination of the %{date} and %{time} in my grok and have this new field to have a date type value. Any help is… --- ## [Sql\_last\_value and implicit time zoned data](https://discuss.elastic.co/t/sql-last-value-and-implicit-time-zoned-data/274797)
**Author:** [@etva](https://discuss.elastic.co/u/etva)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 7:23pm UTC](https://discuss.elastic.co/t/sql-last-value-and-implicit-time-zoned-data/274797 "2021-06-02T19:23:43Z")
Hi all, MySql server is UTC : now() give UTC date . sql\_last\_value is stored in UTC in file ( from MySQL ? ) But data I have to compare to sql\_last \_value are locale date as string. I can't use CONVERT\_TZ with named … --- ## [NEED TO READ ONLY THE NEWEST FILE IN EACH FOLDER](https://discuss.elastic.co/t/need-to-read-only-the-newest-file-in-each-folder/274714)
**Author:** [@Daniel\_Lopez](https://discuss.elastic.co/u/Daniel_Lopez)\ **Replies:** 3\ **Last updated:** [June 2, 2021, 5:51pm UTC](https://discuss.elastic.co/t/need-to-read-only-the-newest-file-in-each-folder/274714 "2021-06-02T17:51:47Z")
Hi guys!! I need to configure logstash to read only the newest file in folders, when I restart always read all files, and I couldn't understand why, files are in NAS, and I mount folders before start reading. This is m… --- ## [Using KV and Grok to parse complex data](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769)
**Author:** [@blackberrySherbet](https://discuss.elastic.co/u/blackberrySherbet)\ **Replies:** 1\ **Last updated:** [June 2, 2021, 3:07pm UTC](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769 "2021-06-02T15:07:37Z")
Hi, My data is structured like this- INFO {"datetime": "2021-06-1 22:13:29.469000", data:{"val1":3.14, "val2": 2.17}} INFO {"datetime": "2021-06-1 21:14:00.469000", data:{"val3":9}} I want to use logstash to process … --- ## [Pipeline not communicate](https://discuss.elastic.co/t/pipeline-not-communicate/274766)
**Author:** [@frank\_rib](https://discuss.elastic.co/u/frank_rib)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 2:27pm UTC](https://discuss.elastic.co/t/pipeline-not-communicate/274766 "2021-06-02T14:27:46Z")
Hello expert, In order to parse openbsd events I chosed to use pipeline to pipleline communications, I have three pipelines named input which point to openbsd\_2\_ecs pipleline and this last point to output pipeline. Ple… --- ## [Different version between Elasticsearch and Logstash](https://discuss.elastic.co/t/different-version-between-elasticsearch-and-logstash/274567)
**Author:** [@111475](https://discuss.elastic.co/u/111475)\ **Replies:** 5\ **Last updated:** [June 2, 2021, 1:51pm UTC](https://discuss.elastic.co/t/different-version-between-elasticsearch-and-logstash/274567 "2021-06-02T13:51:10Z")
Is it okay to run Logstash with version(7.13.0) different from our Elasticsearch version(7.10.2)? --- ## [Configuration file](https://discuss.elastic.co/t/configuration-file/274758)
**Author:** [@Krunal](https://discuss.elastic.co/u/Krunal)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 1:40pm UTC](https://discuss.elastic.co/t/configuration-file/274758 "2021-06-02T13:40:55Z")
Hello Expert, I am having csv file with more than 5 tables in it... i would like extract data from 2 of them. but column names are different and data structure is different..... and one more thing that i would like to a… --- ## [Http\_pollar input is not working with the certificate](https://discuss.elastic.co/t/http-pollar-input-is-not-working-with-the-certificate/274748)
**Author:** [@anjilinga](https://discuss.elastic.co/u/anjilinga)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 12:26pm UTC](https://discuss.elastic.co/t/http-pollar-input-is-not-working-with-the-certificate/274748 "2021-06-02T12:26:43Z")
Hi We are using http\_poller to query elastic in logstash input with the selfsigned certificate. but not receieving any result and not prompting any error in the logstash. tls is enabled at elastic. when i run the same qu… --- ## [How to connect multiple pipelines within the same Logstash instance running inside a docker container](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446)
**Author:** [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Replies:** 5\ **Last updated:** [June 2, 2021, 12:25pm UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446 "2021-06-02T12:25:21Z")
Currently we have a data stream in elasticsearch, lets call it app\_stream in which we are ingesting log messages from an application XMLTransformer through FileBeats(port=5044) installed on the server where the applicati… --- ## [Unable to bind Syslog port](https://discuss.elastic.co/t/unable-to-bind-syslog-port/274728)
**Author:** [@Sputnick](https://discuss.elastic.co/u/Sputnick)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 9:23am UTC](https://discuss.elastic.co/t/unable-to-bind-syslog-port/274728 "2021-06-02T09:23:23Z")
Hello, I am on Ubuntu 20.04 running Logstash as a collector for syslog messages from the rest of my network. I have configured java with elevated privileges but Logstash is unable to bind UDP port 514. This is a simila… --- ## [Logstash not showing generated output](https://discuss.elastic.co/t/logstash-not-showing-generated-output/274677)
**Author:** [@Ashwani\_Shukla](https://discuss.elastic.co/u/Ashwani_Shukla)\ **Replies:** 2\ **Last updated:** [June 2, 2021, 3:50am UTC](https://discuss.elastic.co/t/logstash-not-showing-generated-output/274677 "2021-06-02T03:50:50Z")
The pipeline file name is newstash.conf input{ file{ path =\> \[ "/etc/logstash/sample.txt" \] } } output { stdout{ } } The sample.txt file has some string only. The output is l… --- ## [Correct way to use the mutate rename filter](https://discuss.elastic.co/t/correct-way-to-use-the-mutate-rename-filter/274683)
**Author:** [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Replies:** 4\ **Last updated:** [June 2, 2021, 12:23am UTC](https://discuss.elastic.co/t/correct-way-to-use-the-mutate-rename-filter/274683 "2021-06-02T00:23:25Z")
Hello, this is weird, For the last couple of years I've been using the mutate rename filter in the following way, with one rename option for every field inside the same mutate block. mutate { rename =\> { "field1" =\>… --- ## [Logstash](https://discuss.elastic.co/t/logstash/273611)
**Author:** [@httrack](https://discuss.elastic.co/u/httrack)\ **Replies:** 3\ **Last updated:** [June 1, 2021, 4:05pm UTC](https://discuss.elastic.co/t/logstash/273611 "2021-06-01T16:05:29Z")
Hello Team. I am having problems with a client. How can I activate the filebeat-\* index and the Logstash-\* index to have the syslog in the Discover part of Kibana --- ## [Extract fields from JSON Flie to Elastic using Logstash filters](https://discuss.elastic.co/t/extract-fields-from-json-flie-to-elastic-using-logstash-filters/274610)
**Author:** [@Sirine](https://discuss.elastic.co/u/Sirine)\ **Replies:** 5\ **Last updated:** [June 1, 2021, 3:18pm UTC](https://discuss.elastic.co/t/extract-fields-from-json-flie-to-elastic-using-logstash-filters/274610 "2021-06-01T15:18:50Z")
Hello, I'm trying to extract fields from my JSON file, but I have \_jsonparsefailure" error, I tried many other filters like grok Split or vk but always the same results, logstash doesn't extract the values of the fields… --- ## [Add a field depending on first log](https://discuss.elastic.co/t/add-a-field-depending-on-first-log/274653)
**Author:** [@A\_Chichi](https://discuss.elastic.co/u/A_Chichi)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 2:37pm UTC](https://discuss.elastic.co/t/add-a-field-depending-on-first-log/274653 "2021-06-01T14:37:14Z")
Hello, I would like to add a field in my logstash pipeline. But this field would be a dynamic field. Here is a sample of my file : date\_log;job\_id;service\_id;name;progress;status;duration 2021-06-01 16:16:03;nJzP4g;7… --- ## [Elasticsearch seems not to receive logs from logstash](https://discuss.elastic.co/t/elasticsearch-seems-not-to-receive-logs-from-logstash/274648)
**Author:** [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 2:20pm UTC](https://discuss.elastic.co/t/elasticsearch-seems-not-to-receive-logs-from-logstash/274648 "2021-06-01T14:20:59Z")
Hi, anyone help me sort out why elasticsearch seems not to receive logs from logstash. I am new to ELK. here is my config so far. TIA /etc/elasticsearch/elasticsearch.yml network.host: 0.0.0.0 transport.host: localhos… --- ## [How to use mutate filter plugin in output](https://discuss.elastic.co/t/how-to-use-mutate-filter-plugin-in-output/274578)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 1\ **Last updated:** [June 1, 2021, 1:40pm UTC](https://discuss.elastic.co/t/how-to-use-mutate-filter-plugin-in-output/274578 "2021-06-01T13:40:36Z")
I have a logstash configuration. With that configuration logstash do some operation in filter and send outputs. Everything works well. I want one more elasticsearch output in same configuration file. I mean after parsin… --- ## [Deliberately fail event in logstash to requeue it](https://discuss.elastic.co/t/deliberately-fail-event-in-logstash-to-requeue-it/274635)
**Author:** [@tumbledwyer](https://discuss.elastic.co/u/tumbledwyer)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 1:19pm UTC](https://discuss.elastic.co/t/deliberately-fail-event-in-logstash-to-requeue-it/274635 "2021-06-01T13:19:09Z")
Hi Is there a way to fail an event so that it can be requeued for processing later? Let's take this conf for example: input { pipeline { address =\> "previous-pipe" } } filter { elasticsearch { hosts =\> \[… --- ## [Logstash one to one mapping of two lines](https://discuss.elastic.co/t/logstash-one-to-one-mapping-of-two-lines/274634)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 1:17pm UTC](https://discuss.elastic.co/t/logstash-one-to-one-mapping-of-two-lines/274634 "2021-06-01T13:17:15Z")
hi guys, how to do one to one mapping of below two lines - number of comma separated values in message and header fields can be dynamic but number of items in both fields will be same "message" =\> "error\_count,10,20,3… --- ## [Problem running logstash:7.13.0 on docker](https://discuss.elastic.co/t/problem-running-logstash-7-13-0-on-docker/274627)
**Author:** [@Jonas\_Forte](https://discuss.elastic.co/u/Jonas_Forte)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 12:05pm UTC](https://discuss.elastic.co/t/problem-running-logstash-7-13-0-on-docker/274627 "2021-06-01T12:05:34Z")
Hello everyone, I am having an error when running the logstash on the docker. I'm trying to run logstash on the docker with the following command: command: docker run --rm -it -v /usr/share/logstash/pipeline/logstash.… --- ## [Could not find the logstash.yml](https://discuss.elastic.co/t/could-not-find-the-logstash-yml/274614)
**Author:** [@Kanishk\_Madan](https://discuss.elastic.co/u/Kanishk_Madan)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 10:35am UTC](https://discuss.elastic.co/t/could-not-find-the-logstash-yml/274614 "2021-06-01T10:35:53Z")
I have a problem when i run a file using this command :sudo /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.conf It shows:- Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/… --- ## [Identify event fom JDBC input](https://discuss.elastic.co/t/identify-event-fom-jdbc-input/274607)
**Author:** [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 9:36am UTC](https://discuss.elastic.co/t/identify-event-fom-jdbc-input/274607 "2021-06-01T09:36:08Z")
Hi, I'm currently having issues to identify my JDBC input from my beat input and all others. I'm quite new to using jdbc as input in logstash. I actually have a workflow where data from my beats go through a bunch of f… --- ## [Change value of @timestamp](https://discuss.elastic.co/t/change-value-of-timestamp/274343)
**Author:** [@mar-ro](https://discuss.elastic.co/u/mar-ro)\ **Replies:** 2\ **Last updated:** [June 1, 2021, 9:33am UTC](https://discuss.elastic.co/t/change-value-of-timestamp/274343 "2021-06-01T09:33:29Z")
Hello! How can I remplace the value of @timestamp with the value that I get with a regular expresion? I need change the value of @timestamp because when the data is indexed in elasticsearch it shows in this field the v… --- ## [Filter and return only specific fields using elasticsearch plugin with logstash](https://discuss.elastic.co/t/filter-and-return-only-specific-fields-using-elasticsearch-plugin-with-logstash/274599)
**Author:** [@Nico3](https://discuss.elastic.co/u/Nico3)\ **Replies:** 0\ **Last updated:** [June 1, 2021, 8:51am UTC](https://discuss.elastic.co/t/filter-and-return-only-specific-fields-using-elasticsearch-plugin-with-logstash/274599 "2021-06-01T08:51:14Z")
Hello, I'm trying to create a pipeline with logstash in order to "extract" a specific metric from Elasticsearch (window\_cpu\_time\_total). Here is my pipeline: input { elasticsearch { hosts =\> \["http://localhos… --- ## [Logstash stores field as multifield](https://discuss.elastic.co/t/logstash-stores-field-as-multifield/274331)
**Author:** [@nberens](https://discuss.elastic.co/u/nberens)\ **Replies:** 2\ **Last updated:** [May 30, 2021, 2:26am UTC](https://discuss.elastic.co/t/logstash-stores-field-as-multifield/274331 "2021-05-30T02:26:04Z")
Hello, i am currently migrating from an old elatic 6.8 cluster to a new 7.13. The Logstash pipeline is the same: input { kafka { bootstrap\_servers =\> '{{ logstash\_kafka\_bootstrap }}' … --- ## [Getting Invalid FieldReference error because of non-indexed property name in document](https://discuss.elastic.co/t/getting-invalid-fieldreference-error-because-of-non-indexed-property-name-in-document/274205)
**Author:** [@rs0000](https://discuss.elastic.co/u/rs0000)\ **Replies:** 2\ **Last updated:** [June 1, 2021, 7:33am UTC](https://discuss.elastic.co/t/getting-invalid-fieldreference-error-because-of-non-indexed-property-name-in-document/274205 "2021-06-01T07:33:18Z")
Hi, My Logstash pipeline keeps failing during scroll because of an error. There are multiple documents causing that issue. Those are referral urls that sometimes look weird. The property itself is not indexed in elastic… --- ## [Disable logstash logger for inputs.jdbc](https://discuss.elastic.co/t/disable-logstash-logger-for-inputs-jdbc/274526)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 4\ **Last updated:** [June 1, 2021, 7:15am UTC](https://discuss.elastic.co/t/disable-logstash-logger-for-inputs-jdbc/274526 "2021-06-01T07:15:08Z")
Hi all, I am using Logstash 7.12.1 to update an elasticsearch 7.12.1 index from an oracle database. I have no problem establishing communication with the DB using the below input jdbc declaration: input { jdbc { … --- ## [How I can run jdbc\_streaming filter just one time on multiple jdbc plugin row results](https://discuss.elastic.co/t/how-i-can-run-jdbc-streaming-filter-just-one-time-on-multiple-jdbc-plugin-row-results/274477)
**Author:** [@zeraf29](https://discuss.elastic.co/u/zeraf29)\ **Replies:** 2\ **Last updated:** [June 1, 2021, 6:17am UTC](https://discuss.elastic.co/t/how-i-can-run-jdbc-streaming-filter-just-one-time-on-multiple-jdbc-plugin-row-results/274477 "2021-06-01T06:17:27Z")
Hi. I want to run jdbc\_streaming filter plugin just one time on multiple jdbc plugin row results. For example, I set jdbc input plugin in pipeline, and It return 10 table rows. input{ Jdbc{ #jdbc settings … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=222) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=224)