# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=240 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 241 --- ## [Zeek ingest using logstash on elastic cloud](https://discuss.elastic.co/t/zeek-ingest-using-logstash-on-elastic-cloud/269522)
**Author:** [@kajira](https://discuss.elastic.co/u/kajira)\ **Replies:** 8\ **Last updated:** [April 8, 2021, 10:55pm UTC](https://discuss.elastic.co/t/zeek-ingest-using-logstash-on-elastic-cloud/269522 "2021-04-08T22:55:16Z")
Hi, I am trying the elastic cloud for the first time using 7.12 stack. I would like to send zeek logs to my elastic cloud deployment and the default method recommended is to use filebeats. I would like to stream the log… --- ## [Duplicate data Logstash](https://discuss.elastic.co/t/duplicate-data-logstash/269635)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 4\ **Last updated:** [April 8, 2021, 9:13pm UTC](https://discuss.elastic.co/t/duplicate-data-logstash/269635 "2021-04-08T21:13:53Z")
Hi, guy's i need your help, please with this... I have a problem with the indexing of data from logstash to elastic, I have two pipelines and when I run the service I duplicate the data from one pipeline to the other, … --- ## [Questions re Sniffing in Logstash elasticsearch output](https://discuss.elastic.co/t/questions-re-sniffing-in-logstash-elasticsearch-output/269631)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 3\ **Last updated:** [April 8, 2021, 6:59pm UTC](https://discuss.elastic.co/t/questions-re-sniffing-in-logstash-elasticsearch-output/269631 "2021-04-08T18:59:47Z")
I'm testing Logstash sniffing in our environment for the Elasticsearch output. So far, it appears to resolve a few issues I've had, but I've got a few questions: sniffing =\> true in my elasticsearch output appears to … --- ## [Logstash Parser error - tried to parse field as object, but found a concrete value](https://discuss.elastic.co/t/logstash-parser-error-tried-to-parse-field-as-object-but-found-a-concrete-value/269140)
**Author:** [@Automation\_Scripts](https://discuss.elastic.co/u/Automation_Scripts)\ **Replies:** 13\ **Last updated:** [April 8, 2021, 6:42pm UTC](https://discuss.elastic.co/t/logstash-parser-error-tried-to-parse-field-as-object-but-found-a-concrete-value/269140 "2021-04-08T18:42:57Z")
Hi, I am trying to parse a log but i have this error. \[source\] tried to parse field \[source\] as object, but found a concrete value if "string" in \[tags\] { grok { match =\> \[ "message", "(?\ **Author:** [@gneves](https://discuss.elastic.co/u/gneves)\ **Replies:** 1\ **Last updated:** [April 8, 2021, 5:51pm UTC](https://discuss.elastic.co/t/epoch-to-readable-timestamp/269623 "2021-04-08T17:51:52Z") Hello guys. Hope you are doing well. Im having a little problem trying to convert an event time in epoch to a readable timestamp, per example in UTC. So, the log that i have is something like this: event\_time=1613523… --- ## [XML parser, alter field](https://discuss.elastic.co/t/xml-parser-alter-field/269415)
**Author:** [@Soren\_vdc](https://discuss.elastic.co/u/Soren_vdc)\ **Replies:** 3\ **Last updated:** [April 8, 2021, 3:55pm UTC](https://discuss.elastic.co/t/xml-parser-alter-field/269415 "2021-04-08T15:55:29Z")
Hi, I have this setup for XML parser: xml { source =\> "message" target =\> "xml" add\_field =\> { Audit\_Type =\> "%{\[xml\]\[Audit\_Type\]}" Ext\_Name … --- ## [Logstash](https://discuss.elastic.co/t/logstash/268923)
**Author:** [@wpr](https://discuss.elastic.co/u/wpr)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 2:12pm UTC](https://discuss.elastic.co/t/logstash/268923 "2021-04-08T14:12:25Z")
Hi, I am using Logstash to migrate data from ES 5.6 to ES 7.9. I am aware of the changes between those versions. I have the following mapping for a document in 5.6: "transaction\_information": { … --- ## [I want to input row of csv file](https://discuss.elastic.co/t/i-want-to-input-row-of-csv-file/269538)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 12:12pm UTC](https://discuss.elastic.co/t/i-want-to-input-row-of-csv-file/269538 "2021-04-08T12:12:44Z")
I want to input per one row and csv file haven't column name. Please teach me how to set for logstash. Data flow is filebeat → logstash → elasticsearch. I attached sample image of csv file. --- ## [\[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: undefined method \`\[\]' for #\](https://discuss.elastic.co/t/error-logstash-filters-ruby-ruby-exception-occurred-undefined-method-for-logstash-0x9a18b32/269301)
**Author:** [@rumala](https://discuss.elastic.co/u/rumala)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 9:57am UTC](https://discuss.elastic.co/t/error-logstash-filters-ruby-ruby-exception-occurred-undefined-method-for-logstash-0x9a18b32/269301 "2021-04-08T09:57:12Z")
Hello experts, I'm still learning about ELK. last week i set up my ELK lab. And I got error in my Logstash. I'm using logstash 7.5.1. I would like to parse through all fields of my Logstash and convert field starting wi… --- ## [Security\_exception in logstash log: indices:admin/auto\_create is unauthorized for user](https://discuss.elastic.co/t/security-exception-in-logstash-log-indices-admin-auto-create-is-unauthorized-for-user/269523)
**Author:** [@cotjoey](https://discuss.elastic.co/u/cotjoey)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 9:56am UTC](https://discuss.elastic.co/t/security-exception-in-logstash-log-indices-admin-auto-create-is-unauthorized-for-user/269523 "2021-04-08T09:56:08Z")
Hello, I configured fluentd to forward logs to Logstash (7.10) and the Elasticsearch output throws the following error: \[2021-04-07T17:17:11,390\]\[INFO \]\[logstash.outputs.elasticsearch\]\[main\]\[2662b9682e97be5de9346f00d19… --- ## [Logstash unable to connect ssl enabled elasticsearch](https://discuss.elastic.co/t/logstash-unable-to-connect-ssl-enabled-elasticsearch/269570)
**Author:** [@abhishek\_s1](https://discuss.elastic.co/u/abhishek_s1)\ **Replies:** 0\ **Last updated:** [April 8, 2021, 9:37am UTC](https://discuss.elastic.co/t/logstash-unable-to-connect-ssl-enabled-elasticsearch/269570 "2021-04-08T09:37:23Z")
I was using logstash to ingest data into elasticearch. But now after enabling ssl to elasticsearch (using this) for using alerts & detections, Logstash is unable to connect elasticsearch. Browser or curl command works a… --- ## [Improve logstash performance](https://discuss.elastic.co/t/improve-logstash-performance/269569)
**Author:** [@Paresh\_Vaniya](https://discuss.elastic.co/u/Paresh_Vaniya)\ **Replies:** 0\ **Last updated:** [April 8, 2021, 9:34am UTC](https://discuss.elastic.co/t/improve-logstash-performance/269569 "2021-04-08T09:34:33Z")
Hi All, We are using logstash for full data load from Azure SQL to Elasticsearch & that takes too much time. Currently its taking 12 hours to insert 16 million records(using Default configuration of logstash). I have c… --- ## [Create kibana space in output elasticsearch section in logstash](https://discuss.elastic.co/t/create-kibana-space-in-output-elasticsearch-section-in-logstash/269351)
**Author:** [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 8:48am UTC](https://discuss.elastic.co/t/create-kibana-space-in-output-elasticsearch-section-in-logstash/269351 "2021-04-08T08:48:57Z")
HI all I am using elastic cloud 7.12 with filebeat and logstash through helm. Now, filebeat send all the logs to logstash then logstash does some filters then output to elasticsearch. In the pipeline in output I have … --- ## [Trying to setup logstash with a github repo](https://discuss.elastic.co/t/trying-to-setup-logstash-with-a-github-repo/269541)
**Author:** [@anon65617794](https://discuss.elastic.co/u/anon65617794)\ **Replies:** 5\ **Last updated:** [April 8, 2021, 3:37am UTC](https://discuss.elastic.co/t/trying-to-setup-logstash-with-a-github-repo/269541 "2021-04-08T03:37:44Z")
I'm using this github repo \*as start point. Started elasticsearch,kibana from a docker-compose file and currently trying to add logstash -f logstash.conf to work in command prompt but reaches a error message. = GitHub … --- ## [Can Logstash read multiple text files , store config data available in first lines and add it to all other lines](https://discuss.elastic.co/t/can-logstash-read-multiple-text-files-store-config-data-available-in-first-lines-and-add-it-to-all-other-lines/267830)
**Author:** [@Svett](https://discuss.elastic.co/u/Svett)\ **Replies:** 2\ **Last updated:** [April 8, 2021, 12:00am UTC](https://discuss.elastic.co/t/can-logstash-read-multiple-text-files-store-config-data-available-in-first-lines-and-add-it-to-all-other-lines/267830 "2021-04-08T00:00:44Z")
Hi, I have multiple log files that I want to load in Kibana. Each file has at the beginning few lines with configuration information, that has to be added to each log line from this file. These lines look like that: … --- ## [Elastic - Logstash: Copy data from one field to another field within same index - both the fields are nested](https://discuss.elastic.co/t/elastic-logstash-copy-data-from-one-field-to-another-field-within-same-index-both-the-fields-are-nested/269529)
**Author:** [@Siri2](https://discuss.elastic.co/u/Siri2)\ **Replies:** 0\ **Last updated:** [April 7, 2021, 9:49pm UTC](https://discuss.elastic.co/t/elastic-logstash-copy-data-from-one-field-to-another-field-within-same-index-both-the-fields-are-nested/269529 "2021-04-07T21:49:10Z")
Hi, I am trying to copy data from one field to another field within the same index using filter -\> mutate -\> copy I want to mention that both the fields are nested. When I run the script it doesn't show me if it copied… --- ## [Remove duplicates based on the timestamp and another field](https://discuss.elastic.co/t/remove-duplicates-based-on-the-timestamp-and-another-field/269519)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [April 7, 2021, 8:05pm UTC](https://discuss.elastic.co/t/remove-duplicates-based-on-the-timestamp-and-another-field/269519 "2021-04-07T20:05:22Z")
What method do you recommend to remove duplicates in an index, based on @timestamp and a field? somthing like this: if @timestamp and interface\_name are equal, delete one of the document --- ## [\[ERROR\]\[logstash.agent\] "Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"input\\", \\"filter\\", \\"output\\" at line 1, column 1 (byte 1)"](https://discuss.elastic.co/t/error-logstash-agent-expected-one-of-t-r-n-input-filter-output-at-line-1-column-1-byte-1/269515)
**Author:** [@Eng.Lucy](https://discuss.elastic.co/u/Eng.Lucy)\ **Replies:** 1\ **Last updated:** [April 7, 2021, 8:04pm UTC](https://discuss.elastic.co/t/error-logstash-agent-expected-one-of-t-r-n-input-filter-output-at-line-1-column-1-byte-1/269515 "2021-04-07T20:04:35Z")
I run logstash in a Docker container and occurs a error: \`\`\[ERROR\]\[logstash.agent\] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :messa… --- ## [Check if string is in field](https://discuss.elastic.co/t/check-if-string-is-in-field/269479)
**Author:** [@Automation\_Scripts](https://discuss.elastic.co/u/Automation_Scripts)\ **Replies:** 9\ **Last updated:** [April 7, 2021, 7:20pm UTC](https://discuss.elastic.co/t/check-if-string-is-in-field/269479 "2021-04-07T19:20:18Z")
Hi, I need some help checking if a string is contained within the value of a field. my json has a key "log.file.path.keyword":\["myfolder/mypath/mylog.log" I tried to parse it with filter{ if "mylog.log" in \[log.file… --- ## [Custom match pattern based on log message type](https://discuss.elastic.co/t/custom-match-pattern-based-on-log-message-type/269423)
**Author:** [@sergle](https://discuss.elastic.co/u/sergle)\ **Replies:** 1\ **Last updated:** [April 7, 2021, 4:53pm UTC](https://discuss.elastic.co/t/custom-match-pattern-based-on-log-message-type/269423 "2021-04-07T16:53:46Z")
Hey there, Would very much appreciate some input on something I am trying to achieve. The setup: NLog -\> (UDP) -\> Logstash (ELK) The NLog is configured to send different message structures to Logstash, all in the for… --- ## [Safely remove tcp input plugin's "host" and "port" fields](https://discuss.elastic.co/t/safely-remove-tcp-input-plugins-host-and-port-fields/269492)
**Author:** [@Supermathie](https://discuss.elastic.co/u/Supermathie)\ **Replies:** 3\ **Last updated:** [April 7, 2021, 4:44pm UTC](https://discuss.elastic.co/t/safely-remove-tcp-input-plugins-host-and-port-fields/269492 "2021-04-07T16:44:56Z")
The tcp input plugin adds host and port fields to received events. Ideally we don't want to add these but since this isn't documented, there doesn't seem to be a way to remove them. What is the best filter to safely re… --- ## [Troubles with German Umlaute ä ö ü ß](https://discuss.elastic.co/t/troubles-with-german-umlaute-a-o-u-ss/269428)
**Author:** [@feva](https://discuss.elastic.co/u/feva)\ **Replies:** 1\ **Last updated:** [April 7, 2021, 4:26pm UTC](https://discuss.elastic.co/t/troubles-with-german-umlaute-a-o-u-ss/269428 "2021-04-07T16:26:40Z")
I have problems with German Umlaute ä ü ö ß. I send a json logmessage from IBMi (AS400) to logstash encoded in ISO8859-1. It is not possible to send them in UTF-8. So I tried to set input { http { host =\> "0.0.0.… --- ## [Elastic Lab 1.4](https://discuss.elastic.co/t/elastic-lab-1-4/269131)
**Author:** [@Troy\_Kelley](https://discuss.elastic.co/u/Troy_Kelley)\ **Replies:** 3\ **Last updated:** [April 7, 2021, 2:59pm UTC](https://discuss.elastic.co/t/elastic-lab-1-4/269131 "2021-04-07T14:59:20Z")
Hello, I am working through the elastic lab 1.4. I am not using virtual lab, I am running mine locally. For some reason I did not have the blogs\_sql.conf when I installed Elastic. So, I found an example on the web, cr… --- ## [Logstash HTTP output plugin not reading metadata fields correctly for user/password fields, what am I doing wrong](https://discuss.elastic.co/t/logstash-http-output-plugin-not-reading-metadata-fields-correctly-for-user-password-fields-what-am-i-doing-wrong/269482)
**Author:** [@R.L](https://discuss.elastic.co/u/R.L)\ **Replies:** 1\ **Last updated:** [April 7, 2021, 2:32pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-not-reading-metadata-fields-correctly-for-user-password-fields-what-am-i-doing-wrong/269482 "2021-04-07T14:32:37Z")
This is my config: filter { prune { blacklist\_names =\> \[ "timestamp", "path", "size", "@version", "host" \] } uuid { target =\> "\[@metadata\]\[snapshot\]\[id\]" overwrit… --- ## [Logstash-oss yum packages missing?](https://discuss.elastic.co/t/logstash-oss-yum-packages-missing/269258)
**Author:** [@mentzerk](https://discuss.elastic.co/u/mentzerk)\ **Replies:** 2\ **Last updated:** [April 7, 2021, 2:12pm UTC](https://discuss.elastic.co/t/logstash-oss-yum-packages-missing/269258 "2021-04-07T14:12:45Z")
Hello, I've been installing the logstash-oss package for some time now from the elastic.co yum repo, but today, it seems that the "-oss" packages are no longer included in the repo. Is this maybe a known change that I m… --- ## [Issues collecting data from Windows servers](https://discuss.elastic.co/t/issues-collecting-data-from-windows-servers/269433)
**Author:** [@Peque](https://discuss.elastic.co/u/Peque)\ **Replies:** 0\ **Last updated:** [April 7, 2021, 9:54am UTC](https://discuss.elastic.co/t/issues-collecting-data-from-windows-servers/269433 "2021-04-07T09:54:31Z")
Hi Forum Just a Noobie for this kind of setup - but linuxnerd for +30 years - I have build a server with Elasticsearch/Kibana and Logstash -and are running into some issues. From Linux - I can collect the data/logs and… --- ## [How to change the symbol separating decimals?](https://discuss.elastic.co/t/how-to-change-the-symbol-separating-decimals/268090)
**Author:** [@Betaminos](https://discuss.elastic.co/u/Betaminos)\ **Replies:** 2\ **Last updated:** [April 7, 2021, 10:01am UTC](https://discuss.elastic.co/t/how-to-change-the-symbol-separating-decimals/268090 "2021-04-07T10:01:43Z")
Hey there, I am currently using Logstash to consume data and insert it into Elasticsearch. Unfortunately for me, I am living in a country that uses . (dots) to group numbers into thousands and uses , (comma) to separat… --- ## [Configuring Logstash Pipeline in Elasticsearch Cloud (elastic.co)](https://discuss.elastic.co/t/configuring-logstash-pipeline-in-elasticsearch-cloud-elastic-co/269432)
**Author:** [@Paresh\_Vaniya](https://discuss.elastic.co/u/Paresh_Vaniya)\ **Replies:** 0\ **Last updated:** [April 7, 2021, 9:52am UTC](https://discuss.elastic.co/t/configuring-logstash-pipeline-in-elasticsearch-cloud-elastic-co/269432 "2021-04-07T09:52:56Z")
Hi There, How Can we configure logstash pipeline in elastic cloud(PFB) ? Or Do we need to install logstash in server then configure its pipeline in server ? --- ## [I want to add "yyyy-MM-dd" for Time field](https://discuss.elastic.co/t/i-want-to-add-yyyy-mm-dd-for-time-field/269079)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 4\ **Last updated:** [April 7, 2021, 6:25am UTC](https://discuss.elastic.co/t/i-want-to-add-yyyy-mm-dd-for-time-field/269079 "2021-04-07T06:25:53Z")
I want to use "yyyy-MM-dd" from current time. Please teach me how to do it. Raw log. 9:36:02 (tableau) (@tableau-SLOG@) === Memory Usage Info === I use below grok. grok { match =\> { "message" =\> "%{TIME:Time} %{GR… --- ## [Logstash does not work](https://discuss.elastic.co/t/logstash-does-not-work/269284)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 4\ **Last updated:** [April 7, 2021, 4:17am UTC](https://discuss.elastic.co/t/logstash-does-not-work/269284 "2021-04-07T04:17:44Z")
I edited the logstash config file and now logstash does not work. The following error occurs in logstash-plain.log. # tail /var/log/logstash/logstash-plain.log at usr.share.logstash.lib.bootstrap.environment.\