# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=357 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 358 --- ## [Filtering rows from a log based in their last status](https://discuss.elastic.co/t/filtering-rows-from-a-log-based-in-their-last-status/232858)
**Author:** [@ManuelChaverra](https://discuss.elastic.co/u/ManuelChaverra)\ **Replies:** 1\ **Last updated:** [May 15, 2020, 9:44pm UTC](https://discuss.elastic.co/t/filtering-rows-from-a-log-based-in-their-last-status/232858 "2020-05-15T21:44:00Z")
Hello everyone, this is my first time using Elastic Stack so I'm trying to learn a lot of things that I'd like to implement in my work. I'm using Filebeat to send logs files to logstash, then I'm filtering information a… --- ## [Best way to drop event based on the value(in an array) of a field](https://discuss.elastic.co/t/best-way-to-drop-event-based-on-the-value-in-an-array-of-a-field/232883)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [May 15, 2020, 9:32pm UTC](https://discuss.elastic.co/t/best-way-to-drop-event-based-on-the-value-in-an-array-of-a-field/232883 "2020-05-15T21:32:26Z")
Hi I want to drop an event if the field "group" contains any of the values in an array, it seems that this is the way to do it, but I was wondering if this is the best way to do it, especially because the array has arrou… --- ## [Upgrade 5.6 to 6.8](https://discuss.elastic.co/t/upgrade-5-6-to-6-8/232872)
**Author:** [@andreacfm](https://discuss.elastic.co/u/andreacfm)\ **Replies:** 1\ **Last updated:** [May 15, 2020, 8:21pm UTC](https://discuss.elastic.co/t/upgrade-5-6-to-6-8/232872 "2020-05-15T20:21:16Z")
I need to upgrade an ELK system from 5.6.2 to 6.8.8. My plan is to upgrade before ES through a rolling upgrade and Kibana and later to upgrade logstash. Will logstash 5.6 work normally with ES 6.8.8 ? --- ## [Introduce Time delay in Mutate Http Filter plugin](https://discuss.elastic.co/t/introduce-time-delay-in-mutate-http-filter-plugin/232731)
**Author:** [@Vinay\_Raju](https://discuss.elastic.co/u/Vinay_Raju)\ **Replies:** 1\ **Last updated:** [May 15, 2020, 3:32pm UTC](https://discuss.elastic.co/t/introduce-time-delay-in-mutate-http-filter-plugin/232731 "2020-05-15T15:32:54Z")
I have a requirement where we would be querying clientid's from postgres DB and pass each of clientid to one of HTTP API and dump the output to file output. Now the DB contains some 100K unique clientid's and i need to … --- ## [Inclusion and regex on conditionals](https://discuss.elastic.co/t/inclusion-and-regex-on-conditionals/232756)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 2\ **Last updated:** [May 15, 2020, 3:29pm UTC](https://discuss.elastic.co/t/inclusion-and-regex-on-conditionals/232756 "2020-05-15T15:29:54Z")
This will match the exact match of "SCD" or will also match "scd" or "SCD-abc"? works like a linux grep? if \[hostgroup\] =~ /SCD/ { mutate { add\_field =\> { "indexname" =\> "scd" }}} What is the diference with: … --- ## [Sending data to to different Elasticsearch instances from Logstash](https://discuss.elastic.co/t/sending-data-to-to-different-elasticsearch-instances-from-logstash/232830)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 2\ **Last updated:** [May 15, 2020, 1:45pm UTC](https://discuss.elastic.co/t/sending-data-to-to-different-elasticsearch-instances-from-logstash/232830 "2020-05-15T13:45:41Z")
I'm looking to send data to two different Elasticsearch instances. If pretty sure that each one of these options will work but many months ago I had a similar setup in a different environment and the way I had implement… --- ## [GeoJSON from location/change datatype of location to Geo\_Point](https://discuss.elastic.co/t/geojson-from-location-change-datatype-of-location-to-geo-point/232805)
**Author:** [@shubham184](https://discuss.elastic.co/u/shubham184)\ **Replies:** 1\ **Last updated:** [May 15, 2020, 1:28pm UTC](https://discuss.elastic.co/t/geojson-from-location-change-datatype-of-location-to-geo-point/232805 "2020-05-15T13:28:07Z")
Hello, I have incoming data in the below format "TestItem": { "OriginCountry": "IN", "OriginCity": "Banglore", "DestinationCountry": "US", "DestinationCity":"New York"} I want to ch… --- ## [Logstash service failure ''Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError"](https://discuss.elastic.co/t/logstash-service-failure-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror/232728)
**Author:** [@soufiane\_adn](https://discuss.elastic.co/u/soufiane_adn)\ **Replies:** 9\ **Last updated:** [May 15, 2020, 1:11pm UTC](https://discuss.elastic.co/t/logstash-service-failure-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror/232728 "2020-05-15T13:11:38Z")
Hello, im trying to run logstash as service by systemctl start logstash but i found the following error on log file \[2020-05-15T00:12:10,751\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStas… --- ## [Sending Data from MySql to ES](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609)
**Author:** [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Replies:** 7\ **Last updated:** [May 15, 2020, 11:08am UTC](https://discuss.elastic.co/t/sending-data-from-mysql-to-es/232609 "2020-05-15T11:08:11Z")
Hi , i am sending 447 documents from logstash but ES is recieving only 43 , my config file : input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://ipaddress:3306/zendb" jdbc\_user =\> "\*\*\*\*" jdbc\_password =\> "\*\*\*\*\*… --- ## [Logstash-to-logstash communication in 2020 - with load balancing](https://discuss.elastic.co/t/logstash-to-logstash-communication-in-2020-with-load-balancing/232602)
**Author:** [@nico127](https://discuss.elastic.co/u/nico127)\ **Replies:** 4\ **Last updated:** [May 15, 2020, 7:55am UTC](https://discuss.elastic.co/t/logstash-to-logstash-communication-in-2020-with-load-balancing/232602 "2020-05-15T07:55:11Z")
Hello, My problem is pretty similar to this one: https://discuss.elastic.co/t/logstash-to-logstash-cluster-with-or-without-load-balancer/178694 I contacted the author, who told me he hadn't found a good solution yet. I… --- ## [Mutate a Nested Field](https://discuss.elastic.co/t/mutate-a-nested-field/231954)
**Author:** [@ckough](https://discuss.elastic.co/u/ckough)\ **Replies:** 5\ **Last updated:** [May 15, 2020, 7:28am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954 "2020-05-15T07:28:19Z")
Hi there, I have an index that looks like so: { "\_index": "ec2\_logs-index-2020.05.11", "\_type": "\_doc", "\_id": "V3LAAnIBk2krYRHTuxCC", "\_version": 1, "\_score": null, "\_source": { "owner": "104115521938"… --- ## [Cannot mutate message in filebeat index created with index template](https://discuss.elastic.co/t/cannot-mutate-message-in-filebeat-index-created-with-index-template/232723)
**Author:** [@Tihomir\_Ilchev](https://discuss.elastic.co/u/Tihomir_Ilchev)\ **Replies:** 1\ **Last updated:** [May 14, 2020, 11:56pm UTC](https://discuss.elastic.co/t/cannot-mutate-message-in-filebeat-index-created-with-index-template/232723 "2020-05-14T23:56:07Z")
Hello, I cannot mutate a message and add a field if the index is created from a template, having a lifecycle policy. The pipeline that I am using is: input { beats { client\_inactivity\_timeout =\>… --- ## [Beginners question - error while loading data from SQL to Elasticsearch](https://discuss.elastic.co/t/beginners-question-error-while-loading-data-from-sql-to-elasticsearch/232557)
**Author:** [@oshritbu](https://discuss.elastic.co/u/oshritbu)\ **Replies:** 1\ **Last updated:** [May 14, 2020, 11:36pm UTC](https://discuss.elastic.co/t/beginners-question-error-while-loading-data-from-sql-to-elasticsearch/232557 "2020-05-14T23:36:37Z")
Hi, I am using the following config file to upload a simple table to Elasticsearch: input { jdbc { jdbc\_driver\_library =\> "C:/Users/michaeos/Desktop/ELK/logstash-7.6.2/sqljdbc\_4.2/enu/jre8/sqljdbc42.jar" jdbc\_connec… --- ## [Geopoint data type not getting updated](https://discuss.elastic.co/t/geopoint-data-type-not-getting-updated/232635)
**Author:** [@shubham184](https://discuss.elastic.co/u/shubham184)\ **Replies:** 5\ **Last updated:** [May 14, 2020, 10:45pm UTC](https://discuss.elastic.co/t/geopoint-data-type-not-getting-updated/232635 "2020-05-14T22:45:26Z")
The issue is location is not being shown as geo\_point data type so i am not able to use the data in the map section. Hello there, I have a JSON in below format { "id": "\*\*\*\*\*\*", "key": "\*\*\*\*\*\*", "v… --- ## [Kafka Output - Increasing batching reduces throughput](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426)
**Author:** [@taylorn](https://discuss.elastic.co/u/taylorn)\ **Replies:** 4\ **Last updated:** [May 14, 2020, 10:23pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426 "2020-05-14T22:23:45Z")
I'm seeing poor performance with the Kafka output plugin when I increase the linger\_ms and batch\_size settings. This is the opposite behavior I would expect after reading the Kafka producer documentation. Increasing batc… --- ## [My Logstash is not parsing data properly](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399)
**Author:** [@pjepsen](https://discuss.elastic.co/u/pjepsen)\ **Replies:** 21\ **Last updated:** [May 14, 2020, 9:50pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399 "2020-05-14T21:50:53Z")
I have been working on a project to upload VPN logs to a logstash instance that I created, and while I am able to see data being sent to the logstash, I am seeing a "\_dateparsefailure" error showing under \[tags\] in my ou… --- ## [How to merge two document in logstash](https://discuss.elastic.co/t/how-to-merge-two-document-in-logstash/232531)
**Author:** [@CambridgeLv](https://discuss.elastic.co/u/CambridgeLv)\ **Replies:** 1\ **Last updated:** [May 14, 2020, 7:01pm UTC](https://discuss.elastic.co/t/how-to-merge-two-document-in-logstash/232531 "2020-05-14T19:01:48Z")
Hi, I am using ELK to visualize the data in logs. Is it possible to merge two documents into one? As I know, each line from filebeat will make one doc output to elasticsearch(or with multi-lines input, then multi-lines w… --- ## [Parsing log](https://discuss.elastic.co/t/parsing-log/232289)
**Author:** [@Luiss\_Anggel\_Carbone](https://discuss.elastic.co/u/Luiss_Anggel_Carbone)\ **Replies:** 3\ **Last updated:** [May 14, 2020, 6:40pm UTC](https://discuss.elastic.co/t/parsing-log/232289 "2020-05-14T18:40:14Z")
Hello greetings I made the configuration to integrate FileBeat, Logstash, and Elastic. I need to create a filter every time I find an ERROR in the log. Any solution for this? Where can I find documentation on this, I … --- ## [Having Issues With JSON Input](https://discuss.elastic.co/t/having-issues-with-json-input/232548)
**Author:** [@astuffedtiger](https://discuss.elastic.co/u/astuffedtiger)\ **Replies:** 4\ **Last updated:** [May 14, 2020, 5:16pm UTC](https://discuss.elastic.co/t/having-issues-with-json-input/232548 "2020-05-14T17:16:40Z")
Hello all, I've looked up and down StackOverflow and these forums along with the Logstash docs, and I can't for the life of me get this working. I am simply trying to use the message field as my source so that the items… --- ## [Adding Field from one index to another index / Merging two Index](https://discuss.elastic.co/t/adding-field-from-one-index-to-another-index-merging-two-index/232518)
**Author:** [@shubham184](https://discuss.elastic.co/u/shubham184)\ **Replies:** 4\ **Last updated:** [May 14, 2020, 4:56pm UTC](https://discuss.elastic.co/t/adding-field-from-one-index-to-another-index-merging-two-index/232518 "2020-05-14T16:56:57Z")
Hello, I have 2 tables coming in as CouchDb changes in logstash, which get converted into two separate indexes. Problem statement: I want to either merge the two indexes or add the fields in one of the indexes, I was th… --- ## [Pipeline to pipeline communication](https://discuss.elastic.co/t/pipeline-to-pipeline-communication/232541)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 3\ **Last updated:** [May 14, 2020, 4:54pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication/232541 "2020-05-14T16:54:56Z")
HI, I got about 15 pipelines, getting data from a DB, and I was thinking to point all those pipelines to another pipeline that will filter and index the data and various questions arise in my mind: 1-Will this create a… --- ## [Logstash - Json file input question](https://discuss.elastic.co/t/logstash-json-file-input-question/232484)
**Author:** [@Yawn](https://discuss.elastic.co/u/Yawn)\ **Replies:** 2\ **Last updated:** [May 14, 2020, 4:39pm UTC](https://discuss.elastic.co/t/logstash-json-file-input-question/232484 "2020-05-14T16:39:16Z")
Hello, I am trying to load the following from a json file, and I had a couple of questions.. \[ { "\_index": "index-001", "\_type": "xxx", "\_id": "m2p2A3EBY37dRdt4jKyq", "\_score": 1.0, "\_source": … --- ## [Duplicate entries in kibana from logstash](https://discuss.elastic.co/t/duplicate-entries-in-kibana-from-logstash/232470)
**Author:** [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Replies:** 9\ **Last updated:** [May 14, 2020, 3:35pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-kibana-from-logstash/232470 "2020-05-14T15:35:03Z")
Hello there, Today I just realized that all my cisco vpn entries are duplicated. I double checked the code and could not find any reason. Can someone help me? Follow the pics: Really appreciate for the attention… --- ## [Drop milliseconds from timestamp](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264)
**Author:** [@alberto1](https://discuss.elastic.co/u/alberto1)\ **Replies:** 7\ **Last updated:** [May 14, 2020, 2:35pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264 "2020-05-14T14:35:23Z")
Hi, I am trying to drop the 4 miliseconds from the timestamp but i cannot make it work and it's given me an error of \_dateparsefailure the bit where I have the error is: grok { match =\> \[ "message", "%{TIMESTAMP\_ISO… --- ## [Mutate filter specific word](https://discuss.elastic.co/t/mutate-filter-specific-word/231946)
**Author:** [@Jeffreyshoptrader](https://discuss.elastic.co/u/Jeffreyshoptrader)\ **Replies:** 5\ **Last updated:** [May 14, 2020, 1:59pm UTC](https://discuss.elastic.co/t/mutate-filter-specific-word/231946 "2020-05-14T13:59:38Z")
Hi guys, I'm receiving information from specific webservers. I wish to filter out lines with errors from my log files. To start I want to add a field for PHP fatal errors. Right now, I have this: input { bea… --- ## [Retrieve Fields from the String](https://discuss.elastic.co/t/retrieve-fields-from-the-string/231450)
**Author:** [@karthik3](https://discuss.elastic.co/u/karthik3)\ **Replies:** 2\ **Last updated:** [May 14, 2020, 1:36pm UTC](https://discuss.elastic.co/t/retrieve-fields-from-the-string/231450 "2020-05-14T13:36:39Z")
Hi, How we can get the value from the string in logstash. Sample Log: 2020-04-27 00:44:30,445 INFO START \[\[Thread1\]\]logger - { "sourcetype":"MachineA", "event":{"timestamp":"2020-04-29T00:00:00.000","logLevel": "Deb… --- ## [Logstash not running when configured with multiple pipline yaml configuration file](https://discuss.elastic.co/t/logstash-not-running-when-configured-with-multiple-pipline-yaml-configuration-file/232594)
**Author:** [@balavenkatesh3322](https://discuss.elastic.co/u/balavenkatesh3322)\ **Replies:** 2\ **Last updated:** [May 14, 2020, 1:20pm UTC](https://discuss.elastic.co/t/logstash-not-running-when-configured-with-multiple-pipline-yaml-configuration-file/232594 "2020-05-14T13:20:09Z")
This is my pipeline configuration file - - pipeline.id: my-pipeline\_1 path.config: "C:\\\\logstash\\\\logstash-7.6.1\\\\config\\\\mongodata\_non\_mcn.conf" - pipeline.id: my-other-pipeline path.config: "C:\\\\log… --- ## [Logstash http input response](https://discuss.elastic.co/t/logstash-http-input-response/232650)
**Author:** [@nitzan.karni](https://discuss.elastic.co/u/nitzan.karni)\ **Replies:** 0\ **Last updated:** [May 14, 2020, 1:16pm UTC](https://discuss.elastic.co/t/logstash-http-input-response/232650 "2020-05-14T13:16:05Z")
Hi there, I have set up a http input plugin on my logstash with certain port. As expected when I refer to the correct url the event is recieved in the logstash pipeline and I get "ok" message. What I would like to do … --- ## [Failure when parsing logs -\> results in a lot of worthless Logs](https://discuss.elastic.co/t/failure-when-parsing-logs-results-in-a-lot-of-worthless-logs/232612)
**Author:** [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)\ **Replies:** 3\ **Last updated:** [May 14, 2020, 12:28pm UTC](https://discuss.elastic.co/t/failure-when-parsing-logs-results-in-a-lot-of-worthless-logs/232612 "2020-05-14T12:28:16Z")
Hello community - i've set up a logstash config to parse some VmWare Host Logs, but i see an error in my Logstash logs: error: :response=\>{"index"=\>{"\_index"=\>"vmware-2020.05.14", "\_type"=\>"\_doc", "\_id"=\>"ejurEnIBcPJ… --- ## [Logstash is not picking up the files](https://discuss.elastic.co/t/logstash-is-not-picking-up-the-files/232132)
**Author:** [@srinjan\_dutta](https://discuss.elastic.co/u/srinjan_dutta)\ **Replies:** 6\ **Last updated:** [May 14, 2020, 12:26pm UTC](https://discuss.elastic.co/t/logstash-is-not-picking-up-the-files/232132 "2020-05-14T12:26:47Z")
Hi All, I am stuck with this for 3 -4 weeks . I have a configuration file as below and trying to pick up a xml file using logstash. I have used all the resources available under the sun to resolve it but to no avail. I… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=356) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=358)