# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=120 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 121 --- ## [Remove everything up to -\>](https://discuss.elastic.co/t/remove-everything-up-to/312623)
**Author:** [@ombit](https://discuss.elastic.co/u/ombit)\ **Replies:** 2\ **Last updated:** [August 22, 2022, 4:21pm UTC](https://discuss.elastic.co/t/remove-everything-up-to/312623 "2022-08-22T16:21:00Z")
Hi all, I have a log that is created from various different source servers. Using Wazuh, each item is forwarded to this one log which is then intended to go off to a remote QRADAR server. Wazuh is adding its own part … --- ## [¿Es posible indexar dinamicamente información en varios indices según un campo en un documento json?](https://discuss.elastic.co/t/es-posible-indexar-dinamicamente-informacion-en-varios-indices-segun-un-campo-en-un-documento-json/312635)
**Author:** [@Manuel\_Vazquez](https://discuss.elastic.co/u/Manuel_Vazquez)\ **Replies:** 1\ **Last updated:** [August 22, 2022, 4:04pm UTC](https://discuss.elastic.co/t/es-posible-indexar-dinamicamente-informacion-en-varios-indices-segun-un-campo-en-un-documento-json/312635 "2022-08-22T16:04:23Z")
¿Aun no he profundizado en la documentación de logstash, pero existe la posibiidad de indexar un documento json de forma que el nombre del indice donde se va indexar el documento venga o se calcule por el contenido o par… --- ## [Field is mapped as object but found a concrete value](https://discuss.elastic.co/t/field-is-mapped-as-object-but-found-a-concrete-value/312638)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 0\ **Last updated:** [August 22, 2022, 3:43pm UTC](https://discuss.elastic.co/t/field-is-mapped-as-object-but-found-a-concrete-value/312638 "2022-08-22T15:43:49Z")
Hello guys, I know there is a boatload of threads about this topic, but unfortunately i cannot resolve my issue. I have an index that has this mapping: "debug": { "properties" : { "input" : { "properties" : { … --- ## [Logstash behavior to output to Elastic Search closed index](https://discuss.elastic.co/t/logstash-behavior-to-output-to-elastic-search-closed-index/312590)
**Author:** [@sg.gpar](https://discuss.elastic.co/u/sg.gpar)\ **Replies:** 0\ **Last updated:** [August 22, 2022, 9:23am UTC](https://discuss.elastic.co/t/logstash-behavior-to-output-to-elastic-search-closed-index/312590 "2022-08-22T09:23:54Z")
logstash output is configured as Elastic Search index. Somehow index was closed in between after loading some data, For pending data it will reload automatically after index was opened ? --- ## [Determining which item in patterns array of a grok plugin was matched](https://discuss.elastic.co/t/determining-which-item-in-patterns-array-of-a-grok-plugin-was-matched/312538)
**Author:** [@who](https://discuss.elastic.co/u/who)\ **Replies:** 0\ **Last updated:** [August 21, 2022, 6:31am UTC](https://discuss.elastic.co/t/determining-which-item-in-patterns-array-of-a-grok-plugin-was-matched/312538 "2022-08-21T06:31:48Z")
Assuming we're using the grok filter plugin in the "multiple patterns for a single field" mode: filter { grok { match =\> { "message" =\> \[ "Duration: %{NUMBER:duration}", … --- ## [Dissectfailure when it shouldn't](https://discuss.elastic.co/t/dissectfailure-when-it-shouldnt/312375)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 3\ **Last updated:** [August 22, 2022, 4:10am UTC](https://discuss.elastic.co/t/dissectfailure-when-it-shouldnt/312375 "2022-08-22T04:10:25Z")
Hi All, I'm puzzled. I have a logstash configuration working on two servers. First is fine, but on the second an error says there is a dissect failure \[2022-08-18T11:13:02,260\]\[WARN \]\[org.logstash.dissect.Dissector\] D… --- ## [Initializing a new grok filter from ruby filter](https://discuss.elastic.co/t/initializing-a-new-grok-filter-from-ruby-filter/311722)
**Author:** [@who](https://discuss.elastic.co/u/who)\ **Replies:** 2\ **Last updated:** [August 21, 2022, 6:19am UTC](https://discuss.elastic.co/t/initializing-a-new-grok-filter-from-ruby-filter/311722 "2022-08-21T06:19:00Z")
I'm going to declare and call grok filter from inside a ruby filter like this: ruby { code =\> "@grok = LogStash::Filters::Grok @grok.new(event.get("message"), "\\d+") #call grok plugin for this pipleline " }… --- ## [Jdbc\_streaming statement using a field from input?](https://discuss.elastic.co/t/jdbc-streaming-statement-using-a-field-from-input/312537)
**Author:** [@deep08](https://discuss.elastic.co/u/deep08)\ **Replies:** 0\ **Last updated:** [August 21, 2022, 2:55am UTC](https://discuss.elastic.co/t/jdbc-streaming-statement-using-a-field-from-input/312537 "2022-08-21T02:55:34Z")
Hello , I have a jdbc\_streaming filter and I want to pass statement as a field value from input . I tried different ways to inject field value in statement but always seeing "exception":"Java::JavaSql::SQLSyntaxErrorEx… --- ## [Logstash http input fails](https://discuss.elastic.co/t/logstash-http-input-fails/312268)
**Author:** [@cihady](https://discuss.elastic.co/u/cihady)\ **Replies:** 3\ **Last updated:** [August 20, 2022, 6:56am UTC](https://discuss.elastic.co/t/logstash-http-input-fails/312268 "2022-08-20T06:56:27Z")
I am trying to api call logstash by using http input but when I send request I got error below Here is my config file input { http { host=\>"0.0.0.0" port =\>"2020" } } output { elasticsearch { hosts =\> … --- ## [How to write a filter for a file containing a mix of xml and non xml messages](https://discuss.elastic.co/t/how-to-write-a-filter-for-a-file-containing-a-mix-of-xml-and-non-xml-messages/312503)
**Author:** [@Patr123](https://discuss.elastic.co/u/Patr123)\ **Replies:** 2\ **Last updated:** [August 19, 2022, 9:46pm UTC](https://discuss.elastic.co/t/how-to-write-a-filter-for-a-file-containing-a-mix-of-xml-and-non-xml-messages/312503 "2022-08-19T21:46:15Z")
Hello, I have a log file which is a mix of xml and regular (non xml) lines. I need to apply grok filer + xml filter to the lines that has xml block and apply only grok filter to the regular lines. For that I need help i… --- ## [Using event.get to get variable initialized to string](https://discuss.elastic.co/t/using-event-get-to-get-variable-initialized-to-string/312360)
**Author:** [@Mor\_Y](https://discuss.elastic.co/u/Mor_Y)\ **Replies:** 2\ **Last updated:** [August 19, 2022, 6:32pm UTC](https://discuss.elastic.co/t/using-event-get-to-get-variable-initialized-to-string/312360 "2022-08-19T18:32:18Z")
Hi, I want to add some fields that will have the same values of others but with different names: for Example the input json: {"Environment" : "x", "Name" : "y"} The output will be: {"Environment" : "x", "Name" : "y",… --- ## [SSL issue Logstash sending logs to a Kafka instance](https://discuss.elastic.co/t/ssl-issue-logstash-sending-logs-to-a-kafka-instance/312489)
**Author:** [@shanem](https://discuss.elastic.co/u/shanem)\ **Replies:** 0\ **Last updated:** [August 19, 2022, 5:33pm UTC](https://discuss.elastic.co/t/ssl-issue-logstash-sending-logs-to-a-kafka-instance/312489 "2022-08-19T17:33:27Z")
We are getting a SSL error sending to Kafka from Logstash. We generated a csr, which was signed by the Kafka's CA. Then we took the pub and private key and converted them to a pkcs12 format with openssl with something … --- ## [RELP plugin does not parse syslogs](https://discuss.elastic.co/t/relp-plugin-does-not-parse-syslogs/312365)
**Author:** [@Fastfox](https://discuss.elastic.co/u/Fastfox)\ **Replies:** 2\ **Last updated:** [August 18, 2022, 4:51pm UTC](https://discuss.elastic.co/t/relp-plugin-does-not-parse-syslogs/312365 "2022-08-18T16:51:30Z")
Hi there, Getting to know Elastic stack here and I noticed that normal TCP reliability is not enough for my usecase where the device will be powered off suddenly at at arbitrary times. It looks like rsyslog RELP protoco… --- ## [Grok Pattern for java duration](https://discuss.elastic.co/t/grok-pattern-for-java-duration/312386)
**Author:** [@rokka](https://discuss.elastic.co/u/rokka)\ **Replies:** 1\ **Last updated:** [August 18, 2022, 4:45pm UTC](https://discuss.elastic.co/t/grok-pattern-for-java-duration/312386 "2022-08-18T16:45:02Z")
Hi, how can I parse the Java duration format like "PT20M36.4402321S"? https://docs.oracle.com/javase/8/docs/api/java/time/Duration.html#parse-java.lang.CharSequence- I can´t find an existing pattern on logstash-patter… --- ## [Update enrich index from logstash](https://discuss.elastic.co/t/update-enrich-index-from-logstash/312378)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 2\ **Last updated:** [August 18, 2022, 4:12pm UTC](https://discuss.elastic.co/t/update-enrich-index-from-logstash/312378 "2022-08-18T16:12:22Z")
I have a logstash jdbc pull that updates an index of device information at hourly intervals. That index is used by an enrich processor to add the device information, like sector, country, lat/long, to the data we ingest… --- ## [Performance scaling for Logstash and Elasticsearch based solution](https://discuss.elastic.co/t/performance-scaling-for-logstash-and-elasticsearch-based-solution/312367)
**Author:** [@rakheshkumbi](https://discuss.elastic.co/u/rakheshkumbi)\ **Replies:** 0\ **Last updated:** [August 18, 2022, 9:54am UTC](https://discuss.elastic.co/t/performance-scaling-for-logstash-and-elasticsearch-based-solution/312367 "2022-08-18T09:54:39Z")
Hello everyone, we have a logstash and Elasticsearch-based solution to analyze network protocols, and I am trying to benchmark the whole solution. When I increase the number of Nodes for client/logstash and Elasticsearc… --- ## [Data Transformation : Create Dynamic field using ruby for array](https://discuss.elastic.co/t/data-transformation-create-dynamic-field-using-ruby-for-array/312249)
**Author:** [@priyankamondalhcl](https://discuss.elastic.co/u/priyankamondalhcl)\ **Replies:** 2\ **Last updated:** [August 18, 2022, 8:11am UTC](https://discuss.elastic.co/t/data-transformation-create-dynamic-field-using-ruby-for-array/312249 "2022-08-18T08:11:25Z")
Hi Team, Hope you doing well! I am facing problem in creating dynamic field for an array in logstash. The list of array is below: \[{"usage\_start\_time": "2022-08-07T22:00:00Z", "usage\_end\_time": "2022-08-07T23:00:00Z… --- ## [AWS WAF: Further parsing of http\_request in logstash](https://discuss.elastic.co/t/aws-waf-further-parsing-of-http-request-in-logstash/312347)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 0\ **Last updated:** [August 18, 2022, 7:00am UTC](https://discuss.elastic.co/t/aws-waf-further-parsing-of-http-request-in-logstash/312347 "2022-08-18T07:00:24Z")
Further to the query: Writing multiline grok The raw data comes in json, so the NEW line is technically new line but \\r\\n in the json payload. "GET /myapp/health HTTP/1.1\\\\r\\\\nHost: 52.123.133.46\\\\r\\\\nUser-Agent: Mozil… --- ## [Logstash Reading Same Data](https://discuss.elastic.co/t/logstash-reading-same-data/312251)
**Author:** [@mehmetalix](https://discuss.elastic.co/u/mehmetalix)\ **Replies:** 1\ **Last updated:** [August 18, 2022, 6:49am UTC](https://discuss.elastic.co/t/logstash-reading-same-data/312251 "2022-08-18T06:49:22Z")
Hi, I have a logstash .conf: input { jdbc { jdbc\_driver\_library =\> "${LOGSTASH\_JDBC\_DRIVER\_JAR\_LOCATION}" jdbc\_driver\_class =\> "${LOGSTASH\_JDBC\_DRIVER}" jdbc\_connection\_string =\> "${LOGSTASH… --- ## [Timezone problem when output to Elasticsearch](https://discuss.elastic.co/t/timezone-problem-when-output-to-elasticsearch/312252)
**Author:** [@kent010341](https://discuss.elastic.co/u/kent010341)\ **Replies:** 10\ **Last updated:** [August 18, 2022, 4:36am UTC](https://discuss.elastic.co/t/timezone-problem-when-output-to-elasticsearch/312252 "2022-08-18T04:36:14Z")
I have multiple log files with the date on their file name, and I want to read them with the file input plugin and send them to an Elasticsearch index (the index name contains the date). However, I have some logs being … --- ## [Gcp pubsub input error](https://discuss.elastic.co/t/gcp-pubsub-input-error/312333)
**Author:** [@Gustavo\_Llermaly](https://discuss.elastic.co/u/Gustavo_Llermaly)\ **Replies:** 0\ **Last updated:** [August 18, 2022, 3:02am UTC](https://discuss.elastic.co/t/gcp-pubsub-input-error/312333 "2022-08-18T03:02:56Z")
Logstash 8.3.3, having this error with gcp pubsub input. Trying with basic settings. \[2022-08-17T22:52:45,136\]\[ERROR\]\[logstash.javapipeline \]\[main\]\[69466b68efba17b190dd16bcd12466cd5488773b860c9a0f79fa6c7c65f5ec61\] A … --- ## [Developing Custom Parser for Logs](https://discuss.elastic.co/t/developing-custom-parser-for-logs/312021)
**Author:** [@sta02](https://discuss.elastic.co/u/sta02)\ **Replies:** 2\ **Last updated:** [August 17, 2022, 3:36pm UTC](https://discuss.elastic.co/t/developing-custom-parser-for-logs/312021 "2022-08-17T15:36:26Z")
Hello, In my installation, the output of Logstash is set to Azure Sentinel. The installation is working fine. However, the client prefers to send logs from inhouse applications to Sentinel via Syslog. Is it possible to… --- ## [How to read mongoDB logs from logstash jdbc plugin](https://discuss.elastic.co/t/how-to-read-mongodb-logs-from-logstash-jdbc-plugin/312301)
**Author:** [@sundar\_elk](https://discuss.elastic.co/u/sundar_elk)\ **Replies:** 0\ **Last updated:** [August 17, 2022, 3:00pm UTC](https://discuss.elastic.co/t/how-to-read-mongodb-logs-from-logstash-jdbc-plugin/312301 "2022-08-17T15:00:28Z")
Hi All, We are trying to get logs from MongoDB database using logstash jdbc plugin and below is the Logstash version :- 7.16.2 MongoDB version :- 4.2 MongoJDBC driver version :- mongojdbc4.1.jar (download jar from Dow… --- ## [Logstash parse json child element, format and insert into elasticsearch](https://discuss.elastic.co/t/logstash-parse-json-child-element-format-and-insert-into-elasticsearch/312230)
**Author:** [@babuzrb](https://discuss.elastic.co/u/babuzrb)\ **Replies:** 9\ **Last updated:** [August 17, 2022, 2:46pm UTC](https://discuss.elastic.co/t/logstash-parse-json-child-element-format-and-insert-into-elasticsearch/312230 "2022-08-17T14:46:50Z")
I have a json file like this: "fruits": { "fruit": \[ { "id": 1, "label": "test", "tag": "fine", "start": "4", "end": "9" }, { "id": 2, "lab… --- ## [Logstash DLQ dir empty, but all events end up duplicated in the Elasticsearch's dlq index](https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250)
**Author:** [@kudlatyjoe](https://discuss.elastic.co/u/kudlatyjoe)\ **Replies:** 0\ **Last updated:** [August 17, 2022, 6:22am UTC](https://discuss.elastic.co/t/logstash-dlq-dir-empty-but-all-events-end-up-duplicated-in-the-elasticsearchs-dlq-index/312250 "2022-08-17T06:22:41Z")
Hi all, I'm currently trying to introduce the dead-letter-plugin to the ELK stack that we use for collecting application logs. We've been having some issues with the logging functionality breaking down and losing some … --- ## [Edit file using logstash](https://discuss.elastic.co/t/edit-file-using-logstash/312157)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 2\ **Last updated:** [August 16, 2022, 6:08pm UTC](https://discuss.elastic.co/t/edit-file-using-logstash/312157 "2022-08-16T18:08:55Z")
Hi, I have already created a topic here to add \[\] to the beginning and end of the file and I can't find the solution so I want to ask if I can edit the file using logstash now my file like this: {"field1": "value1","fi… --- ## [Add \[ \] to codec =\> json](https://discuss.elastic.co/t/add-to-codec-json/312104)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 4\ **Last updated:** [August 16, 2022, 5:49pm UTC](https://discuss.elastic.co/t/add-to-codec-json/312104 "2022-08-16T17:49:27Z")
Hi, can i add \[ \] to json file ? because output{ file { codec =\> json path =\> \["D:/elastic\_stack/test/temp.json"\] } } it gives me { 'a':'a','b':'b'}{'c':'c'} i want to add \[ \] so i wanted like this \[{ 'a' : 'a','b':'… --- ## [Logstash rabbitmq input plugin doesn't read "host" value](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-doesnt-read-host-value/312204)
**Author:** [@thinkorhiking](https://discuss.elastic.co/u/thinkorhiking)\ **Replies:** 0\ **Last updated:** [August 16, 2022, 5:01pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-doesnt-read-host-value/312204 "2022-08-16T17:01:42Z")
I'm having trouble to setup logstash rabbitmq input plugin, to connect to the rabbitmq pod host name. Why it still tries to connect @localhost. logstash does not read the config correctly. Is this a bug or is there any … --- ## [Could not index event to Elasticsearch](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/312023)
**Author:** [@syedabdullah](https://discuss.elastic.co/u/syedabdullah)\ **Replies:** 6\ **Last updated:** [August 16, 2022, 3:57pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/312023 "2022-08-16T15:57:58Z")
Hi I am getting this error in Elasticsearch (Please look at the bolded statements), any sort of help will be welcome. Thank you so much in advance: \[2022-05-03T12:41:26,871\]\[WARN \]\[logstash.outputs.elasticsearch\]\[rtt\_kf… --- ## [Filter out (drop) a log sent to Logstash, based on the values of its fields](https://discuss.elastic.co/t/filter-out-drop-a-log-sent-to-logstash-based-on-the-values-of-its-fields/310664)
**Author:** [@zoug](https://discuss.elastic.co/u/zoug)\ **Replies:** 5\ **Last updated:** [July 28, 2022, 1:47pm UTC](https://discuss.elastic.co/t/filter-out-drop-a-log-sent-to-logstash-based-on-the-values-of-its-fields/310664 "2022-07-28T13:47:15Z")
Hello, I use a pipeline to ingest logs generated by suricata, everything set up with filebeat setup --pipelines --modules suricata,\. I don't want to ingest any "event" logs, only "alert" logs. So I added … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=119) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=121)