# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=488 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 489 --- ## [Nested json using logstash mutate filter](https://discuss.elastic.co/t/nested-json-using-logstash-mutate-filter/196551)
**Author:** [@chimbu3306](https://discuss.elastic.co/u/chimbu3306)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 3:26pm UTC](https://discuss.elastic.co/t/nested-json-using-logstash-mutate-filter/196551 "2019-08-23T15:26:39Z")
I have the below json where i am trying to create a new json field using the mutate filter but i am not getting the expected results. below is my json and the filter i used Actual Json: { "@timestamp": "2019-08-23T14:5… --- ## [Removing a filed in a json using mutate filter in logstash](https://discuss.elastic.co/t/removing-a-filed-in-a-json-using-mutate-filter-in-logstash/195746)
**Author:** [@chimbu3306](https://discuss.elastic.co/u/chimbu3306)\ **Replies:** 4\ **Last updated:** [August 23, 2019, 2:56pm UTC](https://discuss.elastic.co/t/removing-a-filed-in-a-json-using-mutate-filter-in-logstash/195746 "2019-08-23T14:56:58Z")
I have a json where i need to remove a field using mutate or any other filter, { "clientUserAgent": null, "environment": "Logs\_0z9uc0nzlh/qa", "timestamp": 1566220446505, "sequence": 3, "… --- ## [Logstash output pretty json file?](https://discuss.elastic.co/t/logstash-output-pretty-json-file/196520)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 2:10pm UTC](https://discuss.elastic.co/t/logstash-output-pretty-json-file/196520 "2019-08-23T14:10:33Z")
Hi, I know, that I can use json\_lines codec for file output to generate a one json line per event. Is there also a possibility to create a pretty json file as file output which is easier readable for humans? Thanks, A… --- ## [Default Logstash Encryption to Elastic Cloud?](https://discuss.elastic.co/t/default-logstash-encryption-to-elastic-cloud/196530)
**Author:** [@tyler\_hilsabeck](https://discuss.elastic.co/u/tyler_hilsabeck)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 2:07pm UTC](https://discuss.elastic.co/t/default-logstash-encryption-to-elastic-cloud/196530 "2019-08-23T14:07:56Z")
Will Logstash automatically encrypt traffic to an elastic cloud instance if the cloud endpoint it specified as https? Upon packet inspection it seems that it is but I want to verify. This is my pipefile config: output{… --- ## [How to terminate logstash at EOF in a single pipeline?](https://discuss.elastic.co/t/how-to-terminate-logstash-at-eof-in-a-single-pipeline/196534)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 2:07pm UTC](https://discuss.elastic.co/t/how-to-terminate-logstash-at-eof-in-a-single-pipeline/196534 "2019-08-23T14:07:29Z")
Hi, We will get csv exports of error data from our customer and want to import this export into our own elasticsearch instance to analyze the provided data (log data). We use losgtash to import csv files to elasticsear… --- ## [Csv filter plugin: how to deal with masked quotations?](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [August 23, 2019, 2:05pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488 "2019-08-23T14:05:03Z")
Hi, I have following csv file. "@timestamp";"col\_a";"col\_b";"col\_c";"col\_d" "2019-08-23 04:43:16.821";"\";b;c;d I am trying to parse it in logstash with following filter: filt… --- ## [All syslogs have same severity](https://discuss.elastic.co/t/all-syslogs-have-same-severity/196390)
**Author:** [@jmorris](https://discuss.elastic.co/u/jmorris)\ **Replies:** 3\ **Last updated:** [August 23, 2019, 1:59pm UTC](https://discuss.elastic.co/t/all-syslogs-have-same-severity/196390 "2019-08-23T13:59:53Z")
I'm still very new to elastic stack and I ditched filebeat because it made all of my syslogs look like they were coming from the same host. I have my syslogs coming into logstash now and for the most part its working, h… --- ## [Error on pipeline configuration with Java plugin](https://discuss.elastic.co/t/error-on-pipeline-configuration-with-java-plugin/196484)
**Author:** [@DottorP](https://discuss.elastic.co/u/DottorP)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 1:58pm UTC](https://discuss.elastic.co/t/error-on-pipeline-configuration-with-java-plugin/196484 "2019-08-23T13:58:56Z")
Hi. I wrote a filter plugin in Java named blob\_to\_clob\_filter. When using it in a pipeline I get the following error: \[2019-08-23T11:20:38,254\]\[DEBUG\]\[logstash.plugins.registry\] On demand adding plugin to the registry {… --- ## [Finding all patterns that match](https://discuss.elastic.co/t/finding-all-patterns-that-match/196539)
**Author:** [@sanketn](https://discuss.elastic.co/u/sanketn)\ **Replies:** 0\ **Last updated:** [August 23, 2019, 1:36pm UTC](https://discuss.elastic.co/t/finding-all-patterns-that-match/196539 "2019-08-23T13:36:17Z")
I am just starting with logstash; I have a use case where bunch of logs(each line is a json) are persisted to S3 by an external service. I have bunch of filters that represents some field values in this log(e.g. key1=val… --- ## [Don't pick me up the Logs of / var / log / httpd / access\_log](https://discuss.elastic.co/t/dont-pick-me-up-the-logs-of-var-log-httpd-access-log/196293)
**Author:** [@cristianaxion](https://discuss.elastic.co/u/cristianaxion)\ **Replies:** 2\ **Last updated:** [August 23, 2019, 9:03am UTC](https://discuss.elastic.co/t/dont-pick-me-up-the-logs-of-var-log-httpd-access-log/196293 "2019-08-23T09:03:31Z")
Hi, I want to link the logs of / var / log / httpd / access\_log in kibana but I didn't pick them up ... I just picked up the logs located in var / log / messages ....Help Pleaseeeeee!!!!! Is urgent. --- ## [Elapsed time between starting log line ending log in a log file](https://discuss.elastic.co/t/elapsed-time-between-starting-log-line-ending-log-in-a-log-file/194698)
**Author:** [@madhu\_sekar\_Akula](https://discuss.elastic.co/u/madhu_sekar_Akula)\ **Replies:** 2\ **Last updated:** [August 23, 2019, 6:21am UTC](https://discuss.elastic.co/t/elapsed-time-between-starting-log-line-ending-log-in-a-log-file/194698 "2019-08-23T06:21:10Z")
Hi iam new to logstash i need to find out the elapsed time between starting event date to ending event date in log file ,i dont have any unique ids for starting event and ending events so how to achive this with logstas… --- ## [Logstash can't recognize csv columns](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396)
**Author:** [@m.thecoder](https://discuss.elastic.co/u/m.thecoder)\ **Replies:** 5\ **Last updated:** [August 23, 2019, 12:01am UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396 "2019-08-23T00:01:35Z")
Hello Guys, I've configured logstah to read multiple csv files, each one has a different columns set (different column names), it reads the first one successfully, but it never reads the columns for any csv after readin… --- ## [Logstash converting data from a flattened table from a SQL query into JSON inner object](https://discuss.elastic.co/t/logstash-converting-data-from-a-flattened-table-from-a-sql-query-into-json-inner-object/196403)
**Author:** [@aohsie](https://discuss.elastic.co/u/aohsie)\ **Replies:** 4\ **Last updated:** [August 22, 2019, 9:55pm UTC](https://discuss.elastic.co/t/logstash-converting-data-from-a-flattened-table-from-a-sql-query-into-json-inner-object/196403 "2019-08-22T21:55:58Z")
I'm looking to index flattened data into Elasticsearch without creating duplicate data. For example, if a person has multiple cars, then a flattened table will look something like this: | Name | Car … --- ## [Adding an Id into an array field and split filter](https://discuss.elastic.co/t/adding-an-id-into-an-array-field-and-split-filter/196223)
**Author:** [@manunc](https://discuss.elastic.co/u/manunc)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 9:40pm UTC](https://discuss.elastic.co/t/adding-an-id-into-an-array-field-and-split-filter/196223 "2019-08-22T21:40:41Z")
Hello, I have a event that I want to split in 2 because the field 'object.Voie' is a json array composed of 2 item. When splitting the document I miss one information, I need to add an Id corresponding to the index of … --- ## [Keeping duplicates](https://discuss.elastic.co/t/keeping-duplicates/196337)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 4\ **Last updated:** [August 22, 2019, 8:35pm UTC](https://discuss.elastic.co/t/keeping-duplicates/196337 "2019-08-22T20:35:13Z")
Hi, Is there anyway to keep duplicate rows as they are and maybe include a number which differentiates them. The duplicate records aren't necessarily duplicates so much as they are multiple instances of the same item. T… --- ## [Bulk insert to Elastic from Logstash "bulk\_path"](https://discuss.elastic.co/t/bulk-insert-to-elastic-from-logstash-bulk-path/196352)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 5\ **Last updated:** [August 22, 2019, 7:42pm UTC](https://discuss.elastic.co/t/bulk-insert-to-elastic-from-logstash-bulk-path/196352 "2019-08-22T19:42:12Z")
I have a huge csv file, like 500k rows, that I'm processing and want to push to elastic This output: elasticsearch { index =\> "%{pipelineId}" hosts =\> \["${domain:port}/"\] } outputs one document at … --- ## [Timezone Precedence in Date Filter](https://discuss.elastic.co/t/timezone-precedence-in-date-filter/196380)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 6:58pm UTC](https://discuss.elastic.co/t/timezone-precedence-in-date-filter/196380 "2019-08-22T18:58:03Z")
TL;DR Which takes precedence in a date filter - the timezone in the timestamp or the timezone specified in the filter? Longer Version A question regarding the date filter. I have the following date filter: date { m… --- ## [Data Type conversion](https://discuss.elastic.co/t/data-type-conversion/196349)
**Author:** [@rohitg](https://discuss.elastic.co/u/rohitg)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 6:10pm UTC](https://discuss.elastic.co/t/data-type-conversion/196349 "2019-08-22T18:10:23Z")
How to convert? Latitude and longitude from string type to float type Date from String type to Date type or DateTime type Thanks. --- ## [How to divide data from one agent into two indexes?](https://discuss.elastic.co/t/how-to-divide-data-from-one-agent-into-two-indexes/196360)
**Author:** [@psyapathy](https://discuss.elastic.co/u/psyapathy)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 6:03pm UTC](https://discuss.elastic.co/t/how-to-divide-data-from-one-agent-into-two-indexes/196360 "2019-08-22T18:03:10Z")
Hello! I have a Windows server, it is winlogbeat, data is sent to Elasticsearch via Logstash. I'm getting data from the security log and sysmon. All this data is sent to the index "index1". But there is data, which ha… --- ## [Epoch unix to date](https://discuss.elastic.co/t/epoch-unix-to-date/196364)
**Author:** [@cjb312](https://discuss.elastic.co/u/cjb312)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 5:58pm UTC](https://discuss.elastic.co/t/epoch-unix-to-date/196364 "2019-08-22T17:58:26Z")
I'm trying to convert 1422083623.329786 to a format like 1-24-2015 7:13:43.329 I've been looking it up through some past posts but I'm pretty lost. My current grok filter for it is %{NUMBER:ts} Thanks in advance --- ## [Logstash Filter to drop Beats events](https://discuss.elastic.co/t/logstash-filter-to-drop-beats-events/196372)
**Author:** [@jwwags92](https://discuss.elastic.co/u/jwwags92)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 5:46pm UTC](https://discuss.elastic.co/t/logstash-filter-to-drop-beats-events/196372 "2019-08-22T17:46:25Z")
I tried hard to filter most events i'm not interested in in the Winlogbeat config. However, I've now got ~100 Windows servers and I don't want to edit 100 files to filter events 256 / 258. I'm trying to use Logstash to d… --- ## [Logstash not creating index daily based on date](https://discuss.elastic.co/t/logstash-not-creating-index-daily-based-on-date/189306)
**Author:** [@svm89](https://discuss.elastic.co/u/svm89)\ **Replies:** 8\ **Last updated:** [August 21, 2019, 2:27pm UTC](https://discuss.elastic.co/t/logstash-not-creating-index-daily-based-on-date/189306 "2019-08-21T14:27:13Z")
Hi guys, I have set up a ELK stack platform for work and I am facing some issues with some indices, basically they are not rotating daily based on the date (like logstash-YYYY.MM.DD). The pipeline config is the followi… --- ## [Partially match beat fields](https://discuss.elastic.co/t/partially-match-beat-fields/196194)
**Author:** [@mattsdevop](https://discuss.elastic.co/u/mattsdevop)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 1:44pm UTC](https://discuss.elastic.co/t/partially-match-beat-fields/196194 "2019-08-22T13:44:50Z")
I see some other topics on partial matches for \[message\]. But, I can't find any topics on partial matches for \[fields\]\[value\] that's sent from filebeat. My situation is that I have the name of a program followed by a ve… --- ## [Xml can't be parsed!](https://discuss.elastic.co/t/xml-cant-be-parsed/196221)
**Author:** [@wzxmt\_l](https://discuss.elastic.co/u/wzxmt_l)\ **Replies:** 4\ **Last updated:** [August 22, 2019, 1:00pm UTC](https://discuss.elastic.co/t/xml-cant-be-parsed/196221 "2019-08-22T13:00:07Z")
hi Here are my xml data and logstash conf.The entire event is entered when the data is entered into logstash;I'm trying to parse xml files with the xpath plugin but it doesn't give me any result! \ **Author:** [@Airn5475](https://discuss.elastic.co/u/Airn5475)\ **Replies:** 0\ **Last updated:** [August 22, 2019, 12:58pm UTC](https://discuss.elastic.co/t/how-to-delete-a-document-in-elasticsearch-using-logstash/196326 "2019-08-22T12:58:10Z") My ETL Stack is JSON File =\> Filebeat =\> Logstash =\> Elasticsearch I am trying to specify an action in my JSON object that is either update or delete Here's a partial of my current JSON { "key":123,"index\_name":"compa… --- ## [Logstash crash on start due to PQ](https://discuss.elastic.co/t/logstash-crash-on-start-due-to-pq/196155)
**Author:** [@matteoclc](https://discuss.elastic.co/u/matteoclc)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 12:57pm UTC](https://discuss.elastic.co/t/logstash-crash-on-start-due-to-pq/196155 "2019-08-22T12:57:03Z")
Hi all, my logstash (6.5.0) crash on start with the following error org.logstash.ackedqueue.io.MmapPageIOV2$PageIOInvalidElementException: Element seqNum 0 is expected to be 3920351 \[2019-08-21T17:15:31,045\]\[ERROR\]\[lo… --- ## [Check if substring exist in array of strings](https://discuss.elastic.co/t/check-if-substring-exist-in-array-of-strings/196296)
**Author:** [@mattia\_conversano](https://discuss.elastic.co/u/mattia_conversano)\ **Replies:** 4\ **Last updated:** [August 22, 2019, 12:54pm UTC](https://discuss.elastic.co/t/check-if-substring-exist-in-array-of-strings/196296 "2019-08-22T12:54:41Z")
Hi Team, I have a problem with an array of strings, my goal is to detect if this string: attack-sqli exist in array like this: "audit\_data":{ "messages":\[ "Warning. detected XSS using libinjection. \[fi… --- ## [Append to nested type object from multiple jdbc input stream using logstash aggregation](https://discuss.elastic.co/t/append-to-nested-type-object-from-multiple-jdbc-input-stream-using-logstash-aggregation/196056)
**Author:** [@sid1990](https://discuss.elastic.co/u/sid1990)\ **Replies:** 3\ **Last updated:** [August 22, 2019, 12:28pm UTC](https://discuss.elastic.co/t/append-to-nested-type-object-from-multiple-jdbc-input-stream-using-logstash-aggregation/196056 "2019-08-22T12:28:30Z")
Hi, I'm trying to read from multiple jdbc inputs and aggregate the result which would be inserted into a field which is mapped to a nested type in elasticsearch. But the input stream is not executed sequentially. I did… --- ## [How to calculate optimal heapsize for Logstash?](https://discuss.elastic.co/t/how-to-calculate-optimal-heapsize-for-logstash/196264)
**Author:** [@viratag26](https://discuss.elastic.co/u/viratag26)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 12:24pm UTC](https://discuss.elastic.co/t/how-to-calculate-optimal-heapsize-for-logstash/196264 "2019-08-22T12:24:13Z")
My logstash keeps breaking down due to JAVA heap memory error. The heap allocated is not sufficient to keep logstash working. Please let me know if there is some method to calculate optimal heap that needs to be allocate… --- ## [Accessing a timestamp within a json structure to be used as @timestamp](https://discuss.elastic.co/t/accessing-a-timestamp-within-a-json-structure-to-be-used-as-timestamp/196307)
**Author:** [@Franco](https://discuss.elastic.co/u/Franco)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 12:16pm UTC](https://discuss.elastic.co/t/accessing-a-timestamp-within-a-json-structure-to-be-used-as-timestamp/196307 "2019-08-22T12:16:58Z")
Hi all. I'm currently trying to parse a json log format like that: {"rmiCall":{"method":"jDVServer#getDocnrs","objectInfo":"jdv.server.jDVServer@53dacd14","params":\["jdv.server.doc.vvdoc.VVDoc",{"PROJECT":"vvrp","WORDD… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=487) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=489)