# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=417 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 418 --- ## [If I add a new field in to my Logstash.conf, my old index logs recognize it?](https://discuss.elastic.co/t/if-i-add-a-new-field-in-to-my-logstash-conf-my-old-index-logs-recognize-it/215788)
**Author:** [@Daniel\_Gonzalez1](https://discuss.elastic.co/u/Daniel_Gonzalez1)\ **Replies:** 2\ **Last updated:** [January 20, 2020, 7:15pm UTC](https://discuss.elastic.co/t/if-i-add-a-new-field-in-to-my-logstash-conf-my-old-index-logs-recognize-it/215788 "2020-01-20T19:15:06Z")
Hello Everyone! Doing the endless comparasion and questions between Splunk and Elastic. With Splunk you can extract all the time new fields from the logs that are already indexed, this is very helpful because if you don… --- ## [\[search engine\] Indexing multimedia files with elk](https://discuss.elastic.co/t/search-engine-indexing-multimedia-files-with-elk/215779)
**Author:** [@Jassem\_Torkhani](https://discuss.elastic.co/u/Jassem_Torkhani)\ **Replies:** 0\ **Last updated:** [January 20, 2020, 4:19pm UTC](https://discuss.elastic.co/t/search-engine-indexing-multimedia-files-with-elk/215779 "2020-01-20T16:19:40Z")
Hello everyone, Am working on a search engine project and am new experiencing with the ELK stack , so after installing and configuring it , I am now called to pass csv files which came originally from parsed output v… --- ## [Dynamic JSON parsing](https://discuss.elastic.co/t/dynamic-json-parsing/215644)
**Author:** [@richard\_N](https://discuss.elastic.co/u/richard_N)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 3:38pm UTC](https://discuss.elastic.co/t/dynamic-json-parsing/215644 "2020-01-20T15:38:34Z")
I'm trying to change a document from this. Is this possible? { "zenoss01.hostname": { "issued\_to": null, "cert\_sans": "", "valid\_till": "2021-01-17", "valid\_from": "2016-01-18", "issuer\_ou": null, "days\_left": 36… --- ## [Can Output file-plugin point to /dev/null?](https://discuss.elastic.co/t/can-output-file-plugin-point-to-dev-null/215645)
**Author:** [@alexolivan](https://discuss.elastic.co/u/alexolivan)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 3:23pm UTC](https://discuss.elastic.co/t/can-output-file-plugin-point-to-dev-null/215645 "2020-01-20T15:23:15Z")
Hi Forum. I'm just discovering how the new ELK+x-pack pieces work together, and, since I've realiced that I've got to create a parallel monitoring cluster to fully learn the current ELK potential, I'm thinking in advanc… --- ## [Aggregate filter usage](https://discuss.elastic.co/t/aggregate-filter-usage/215744)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 2\ **Last updated:** [January 20, 2020, 3:10pm UTC](https://discuss.elastic.co/t/aggregate-filter-usage/215744 "2020-01-20T15:10:44Z")
Hi Team, I want to write filter to compare current event with previous one. My requirement is as below: Let's say first log has event id as 100, I want to check when the second log comes it should be 1 greater than pre… --- ## [Cannot get input from elasticsearch - logstash terminates itself](https://discuss.elastic.co/t/cannot-get-input-from-elasticsearch-logstash-terminates-itself/215713)
**Author:** [@peter\_pan](https://discuss.elastic.co/u/peter_pan)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 3:08pm UTC](https://discuss.elastic.co/t/cannot-get-input-from-elasticsearch-logstash-terminates-itself/215713 "2020-01-20T15:08:49Z")
Hi ELKers, I would like to ask about your help on the following issue, which it is that I cannot get input from elasticsearch into logstash. My configuration is simple as that in the logstash-simple.conf: input { ela… --- ## [Logstash JDBC strange time convert](https://discuss.elastic.co/t/logstash-jdbc-strange-time-convert/215760)
**Author:** [@GaffaOverflow](https://discuss.elastic.co/u/GaffaOverflow)\ **Replies:** 0\ **Last updated:** [January 20, 2020, 2:51pm UTC](https://discuss.elastic.co/t/logstash-jdbc-strange-time-convert/215760 "2020-01-20T14:51:30Z")
Hi Everyone, Im trying to add Logs vom a MSSQL Database to my ELk-Stack. I had some trouble regarding the Timezone. I want a field with the UTC Time (will be the @timestamp field) i had problems with converting the fi… --- ## [How can I remove unwanted parameters in logstash geoip](https://discuss.elastic.co/t/how-can-i-remove-unwanted-parameters-in-logstash-geoip/215673)
**Author:** [@ruranga](https://discuss.elastic.co/u/ruranga)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 2:37pm UTC](https://discuss.elastic.co/t/how-can-i-remove-unwanted-parameters-in-logstash-geoip/215673 "2020-01-20T14:37:41Z")
Hi, Actually I don't need following parameters in logstash geoip out put geoip.city\_name geoip.continent\_code geoip.dma\_code geoip.ip geoip.latitude geoip.location.lat geoip.location.lon geoip.longitude geoip.p… --- ## [Hard regex](https://discuss.elastic.co/t/hard-regex/215714)
**Author:** [@Arnodl](https://discuss.elastic.co/u/Arnodl)\ **Replies:** 4\ **Last updated:** [January 20, 2020, 2:01pm UTC](https://discuss.elastic.co/t/hard-regex/215714 "2020-01-20T14:01:46Z")
hi, i need your help to write regex. It's my e.g message: scc\_check\_file ignored non-ASCII file /etc/sysconfig/auditd: '/etc/sysconfig/auditd: regular file, no read permission' I want to write if, which checks in part … --- ## [How to have a checkpoint using a TimeStamp field (sql\_last\_value)](https://discuss.elastic.co/t/how-to-have-a-checkpoint-using-a-timestamp-field-sql-last-value/215745)
**Author:** [@NaveenSilvester](https://discuss.elastic.co/u/NaveenSilvester)\ **Replies:** 0\ **Last updated:** [January 20, 2020, 1:51pm UTC](https://discuss.elastic.co/t/how-to-have-a-checkpoint-using-a-timestamp-field-sql-last-value/215745 "2020-01-20T13:51:04Z")
Hi ELK community, Here is a challenge that I am facing while tracking a TimeStamp filed in logstash. I am connecting to Postgresql table which has a field named "Recorded\_time" with a datatype TimeStamp. I want to hav… --- ## [Avoid reloading (duplicate) csv records into same index](https://discuss.elastic.co/t/avoid-reloading-duplicate-csv-records-into-same-index/215734)
**Author:** [@bharat1](https://discuss.elastic.co/u/bharat1)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 1:45pm UTC](https://discuss.elastic.co/t/avoid-reloading-duplicate-csv-records-into-same-index/215734 "2020-01-20T13:45:22Z")
Hi All, I have a working config for loading csv records from logstash to elasticsearch. However when I try to restart the logstash service, the same csv file records are reloaded again into same index and creating dupli… --- ## [Output: how to save a field to a pdf file?](https://discuss.elastic.co/t/output-how-to-save-a-field-to-a-pdf-file/211618)
**Author:** [@ITIC](https://discuss.elastic.co/u/ITIC)\ **Replies:** 3\ **Last updated:** [January 20, 2020, 12:39pm UTC](https://discuss.elastic.co/t/output-how-to-save-a-field-to-a-pdf-file/211618 "2020-01-20T12:39:09Z")
Hi there I'm using logstash (docker.elastic.co/logstash/logstash:7.4.0) to ingest data to our open data portal and saving said data to files (.csv, .geojson, etc.). All was well, until we got to one of our data sources… --- ## [More input based on first input(Multiple pipelines?)](https://discuss.elastic.co/t/more-input-based-on-first-input-multiple-pipelines/215442)
**Author:** [@\_Louw](https://discuss.elastic.co/u/_Louw)\ **Replies:** 7\ **Last updated:** [January 20, 2020, 11:05am UTC](https://discuss.elastic.co/t/more-input-based-on-first-input-multiple-pipelines/215442 "2020-01-20T11:05:03Z")
I am currently looking to build a solution, where I can load input based on some other input. So my setup is that I have two inputs: The data I need The timestamp of the latest update of this data I only want to get t… --- ## [CSV filter plugin and changing column headers](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493)
**Author:** [@mhyager](https://discuss.elastic.co/u/mhyager)\ **Replies:** 2\ **Last updated:** [January 20, 2020, 9:40am UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493 "2020-01-20T09:40:29Z")
I have a set of CSV files which have similar but not identical headers. I import these using the automatic header detection and autdetect numerics to set correct data types. Depending on the process that generates them,… --- ## [Logstash Failed to Execute](https://discuss.elastic.co/t/logstash-failed-to-execute/215568)
**Author:** [@Srinivas\_Kondapally](https://discuss.elastic.co/u/Srinivas_Kondapally)\ **Replies:** 1\ **Last updated:** [January 20, 2020, 9:32am UTC](https://discuss.elastic.co/t/logstash-failed-to-execute/215568 "2020-01-20T09:32:16Z")
Hello, I am new to Logstash, Just now download logstash and tried to run the command. logstash -e 'input{stdin{}}output{stdout{}}'. I got an error Sending Logstash logs to C:/Users/adity/Desktop/Elastic\_Search/logstas… --- ## [XML response from unstructured log](https://discuss.elastic.co/t/xml-response-from-unstructured-log/213916)
**Author:** [@Kishorekolla](https://discuss.elastic.co/u/Kishorekolla)\ **Replies:** 6\ **Last updated:** [January 20, 2020, 7:25am UTC](https://discuss.elastic.co/t/xml-response-from-unstructured-log/213916 "2020-01-20T07:25:07Z")
Hi Team, I'm a beginner to logstash. Till now worked on configuring basic Input, Filebeats and Output plugins by following product documents which are available in the official website. Now I have a real time scenario … --- ## [HOW TO logstash convert all fields to integer(or some fields)](https://discuss.elastic.co/t/how-to-logstash-convert-all-fields-to-integer-or-some-fields/215660)
**Author:** [@esongi](https://discuss.elastic.co/u/esongi)\ **Replies:** 2\ **Last updated:** [January 20, 2020, 6:09am UTC](https://discuss.elastic.co/t/how-to-logstash-convert-all-fields-to-integer-or-some-fields/215660 "2020-01-20T06:09:25Z")
Hello I am a foreign developer. Even if the translation is awkward, please understand My process is Elastic \>\> logstash \>\> fileBeat \>\> Kibana Logstash reads and uses csv file. The flow creates a csv file containing the… --- ## [Optimizing Datefield processing by moving from date{} to ruby code](https://discuss.elastic.co/t/optimizing-datefield-processing-by-moving-from-date-to-ruby-code/215665)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 0\ **Last updated:** [January 20, 2020, 5:31am UTC](https://discuss.elastic.co/t/optimizing-datefield-processing-by-moving-from-date-to-ruby-code/215665 "2020-01-20T05:31:05Z")
Hello, Currently in my pipeline I'm parsing out loglines, each log line begins with a timestamp such as: "2020-01-19T00:00:08.127+0800" to a field called logtimestamp, and then later in my pipeline update this to the @t… --- ## [Manipulation with Logstash](https://discuss.elastic.co/t/manipulation-with-logstash/215308)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 12\ **Last updated:** [January 20, 2020, 5:06am UTC](https://discuss.elastic.co/t/manipulation-with-logstash/215308 "2020-01-20T05:06:29Z")
Hi All, I am new to Logstash manipulations and I have no idea how to do the below. I have a sample data as below: Column:Type Incident Response P3 Incident Resolution L1.5 P2 ... The column data is… --- ## [Logstash timezone while creating index](https://discuss.elastic.co/t/logstash-timezone-while-creating-index/215623)
**Author:** [@bdn](https://discuss.elastic.co/u/bdn)\ **Replies:** 1\ **Last updated:** [January 19, 2020, 2:46pm UTC](https://discuss.elastic.co/t/logstash-timezone-while-creating-index/215623 "2020-01-19T14:46:08Z")
As I found logstash create an Index on UTC timezone. How can I create an index on the local timezone? I have set up an ES client that needs to query the index day-wise in local time. output { elasticsearch { hosts … --- ## [Logstash - Losing packets](https://discuss.elastic.co/t/logstash-losing-packets/215084)
**Author:** [@ntran](https://discuss.elastic.co/u/ntran)\ **Replies:** 1\ **Last updated:** [January 19, 2020, 12:29pm UTC](https://discuss.elastic.co/t/logstash-losing-packets/215084 "2020-01-19T12:29:36Z")
HI all, I am ingesting syslogs through logstash and usually I have roughly 9k e/s. However, recently, I have seen it drop down to 5k e/s and fluctuate between 5k e/s - 9k e/s. Can someone help me diagnose why this is h… --- ## [Exception in executing the jdbc query](https://discuss.elastic.co/t/exception-in-executing-the-jdbc-query/215564)
**Author:** [@jagan](https://discuss.elastic.co/u/jagan)\ **Replies:** 3\ **Last updated:** [January 19, 2020, 2:20am UTC](https://discuss.elastic.co/t/exception-in-executing-the-jdbc-query/215564 "2020-01-19T02:20:26Z")
Hi guru's, Please help me to solve the issue the as i am facing the exception in executing the jdbc query: The error is as follows: \[WARN \]\[logstash.inputs.jdbc \] Exception when executing JDBC query {:exception=\>#\ **Author:** [@droidus](https://discuss.elastic.co/u/droidus)\ **Replies:** 4\ **Last updated:** [January 19, 2020, 12:29am UTC](https://discuss.elastic.co/t/logstash-installation-on-aws-ubuntu-instance/213761 "2020-01-19T00:29:50Z") I am trying to get ElasticSearch setup on my Ubuntu server 18 on AWS. logstash seems to be installed: "logstash/stable,now 1:7.5.1-1 all \[installed\]" But when I check the config of logstash, it seems like logstash is n… --- ## [Logstash Output wrong - Error in visualization \[esaggs\] \> "field" is a required parameter](https://discuss.elastic.co/t/logstash-output-wrong-error-in-visualization-esaggs-field-is-a-required-parameter/215412)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 3\ **Last updated:** [January 18, 2020, 2:56pm UTC](https://discuss.elastic.co/t/logstash-output-wrong-error-in-visualization-esaggs-field-is-a-required-parameter/215412 "2020-01-18T14:56:48Z")
Hi, I recieve in Kibana the following error: "\[esaggs\] \> Saved "field" parameter is now invalid. Please select a new field." In logstash log i see also: \[2020-01-17T06:10:41,665\]\[WARN \]\[logstash.outputs.elasticsearc… --- ## [Json data parsed from logstash http poller plugin cannot be queried in kibana maps](https://discuss.elastic.co/t/json-data-parsed-from-logstash-http-poller-plugin-cannot-be-queried-in-kibana-maps/215559)
**Author:** [@durgesh](https://discuss.elastic.co/u/durgesh)\ **Replies:** 0\ **Last updated:** [January 18, 2020, 12:35pm UTC](https://discuss.elastic.co/t/json-data-parsed-from-logstash-http-poller-plugin-cannot-be-queried-in-kibana-maps/215559 "2020-01-18T12:35:33Z")
input { http\_poller { urls =\> { test =\> { url =\> "http://myurl:80/test" headers =\> { Accept =\> "application/json" } } } schedule =\> { cron =\> "\* \* \* \* \* UTC"} codec =\> "json" } } filter { split { field =\> … --- ## [Adding characters to a field](https://discuss.elastic.co/t/adding-characters-to-a-field/215513)
**Author:** [@megajune](https://discuss.elastic.co/u/megajune)\ **Replies:** 1\ **Last updated:** [January 17, 2020, 10:52pm UTC](https://discuss.elastic.co/t/adding-characters-to-a-field/215513 "2020-01-17T22:52:52Z")
I have this value in a field: 881908396bfd I want to format it like this: 88:19:08:39:6b:fd What's the "best" way to do that? --- ## [Splitting a string to json like results](https://discuss.elastic.co/t/splitting-a-string-to-json-like-results/215503)
**Author:** [@Newtoelastic](https://discuss.elastic.co/u/Newtoelastic)\ **Replies:** 10\ **Last updated:** [January 17, 2020, 8:07pm UTC](https://discuss.elastic.co/t/splitting-a-string-to-json-like-results/215503 "2020-01-17T20:07:03Z")
Hey there, I have my own logs coming into the logstash that are being created from my python script. The format for the logs is: %{DATA:Sender}|%{DATA:Recipient}|%{DATA:Subject}|(%{DATA:Links})?$ So far so good. Proble… --- ## [Couchdb\_changes gives Invalid FieldReference error when parsing couchdb which hosts couchapp](https://discuss.elastic.co/t/couchdb-changes-gives-invalid-fieldreference-error-when-parsing-couchdb-which-hosts-couchapp/214685)
**Author:** [@Raymond\_Tsang](https://discuss.elastic.co/u/Raymond_Tsang)\ **Replies:** 11\ **Last updated:** [January 17, 2020, 4:59pm UTC](https://discuss.elastic.co/t/couchdb-changes-gives-invalid-fieldreference-error-when-parsing-couchdb-which-hosts-couchapp/214685 "2020-01-17T16:59:58Z")
Here is the log: Jan 10 16:45:25 Andromeda logstash\[12489\]: \[2020-01-10T16:45:25,081\]\[INFO \]\[logstash.inputs.couchdbchanges\]\[main\] Connecting to CouchDB \_changes stream at: {:host=\>"localhost", :port=\>"5984", :db=\>"mate… --- ## [Grok positional structure problem](https://discuss.elastic.co/t/grok-positional-structure-problem/215475)
**Author:** [@Ahmad\_Moh\_d\_El\_Hasan](https://discuss.elastic.co/u/Ahmad_Moh_d_El_Hasan)\ **Replies:** 4\ **Last updated:** [January 17, 2020, 4:25pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475 "2020-01-17T16:25:18Z")
Dear sirs I have a question about the grok plugin: I have some logs which are built in the following way and my problem is that the order of my fields can mutate from log to log (in the example here I should extract f… --- ## [Logstash Input From Elastic HTTPS](https://discuss.elastic.co/t/logstash-input-from-elastic-https/215313)
**Author:** [@reyhanadp](https://discuss.elastic.co/u/reyhanadp)\ **Replies:** 1\ **Last updated:** [January 17, 2020, 4:05pm UTC](https://discuss.elastic.co/t/logstash-input-from-elastic-https/215313 "2020-01-17T16:05:18Z")
hi I want to retrieve data from an elasticsearch server that uses https. logstash config that I use to retrieve data from elasticsearch as follows : input { elasticsearch { hosts =\> \["https://RVRPAOR1ABC19WD:9200"… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=416) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=418)