# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=420 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 421 --- ## [Stdout codec json\_lines is empty](https://discuss.elastic.co/t/stdout-codec-json-lines-is-empty/214834)
**Author:** [@Kostis95](https://discuss.elastic.co/u/Kostis95)\ **Replies:** 1\ **Last updated:** [January 14, 2020, 2:57pm UTC](https://discuss.elastic.co/t/stdout-codec-json-lines-is-empty/214834 "2020-01-14T14:57:15Z")
Hi there, I'm still trying to connect my postgresql database to elastic. I've got all the required files: logstash.yml, pipelines.yml and p1.conf (in my local/etc/logstash). However I'm only getting Successfully starte… --- ## [Extracting multiline xml fields from a log](https://discuss.elastic.co/t/extracting-multiline-xml-fields-from-a-log/214954)
**Author:** [@angelcab](https://discuss.elastic.co/u/angelcab)\ **Replies:** 1\ **Last updated:** [January 14, 2020, 2:34pm UTC](https://discuss.elastic.co/t/extracting-multiline-xml-fields-from-a-log/214954 "2020-01-14T14:34:40Z")
Hello I'm struggling with a problem trying to extract indepent fields from a log which contains a XML embebbed. I'm using logstash for the ingesting into elasticsearch. First, I collect the logs with filebeat and send … --- ## [Fluent codec and multline](https://discuss.elastic.co/t/fluent-codec-and-multline/212504)
**Author:** [@bquevat](https://discuss.elastic.co/u/bquevat)\ **Replies:** 2\ **Last updated:** [January 14, 2020, 1:52pm UTC](https://discuss.elastic.co/t/fluent-codec-and-multline/212504 "2020-01-14T13:52:06Z")
Hello, I use the fluent codec in my input. It receives logs from Java applications and I would like to make Java stack trace into a single event. This can be done with the multiline plugin. How can I benefit from the… --- ## [Created nested fields from Xpath & check for existing documents](https://discuss.elastic.co/t/created-nested-fields-from-xpath-check-for-existing-documents/214414)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 2\ **Last updated:** [January 14, 2020, 11:46am UTC](https://discuss.elastic.co/t/created-nested-fields-from-xpath-check-for-existing-documents/214414 "2020-01-14T11:46:59Z")
I have two questions; parsing xml data & adding it to an array in a record in an index checking for an existing record in an index and if it exists add the new data of that record to the array of the existing record I… --- ## [Parsing json file with logstash](https://discuss.elastic.co/t/parsing-json-file-with-logstash/214080)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 11\ **Last updated:** [January 14, 2020, 10:27am UTC](https://discuss.elastic.co/t/parsing-json-file-with-logstash/214080 "2020-01-14T10:27:15Z")
Happy new year everyone! hoping someone can shed some light on this, i have a weird issue i cant set right with parsing a json file. This is the source json file { "SHA256": "766be5c99ba674f985ce844add4bc5ec423e90811f… --- ## [Logstash - csv with a json column - Add a new nested field from the existing field if it is integer](https://discuss.elastic.co/t/logstash-csv-with-a-json-column-add-a-new-nested-field-from-the-existing-field-if-it-is-integer/214967)
**Author:** [@Lakshmi\_Narasimhan\_V](https://discuss.elastic.co/u/Lakshmi_Narasimhan_V)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 9:32am UTC](https://discuss.elastic.co/t/logstash-csv-with-a-json-column-add-a-new-nested-field-from-the-existing-field-if-it-is-integer/214967 "2020-01-14T09:32:44Z")
Below are the details: - CSV File, visitid,labreport, 11111,"\[{""test"":""BUN"",""value"":""0.5""},{""test"":""Creatinie"",""value"":""0.8""},{""test"":""Potassium"",""value"":""3.5""}\]", 22222,"\[{""test"":""BUN"",""v… --- ## [Logstash nested data update is not working as expected](https://discuss.elastic.co/t/logstash-nested-data-update-is-not-working-as-expected/214965)
**Author:** [@Ram\_29](https://discuss.elastic.co/u/Ram_29)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 9:27am UTC](https://discuss.elastic.co/t/logstash-nested-data-update-is-not-working-as-expected/214965 "2020-01-14T09:27:17Z")
Hi I'm having a nested field as below "appdata": { "type":"nested", "include\_in\_parent":true, "properties": { "accessType": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignor… --- ## [I cant set values in new fields](https://discuss.elastic.co/t/i-cant-set-values-in-new-fields/214930)
**Author:** [@Jonny3](https://discuss.elastic.co/u/Jonny3)\ **Replies:** 1\ **Last updated:** [January 14, 2020, 8:32am UTC](https://discuss.elastic.co/t/i-cant-set-values-in-new-fields/214930 "2020-01-14T08:32:16Z")
I use grok to differentiate the content in each line of the log and mutate to create a new field and be able to assign those values ​​but it doesn't work, write the configuration content literally. I don't have compilin… --- ## [Streaming events from Elastic](https://discuss.elastic.co/t/streaming-events-from-elastic/214923)
**Author:** [@Olatunde\_Tokun](https://discuss.elastic.co/u/Olatunde_Tokun)\ **Replies:** 0\ **Last updated:** [January 14, 2020, 2:46am UTC](https://discuss.elastic.co/t/streaming-events-from-elastic/214923 "2020-01-14T02:46:06Z")
I currently use logstash to stream logs from Elasticsearch to an syslog connector on a schedule (say 5mins). Every 5 mins logstash runs and grabs events in the index then fires them off to the syslog output. Input (Elas… --- ## [Parsing DNS Logs from GCP Cloud](https://discuss.elastic.co/t/parsing-dns-logs-from-gcp-cloud/214911)
**Author:** [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 11:32pm UTC](https://discuss.elastic.co/t/parsing-dns-logs-from-gcp-cloud/214911 "2020-01-13T23:32:33Z")
Is there a parser for the DNS logs in Logstash? I've noticed some of my logs are not being parsed, the ones with multiple lines. This shows up as a single field. How can I parse it? jsonPayload.rdata google.com. 299 IN … --- ## [Extracting parts of different timestamp fields](https://discuss.elastic.co/t/extracting-parts-of-different-timestamp-fields/214915)
**Author:** [@Peter\_Li](https://discuss.elastic.co/u/Peter_Li)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 11:30pm UTC](https://discuss.elastic.co/t/extracting-parts-of-different-timestamp-fields/214915 "2020-01-13T23:30:10Z")
I have many timestamp fields: AdmitDateTime DCDateTime etc. I want to pull out the yyyy, MM, dd parts individually from them, eg copy =\> { "AdmitDateTime" =\> "@timestamp" } add\_field =\> { AdmitYear =\> "%{+yyyy}" … --- ## [Drop filter is not working for certain string](https://discuss.elastic.co/t/drop-filter-is-not-working-for-certain-string/214901)
**Author:** [@hadars](https://discuss.elastic.co/u/hadars)\ **Replies:** 3\ **Last updated:** [January 13, 2020, 11:29pm UTC](https://discuss.elastic.co/t/drop-filter-is-not-working-for-certain-string/214901 "2020-01-13T23:29:09Z")
Hi, I am trying to apply filter only if message has certain string. For example my message has "skipping L1 entity store" and my config looks like, filter { if \[msg\] == "skipping L1 entity store" { dro… --- ## [Open source parsers](https://discuss.elastic.co/t/open-source-parsers/214914)
**Author:** [@Ramicoh](https://discuss.elastic.co/u/Ramicoh)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 11:02pm UTC](https://discuss.elastic.co/t/open-source-parsers/214914 "2020-01-13T23:02:25Z")
Hey, We created some open source parsers for Logstash, customized for some common software products (Symantec, CarbonBlack etc.): https://github.com/empow/logstash-parsers/ I would love to hear your opinions - how usef… --- ## [Multiple Grok patterns or can I use Regex?](https://discuss.elastic.co/t/multiple-grok-patterns-or-can-i-use-regex/214910)
**Author:** [@IshanP1](https://discuss.elastic.co/u/IshanP1)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 10:37pm UTC](https://discuss.elastic.co/t/multiple-grok-patterns-or-can-i-use-regex/214910 "2020-01-13T22:37:29Z")
I am making a centralised syslogging server. Do i have to make a different grok filter for every single different format of syslog or can I use something like regex to pick out certain elements of a log? Dec 16 15:01:13… --- ## [Timestamp explanation](https://discuss.elastic.co/t/timestamp-explanation/214749)
**Author:** [@amhulli](https://discuss.elastic.co/u/amhulli)\ **Replies:** 2\ **Last updated:** [January 13, 2020, 10:07pm UTC](https://discuss.elastic.co/t/timestamp-explanation/214749 "2020-01-13T22:07:23Z")
Hi all, I'm using ELK 7.4.2 I have an issue where Kibana throws up an error when I click on any of the bars on the graph in the discover tab (it complains about: Request to Elasticsearch failed: {"error":{"root\_cause"… --- ## [How to send data from HTTP input to ElasticSearch using Logstash and jdbc\_streaming filter?](https://discuss.elastic.co/t/how-to-send-data-from-http-input-to-elasticsearch-using-logstash-and-jdbc-streaming-filter/214898)
**Author:** [@aliabbasifard](https://discuss.elastic.co/u/aliabbasifard)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 9:52pm UTC](https://discuss.elastic.co/t/how-to-send-data-from-http-input-to-elasticsearch-using-logstash-and-jdbc-streaming-filter/214898 "2020-01-13T21:52:06Z")
I want to send data from Http to elasticsearch using logstash and I want to enrich my data using jdbc\_streaming filter plugin. This is my logstash config: input { http { id =\> "sensor\_data\_http\_input" use… --- ## [Logstash JDBC SSL Error](https://discuss.elastic.co/t/logstash-jdbc-ssl-error/209631)
**Author:** [@newtologstash](https://discuss.elastic.co/u/newtologstash)\ **Replies:** 9\ **Last updated:** [January 13, 2020, 8:54pm UTC](https://discuss.elastic.co/t/logstash-jdbc-ssl-error/209631 "2020-01-13T20:54:42Z")
Hi, I'm trying to use jdbc-input plugin to pull the data from oracle. It is working fine when using TCP connection string, but getting below error when trying to connect using TCP"S" connection (SSL). \[ERROR\]\[logstash.… --- ## [Parsing EventData field correctly from json input](https://discuss.elastic.co/t/parsing-eventdata-field-correctly-from-json-input/214891)
**Author:** [@bensonmp](https://discuss.elastic.co/u/bensonmp)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 7:29pm UTC](https://discuss.elastic.co/t/parsing-eventdata-field-correctly-from-json-input/214891 "2020-01-13T19:29:47Z")
Hi, I'm new to using ELK. I'm looking for some help to properly parse the EventData field from my windows logs. Here's an example event: \ \ \<… --- ## [Append to a property if exists](https://discuss.elastic.co/t/append-to-a-property-if-exists/214889)
**Author:** [@Sbezgoan](https://discuss.elastic.co/u/Sbezgoan)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 6:37pm UTC](https://discuss.elastic.co/t/append-to-a-property-if-exists/214889 "2020-01-13T18:37:56Z")
Hello, Input csv looks like : "","Code","Date","Time","Open","High","Low","Close","Volume" "1","3IINFOTECH",20150703,"09:16:00",5.55,4.55,4.55,4.55,835 "2","3IINFOTECH",20150703,"09:17:00",5.55,4.55,4.55,4.55,390 L… --- ## [Filebeat / Logstash pipeline exception parsing csv](https://discuss.elastic.co/t/filebeat-logstash-pipeline-exception-parsing-csv/212896)
**Author:** [@galexander87](https://discuss.elastic.co/u/galexander87)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-logstash-pipeline-exception-parsing-csv/212896 "2020-01-13T15:14:31Z")
I am trying to send some reports in CSV format to logstash and then Elasticsearch. I am getting this pipeline error: \[WARN \] \[io.netty.channel.DefaultChannelPipeline\] An exceptionCaught() event was fired, and it reache… --- ## [Extracting the last word of an event/line (Grok)](https://discuss.elastic.co/t/extracting-the-last-word-of-an-event-line-grok/214774)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 4\ **Last updated:** [January 13, 2020, 3:04pm UTC](https://discuss.elastic.co/t/extracting-the-last-word-of-an-event-line-grok/214774 "2020-01-13T15:04:11Z")
Hello all, i am looking for a Grok pattern to retrieve the last word a matching line with literal. input (file content) THISLINE 1,011.53EUR 12,815.50EUR 2,940.17 100,011.53EUR filter grok{ match =\> { … --- ## [Import general configuration options](https://discuss.elastic.co/t/import-general-configuration-options/214846)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 1:50pm UTC](https://discuss.elastic.co/t/import-general-configuration-options/214846 "2020-01-13T13:50:47Z")
Hi, Is there a way to import some options (mostly for the input and the output part) from a file to keep the different configuration files more transparent and occasionally the changes easier? Thanks! --- ## [How can i setting that](https://discuss.elastic.co/t/how-can-i-setting-that/214758)
**Author:** [@pageno11](https://discuss.elastic.co/u/pageno11)\ **Replies:** 1\ **Last updated:** [January 13, 2020, 1:11pm UTC](https://discuss.elastic.co/t/how-can-i-setting-that/214758 "2020-01-13T13:11:45Z")
How can i setting that Server 1 Input milti files (differnt path) Output logstash tcp Server 2 Input logstash tcp Output also multi files (diffrent path) Only use logstash but in/out is diffrent server. Also, Can… --- ## [Mutiple filebeat servers to Single logstash server](https://discuss.elastic.co/t/mutiple-filebeat-servers-to-single-logstash-server/214725)
**Author:** [@pathfinder225](https://discuss.elastic.co/u/pathfinder225)\ **Replies:** 3\ **Last updated:** [January 13, 2020, 12:51pm UTC](https://discuss.elastic.co/t/mutiple-filebeat-servers-to-single-logstash-server/214725 "2020-01-13T12:51:58Z")
Hi, New to ELK and I am doing a production setup . I have 3 servers and installed filebeat on each of them. now i am sending these 3 filebeat inputs to single logstash using below config. In all 3 servers Filebeat ou… --- ## [Successfullystarted logstash but it cant read my file even configuration is done without error msg](https://discuss.elastic.co/t/successfullystarted-logstash-but-it-cant-read-my-file-even-configuration-is-done-without-error-msg/214731)
**Author:** [@saravana\_hariharan](https://discuss.elastic.co/u/saravana_hariharan)\ **Replies:** 3\ **Last updated:** [January 13, 2020, 12:51pm UTC](https://discuss.elastic.co/t/successfullystarted-logstash-but-it-cant-read-my-file-even-configuration-is-done-without-error-msg/214731 "2020-01-13T12:51:22Z")
My config file input { file { path =\>"C:\\Users\\pgt0005\\Desktop\\data\\100-contacts.csv" start\_position =\> "beginning" } } filter { csv{ separator=\>"," columns=\>\["first\_name","last\_name","company\_name","address","… --- ## [How to fetch an attribute and create a field with value](https://discuss.elastic.co/t/how-to-fetch-an-attribute-and-create-a-field-with-value/214821)
**Author:** [@Rajesh\_M](https://discuss.elastic.co/u/Rajesh_M)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 11:50am UTC](https://discuss.elastic.co/t/how-to-fetch-an-attribute-and-create-a-field-with-value/214821 "2020-01-13T11:50:00Z")
Hi, I am parcing a log file and generating the report. The report contains the user and few other fields related to application. I need to find an attribute of the user fetched from log and create a new field with the v… --- ## [Logstash JDBC Connection string multiple entries in Environment variable](https://discuss.elastic.co/t/logstash-jdbc-connection-string-multiple-entries-in-environment-variable/214801)
**Author:** [@JP\_Smith](https://discuss.elastic.co/u/JP_Smith)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 10:47am UTC](https://discuss.elastic.co/t/logstash-jdbc-connection-string-multiple-entries-in-environment-variable/214801 "2020-01-13T10:47:07Z")
I want to add multiple entries in my environment variable and then pass them into my logstash jdbc connection string. When I do that it reads all the values in my variable at once and then fails with my connection attemp… --- ## [How to set multiple hosts in elasticsearch-output via environment variable?](https://discuss.elastic.co/t/how-to-set-multiple-hosts-in-elasticsearch-output-via-environment-variable/214802)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 10:47am UTC](https://discuss.elastic.co/t/how-to-set-multiple-hosts-in-elasticsearch-output-via-environment-variable/214802 "2020-01-13T10:47:48Z")
Hi, I am trying to configure elasticsearch hosts via environment variable to be used in elasticsearch-output plugin. I want to set them as array for loadbalancing in the multi node cluster without the need for an extern… --- ## [How to install mutate filter in logstash?](https://discuss.elastic.co/t/how-to-install-mutate-filter-in-logstash/214616)
**Author:** [@akuninja](https://discuss.elastic.co/u/akuninja)\ **Replies:** 2\ **Last updated:** [January 13, 2020, 10:46am UTC](https://discuss.elastic.co/t/how-to-install-mutate-filter-in-logstash/214616 "2020-01-13T10:46:21Z")
Hi, I have that plugin from gitHub and I'm trying to follow the instructions in README, since I do not use Ruby, mine are (I believe) limited to: Install plugin # Logstash 2.3 and higher bin/logstash-plugin install --… --- ## [Filter only on the current date on my sql](https://discuss.elastic.co/t/filter-only-on-the-current-date-on-my-sql/214795)
**Author:** [@Haalanam](https://discuss.elastic.co/u/Haalanam)\ **Replies:** 0\ **Last updated:** [January 13, 2020, 10:29am UTC](https://discuss.elastic.co/t/filter-only-on-the-current-date-on-my-sql/214795 "2020-01-13T10:29:15Z")
Hi, I want to make a stop criteria on my logstash loads. On the sql statement i want Something like select \* from mytable where insert\_date := current\_date Will that work ? Thanks [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=419) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=421)