# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=233 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 234 --- ## [IP based failover in Logstash](https://discuss.elastic.co/t/ip-based-failover-in-logstash/271978)
**Author:** [@Debarati\_Goswami](https://discuss.elastic.co/u/Debarati_Goswami)\ **Replies:** 3\ **Last updated:** [May 3, 2021, 5:49pm UTC](https://discuss.elastic.co/t/ip-based-failover-in-logstash/271978 "2021-05-03T17:49:29Z")
Hi All , I have a requirement to collect logs from various sources into Logstash cluster (of 6 nodes) of which 4 nodes are for 4 different types of logs (rsyslog , IPFIX , SFTP and FluentD) and the last two nodes would … --- ## [MongoDB Input threw an exception, restarting {:exception=\>#\}](https://discuss.elastic.co/t/mongodb-input-threw-an-exception-restarting-exception-bson-2221275031010008292cainvoicecceprimary-is-an-invalid-objectid/271932)
**Author:** [@BASHEER\_DS](https://discuss.elastic.co/u/BASHEER_DS)\ **Replies:** 3\ **Last updated:** [May 3, 2021, 4:10pm UTC](https://discuss.elastic.co/t/mongodb-input-threw-an-exception-restarting-exception-bson-2221275031010008292cainvoicecceprimary-is-an-invalid-objectid/271932 "2021-05-03T16:10:41Z")
Hi, I am getting below error while creating pipeline. please help in resolving this error. MongoDB Input threw an exception, restarting {:exception=\>#\ **Author:** [@Rizky\_Hudha](https://discuss.elastic.co/u/Rizky_Hudha)\ **Replies:** 1\ **Last updated:** [May 3, 2021, 3:10pm UTC](https://discuss.elastic.co/t/how-to-modified-configure-logstash-add-geo-point-and-load-data-with-logstash-to-elasticsearch/271984 "2021-05-03T15:10:03Z") I have a column device.latlong where the longtitude and latitude values ​​become one column, how to add a geo\_point file to the logstash configuration, and display it in the kibana visualization and this my configure ap… --- ## [Can we have multiple HTTP input plugin configured into single logstash.conf file like below](https://discuss.elastic.co/t/can-we-have-multiple-http-input-plugin-configured-into-single-logstash-conf-file-like-below/271950)
**Author:** [@Learn\_Mulesoft\_With](https://discuss.elastic.co/u/Learn_Mulesoft_With)\ **Replies:** 1\ **Last updated:** [May 3, 2021, 2:50pm UTC](https://discuss.elastic.co/t/can-we-have-multiple-http-input-plugin-configured-into-single-logstash-conf-file-like-below/271950 "2021-05-03T14:50:50Z")
input { http { host =\> "localhost" port =\> "5044" response\_code =\> 201 } http { host =\> "localhost" port =\> "5045" response\_code =\> 201 } http { host =\> "localhost" port =\> "5046" response\_code =\> 201 } } … --- ## [Geo point for new ECS mapped Geo fields using JDBC](https://discuss.elastic.co/t/geo-point-for-new-ecs-mapped-geo-fields-using-jdbc/271846)
**Author:** [@metalshanked](https://discuss.elastic.co/u/metalshanked)\ **Replies:** 6\ **Last updated:** [May 3, 2021, 2:07pm UTC](https://discuss.elastic.co/t/geo-point-for-new-ecs-mapped-geo-fields-using-jdbc/271846 "2021-05-03T14:07:33Z")
Hello, I am trying to map Geo Lat and Long for internal IPs to ECS geo fields such as \[source\]\[geo\]\[location\] using logstash. The standard geoip filter works well for external ips and i see it nicely mapped to the ecs … --- ## [Logstash / Protobuf plugin, removing oneOf fields that aren't set (are null)](https://discuss.elastic.co/t/logstash-protobuf-plugin-removing-oneof-fields-that-arent-set-are-null/271994)
**Author:** [@SCollins](https://discuss.elastic.co/u/SCollins)\ **Replies:** 0\ **Last updated:** [May 3, 2021, 1:48pm UTC](https://discuss.elastic.co/t/logstash-protobuf-plugin-removing-oneof-fields-that-arent-set-are-null/271994 "2021-05-03T13:48:43Z")
I'm using the logstash protobuf plugin as a mechanism to read protobuf from kafka, convert it from protobuf and write it to elasticsearch. The problem we are having is, the oneOf fields, despite being null/not set in the… --- ## ["Pipeline does not exist" issue with sonicwall filebeat module](https://discuss.elastic.co/t/pipeline-does-not-exist-issue-with-sonicwall-filebeat-module/271067)
**Author:** [@Tony51](https://discuss.elastic.co/u/Tony51)\ **Replies:** 1\ **Last updated:** [May 3, 2021, 12:41pm UTC](https://discuss.elastic.co/t/pipeline-does-not-exist-issue-with-sonicwall-filebeat-module/271067 "2021-05-03T12:41:21Z")
Hi, I'm sending logs from filebeat to logstash but it I'm getting this error from logstash connecting to elastisearch: \[2021-04-20T10:50:47,200\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] 673b0cb9e015302058e251f502b8… --- ## [The JSON filter can extract empty field names, which are then unusable by Logstash](https://discuss.elastic.co/t/the-json-filter-can-extract-empty-field-names-which-are-then-unusable-by-logstash/271963)
**Author:** [@nico127](https://discuss.elastic.co/u/nico127)\ **Replies:** 0\ **Last updated:** [May 3, 2021, 10:45am UTC](https://discuss.elastic.co/t/the-json-filter-can-extract-empty-field-names-which-are-then-unusable-by-logstash/271963 "2021-05-03T10:45:00Z")
Hi, I've found a situation where the JSON filter and valid JSON can cause unusable Logstash events. It comes from 2 facts: An empty string is a valid JSON key, as explained for example in this stackoverflow answer. It … --- ## [What is the recommend memory size to run logstash?](https://discuss.elastic.co/t/what-is-the-recommend-memory-size-to-run-logstash/271083)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 13\ **Last updated:** [May 3, 2021, 8:47am UTC](https://discuss.elastic.co/t/what-is-the-recommend-memory-size-to-run-logstash/271083 "2021-05-03T08:47:46Z")
Hi all, Please explain What is the recommend memory size to run logstash. Thanks --- ## [A plugin had an unrecoverable error. Will restart this plugin](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/271946)
**Author:** [@Safir](https://discuss.elastic.co/u/Safir)\ **Replies:** 0\ **Last updated:** [May 3, 2021, 8:37am UTC](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/271946 "2021-05-03T08:37:09Z")
/Users/lab/Desktop/logstash/logstash-core/lib/logstash/java\_pipeline.rb:396:in block in start\_input' \[2021-05-03T11:31:51,605\]\[ERROR\]\[logstash.javapipeline \]\[main\]\[6a3d8ecc5c72c85e305635d46ccbd177399eb255a591231ec278b… --- ## [Could not index event to Elasticsearch](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/271929)
**Author:** [@BASHEER\_DS](https://discuss.elastic.co/u/BASHEER_DS)\ **Replies:** 1\ **Last updated:** [May 3, 2021, 7:15am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/271929 "2021-05-03T07:15:44Z")
Hi, I am trying to index my data into Elasticsearch from logstash but I keep getting this type of errors for all indices. Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"… --- ## [Output values into a Json dictionary](https://discuss.elastic.co/t/output-values-into-a-json-dictionary/271919)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [May 3, 2021, 12:36am UTC](https://discuss.elastic.co/t/output-values-into-a-json-dictionary/271919 "2021-05-03T00:36:26Z")
I get the following values from a database: 1234 , first 1, 123 ,third and I want to output them as a json dictionary to a file "1234" =\> {"key1" =\> "first 1" "key2" =\> 123 "key3" =\> "third 1"} i can do this with cod… --- ## [Logstash Error - elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=\>400, :url=\>"http://elastichost:9200/\_bulk"}](https://discuss.elastic.co/t/logstash-error-elasticsearch-encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400-url-http-elastichost-9200-bulk/271854)
**Author:** [@PavanB](https://discuss.elastic.co/u/PavanB)\ **Replies:** 1\ **Last updated:** [May 2, 2021, 9:49pm UTC](https://discuss.elastic.co/t/logstash-error-elasticsearch-encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400-url-http-elastichost-9200-bulk/271854 "2021-05-02T21:49:45Z")
Hi Team, Please help for the below error. This is happening recently as Logstash is not able to ingest data to Elasticsearch. elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=\>4… --- ## [Aggregate Filter mapping](https://discuss.elastic.co/t/aggregate-filter-mapping/271723)
**Author:** [@StashLogs](https://discuss.elastic.co/u/StashLogs)\ **Replies:** 2\ **Last updated:** [May 2, 2021, 8:35pm UTC](https://discuss.elastic.co/t/aggregate-filter-mapping/271723 "2021-05-02T20:35:50Z")
Hello everyone! I am trying to understand this behaviour with the map field syntax in the aggregate filter. In the code below, if I keep the field name as 'syntaxOddity', no data is copied over. But if I change the fie… --- ## [Logstash type error conversion geo\_point type field must be a number](https://discuss.elastic.co/t/logstash-type-error-conversion-geo-point-type-field-must-be-a-number/271914)
**Author:** [@luis\_vaglian](https://discuss.elastic.co/u/luis_vaglian)\ **Replies:** 0\ **Last updated:** [May 2, 2021, 6:11pm UTC](https://discuss.elastic.co/t/logstash-type-error-conversion-geo-point-type-field-must-be-a-number/271914 "2021-05-02T18:11:06Z")
Hi, I have created my index pattern to map my csv file(geoLocation in geo\_point) ''' PUT \_template/geotemplate \*\* {\*\* \*\* "index\_patterns": \[\*\* \*\* "flightgeolocation"\*\* \*\* \],\*\* \*\* "setting… --- ## [Error while updating nested fields using Logstash mutate](https://discuss.elastic.co/t/error-while-updating-nested-fields-using-logstash-mutate/271880)
**Author:** [@sunilmpatil](https://discuss.elastic.co/u/sunilmpatil)\ **Replies:** 3\ **Last updated:** [May 2, 2021, 3:54pm UTC](https://discuss.elastic.co/t/error-while-updating-nested-fields-using-logstash-mutate/271880 "2021-05-02T15:54:40Z")
Hi All, I am trying to load from a SQL to Elastic using Logstash pipeline. I am trying to add new nested document using mutate, but it is overwriting existing one. Here is my existing JSON document: "empid" : 12345, … --- ## [Multiple csv with different columns](https://discuss.elastic.co/t/multiple-csv-with-different-columns/271794)
**Author:** [@dpb\_2000](https://discuss.elastic.co/u/dpb_2000)\ **Replies:** 3\ **Last updated:** [April 30, 2021, 5:04pm UTC](https://discuss.elastic.co/t/multiple-csv-with-different-columns/271794 "2021-04-30T17:04:57Z")
Hi, I am trying to parse multiple csv with different columns based on autodetect\_column\_names =\> true argument. The main problem is that when I upload the data the columns get repeated and only the columns of the first c… --- ## [Document deleted after logstash execution](https://discuss.elastic.co/t/document-deleted-after-logstash-execution/271774)
**Author:** [@sourabhjain104](https://discuss.elastic.co/u/sourabhjain104)\ **Replies:** 4\ **Last updated:** [April 30, 2021, 4:04pm UTC](https://discuss.elastic.co/t/document-deleted-after-logstash-execution/271774 "2021-04-30T16:04:54Z")
Hello, I am using the latest version of ELK (7.11.2) on ubuntu 18.4. I am getting data from postgre using JDBC My logstash.conf file look lile : input { jdbc { jdbc\_connection\_string =\>"jdbc:postgresql://123.0.… --- ## [JSON parser error](https://discuss.elastic.co/t/json-parser-error/271283)
**Author:** [@Pacous](https://discuss.elastic.co/u/Pacous)\ **Replies:** 5\ **Last updated:** [April 30, 2021, 3:50pm UTC](https://discuss.elastic.co/t/json-parser-error/271283 "2021-04-30T15:50:17Z")
Hello, This is my logstash configuration below: filter { grok { match =\> { "message" =\> "%{SYSLOG5424PRI:syslog\_index}-\\s\*%{SYSLOGHOST:syslog\_hostname} %{GREEDYDATA:syslog\_message}" } } json { source =\> "syslog\_me… --- ## [Filter only required logs and send to elastic search](https://discuss.elastic.co/t/filter-only-required-logs-and-send-to-elastic-search/271738)
**Author:** [@ELK\_gj](https://discuss.elastic.co/u/ELK_gj)\ **Replies:** 1\ **Last updated:** [April 30, 2021, 3:32pm UTC](https://discuss.elastic.co/t/filter-only-required-logs-and-send-to-elastic-search/271738 "2021-04-30T15:32:59Z")
Hi, My input file contains 10 lines of logs, out of which need to parse only 5 logs based on a string, say example: NAS. So, out of 10 logs only 5 logs which contains keyword "NAS" should parse and go to elasticsearch. … --- ## [Generate multiple types with different csv](https://discuss.elastic.co/t/generate-multiple-types-with-different-csv/271797)
**Author:** [@dpb\_2000](https://discuss.elastic.co/u/dpb_2000)\ **Replies:** 0\ **Last updated:** [April 30, 2021, 2:53pm UTC](https://discuss.elastic.co/t/generate-multiple-types-with-different-csv/271797 "2021-04-30T14:53:33Z")
Hi, I am trying to generate various types in the same index based on various csv. As I don´t know the amount of csv, making an input for each one would be non-viable. So does anyone know how to generate types with the n… --- ## [Logstash & data streams](https://discuss.elastic.co/t/logstash-data-streams/271787)
**Author:** [@probson](https://discuss.elastic.co/u/probson)\ **Replies:** 0\ **Last updated:** [April 30, 2021, 1:45pm UTC](https://discuss.elastic.co/t/logstash-data-streams/271787 "2021-04-30T13:45:40Z")
Hi, Is it possible to setup logstash to use data streams? I have checked the logstash \> elasticsearch output plugin and cannot find any mention of data\_stream =\> "true" as outlined below, not sure if its not needed o… --- ## [Changing the default user in logstash docker image build](https://discuss.elastic.co/t/changing-the-default-user-in-logstash-docker-image-build/271775)
**Author:** [@gorbroth](https://discuss.elastic.co/u/gorbroth)\ **Replies:** 0\ **Last updated:** [April 30, 2021, 12:57pm UTC](https://discuss.elastic.co/t/changing-the-default-user-in-logstash-docker-image-build/271775 "2021-04-30T12:57:02Z")
I've been tying to run my logstash as a docker container using a self created user on cent os 7 . But from what it looks like the image docker.elastic.co/logstash/logstash:6.4.2 creates it use Logstash user . Went throug… --- ## [Logstash Memory grows unlimited](https://discuss.elastic.co/t/logstash-memory-grows-unlimited/271757)
**Author:** [@KaustavNath](https://discuss.elastic.co/u/KaustavNath)\ **Replies:** 0\ **Last updated:** [April 30, 2021, 9:52am UTC](https://discuss.elastic.co/t/logstash-memory-grows-unlimited/271757 "2021-04-30T09:52:43Z")
Hello, I am using Logstash version 6.8.3. Recently I face issue with memory in logstash in the pipeline for beat tcp input. In the log I have below error "Caused by: java.lang.IndexOutOfBoundsException: writerIndex(214… --- ## [ECS for syslog: why Filebeat does not follow ECS naming convention](https://discuss.elastic.co/t/ecs-for-syslog-why-filebeat-does-not-follow-ecs-naming-convention/271744)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 0\ **Last updated:** [April 30, 2021, 7:19am UTC](https://discuss.elastic.co/t/ecs-for-syslog-why-filebeat-does-not-follow-ecs-naming-convention/271744 "2021-04-30T07:19:30Z")
we're indexing syslogs data via Logstash (without Filebeat). Since it's not that clear, how Logstash manages ECS schema, I checked the Filebeat mapping and realized, that it uses different naming for Syslog: system.sysl… --- ## [Logstash If statement and grok not working](https://discuss.elastic.co/t/logstash-if-statement-and-grok-not-working/271717)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 3\ **Last updated:** [April 30, 2021, 4:52am UTC](https://discuss.elastic.co/t/logstash-if-statement-and-grok-not-working/271717 "2021-04-30T04:52:30Z")
I am using following if statement, not sure what is wrong here, it is giving me \_mutate\_error. Does the / cause this error? if \[operationName\] == "MICROSOFT.RESOURCES/SUBSCRIPTIONS/RESOURCEGROUPS/DELETE" { … --- ## [Export CSV data from Elasticsearch using Logstash](https://discuss.elastic.co/t/export-csv-data-from-elasticsearch-using-logstash/271702)
**Author:** [@ice2021](https://discuss.elastic.co/u/ice2021)\ **Replies:** 2\ **Last updated:** [April 29, 2021, 9:47pm UTC](https://discuss.elastic.co/t/export-csv-data-from-elasticsearch-using-logstash/271702 "2021-04-29T21:47:55Z")
Hi, I am trying to extract pivoted data from ES using Logstash. It works on Kibana but not on Logstash. \`Error: \[400\] {"error":{"root\_cause":\[{"type":"parsing\_exception","reason":"Unknown key for a VALUE\_STRING in \[qu… --- ## [\[LOGSTASH\] How to pars logs with SSH](https://discuss.elastic.co/t/logstash-how-to-pars-logs-with-ssh/270513)
**Author:** [@Adhara](https://discuss.elastic.co/u/Adhara)\ **Replies:** 2\ **Last updated:** [April 29, 2021, 8:40pm UTC](https://discuss.elastic.co/t/logstash-how-to-pars-logs-with-ssh/270513 "2021-04-29T20:40:48Z")
Hello, I'm new in ELK and i don't understand one thing. I installed ELK with basic options. I did some small tests locally with the "File" input and it worked fine. Now, I wanted to parse the logs for a remote machine… --- ## [Conversion of timeformat](https://discuss.elastic.co/t/conversion-of-timeformat/270961)
**Author:** [@mansa](https://discuss.elastic.co/u/mansa)\ **Replies:** 5\ **Last updated:** [April 29, 2021, 5:58pm UTC](https://discuss.elastic.co/t/conversion-of-timeformat/270961 "2021-04-29T17:58:53Z")
Hi Folks we have few logs which comes into time format "2021-04-05 08:04:24+07:00" and i want to convert this time into format "Sep 15 18:58:48" because i am sending logs to different SIEM which reads only syslog time st… --- ## [Drop fields with key that contain string and value \<1](https://discuss.elastic.co/t/drop-fields-with-key-that-contain-string-and-value-1/271672)
**Author:** [@Mike\_Clarke](https://discuss.elastic.co/u/Mike_Clarke)\ **Replies:** 1\ **Last updated:** [April 29, 2021, 5:53pm UTC](https://discuss.elastic.co/t/drop-fields-with-key-that-contain-string-and-value-1/271672 "2021-04-29T17:53:02Z")
I have a document that contains multiple fields with name metrics.time.\*. I would like to be able to drop all fields from the event that have key =\> metrics.time\* and value \< 1. Can this be accomplished? I tried doin… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=232) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=234)