The content length (938946807) is bigger than the maximum allowed string (536870888) sending logstash-plain.log to elasticsearch

Elastic Stack Logstash
1

Hi I have setup filebeat to send logstash logs to logstah then elasticsearch. Getting this error when trying to access it on discover page

Search Error
The content length (938946807) is bigger than the maximum allowed string (536870888)

Error: The content length (938946807) is bigger than the maximum allowed string (536870888)
    at search_interceptor_SearchInterceptor.handleSearchError (http://43.204.205.20:5601/57136/bundles/plugin/data/kibana/data.plugin.js:1:411520)
    at http://43.204.205.20:5601/57136/bundles/plugin/data/kibana/data.plugin.js:1:414261
    at http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:699372
    at s._error (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:6:33819)
    at t.error (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:97199)
    at http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:329:45177
    at o (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:262880)
    at t.error (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:329:45044)
    at Object.error (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:445442)
    at e.error (http://43.204.205.20:5601/57136/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:98005)

- type: log

  enabled: true
  paths:
    - /var/log/logstash/*.log
  fields:
    type: logstash_syslog
  fields_under_root: true

pipeline.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "%{type}%{+YYYY.MM.dd}"
    user => "user"
    password => "pwd"
  }
}
2

I think this is error related to Kibana.

Error: The content length (938946807) is bigger than the maximum allowed string (536870888)
at search_interceptor_SearchInterceptor.handleSearchError (http://ip:5601/57136/bundles/plugin/data/kibana/data.plugin.js:1:411520)

Set server.maxPayloadBytes: 1938946807 in kibana.yml, restart kibana and test.

3

getting this after adding server.maxPayloadBytes

Nov 22 11:54:16 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:16.940+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 22 11:54:19 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:19.017+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
Nov 22 11:54:19 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:19.022+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 22 11:54:19 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:19.196+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
Nov 22 11:54:19 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:19.203+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 22 11:54:20 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:20.861+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 22 11:54:21 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:21.476+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
Nov 22 11:54:21 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:21.481+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
Nov 22 11:54:21 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:21.668+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
Nov 22 11:54:21 ip-10-0-9-223.ap-south-1.compute.internal kibana[7557]: [2022-11-22T11:54:21.676+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 10.0.9.223:9200 error
4

strange. everything worked fine till now! even if i undo the change getting this error now

[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 10.0.9.223:9200
6

i am unable to curl to localhost:9200 ip:9200? kibana works fine for sometime and then elasticsearch fails

7 
Nov 22 11:59:47 ip-10-0-9-223.ap-south-1.compute.internal systemd[1]: Starting Elasticsearch...
Nov 22 12:00:12 ip-10-0-9-223.ap-south-1.compute.internal systemd[1]: Started Elasticsearch.
Nov 22 12:03:33 ip-10-0-9-223.ap-south-1.compute.internal systemd-entrypoint[7689]: java.lang.OutOfMemoryError: Java heap space
Nov 22 12:03:33 ip-10-0-9-223.ap-south-1.compute.internal systemd-entrypoint[7689]: Dumping heap to /var/lib/elasticsearch/java_pid7753.hprof ...
Nov 22 12:03:44 ip-10-0-9-223.ap-south-1.compute.internal systemd-entrypoint[7689]: Heap dump file created [1830582675 bytes in 10.572 secs]
Nov 22 12:03:44 ip-10-0-9-223.ap-south-1.compute.internal systemd-entrypoint[7689]: Terminating due to java.lang.OutOfMemoryError: Java heap space
Nov 22 12:03:44 ip-10-0-9-223.ap-south-1.compute.internal systemd-entrypoint[7689]: ERROR: Elasticsearch exited unexpectedly

was facing the same issue previously. made changes to jvm options in elasticsearch and logstash to 2g. my machine is 8gb ram 50 gb hardisk. with elk+filebeat on it. what should be the settings?

8

You have issue with Elasticsearch.
By default ES use 50% of RAM, if you haven't set in jvm.options

9

Currently elasticsearch jvm options is set to 2g. And logstash jvm options is set to 2g. Is that ok ?

10

2GB for ES is too low, put at least 4 or 8 GB.
For LS, 2 GB should be OK, if you don't have much data, roughly say 5-7 000/sec for 1 GB.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.