I need help... i have problem with reporting CSV on kibana. i want to get CSV with 1.000.000 row from discover but its always failed. i get notif that [Reporting generate CSV error:Unable to generate report Max attempts reached (3)]? what must i do to export CSV in large volume?
Depending on why your report is failing there are a few settings you can tweak in your kibana.yml:
-
xpack.reporting.csv.maxSizeBytes: by default this is set to 1024 * 1024 * 10 = 10MB, you can increase if your export will be larger -
xpack.reporting.csv.scroll.duration, default 30s, increase if you see errors in the log about the scroll not being found or expiring
If these don't work, try looking at the logs for information about why the report is failing. It's possible you have a scripted field that fails on some documents. If so, fix the scripted field to handle the data in the document that fails, or remove it.
1 Like
i have setting the xpack.reporting.csv.maxSizeBytes but now i get another error when reporting CSV. i get notif [Error: Failed to decrypt report job data. Please ensure that xpack.reporting.encryptionKey is set and re-generate this report. Error: Unsupported state or unable to authenticate data]. must i set xpack.reporting.encryptionKey?
Yes it is required for stable usage across restarts. You should see a warning when starting Kibana without it.
I had same type of problem maxsize reach as report was big. I got around by doing it via logstash
output csv was about 500mb.
input {
elasticsearch {
hosts => ["host1:9200", "host2:9200"]
index => "index-2020"
user => "elastic"
password => "xxxx"
}
}
filter {
mutate { remove_field => ["field1","field2","@timestamp","@version"] }
}
output {
csv {
fields => [ "myfield1","myfield2","myfield3" ]
csv_options => { "col_sep" => "," }
path => "/data01/csv_files/test.csv"
}
stdout { codec => rubydebug }
}I had did a config on xpack.reporting.encryptionKey, but its still error with error message [Max attempts reached (3)]. in the log i have massage
{"type":"log","@timestamp":"2020-01-17T07:05:37Z","tags":["reporting","warning"],"pid":6533,"message":"xpack.reporting.csv.maxSizeBytes (1309355388) is higher than ElasticSearch's http.max_content_length (104857600). Please set http.max_content_length in ElasticSearch to match, or lower your xpack.reporting.csv.maxSizeBytes in Kibana to avoid this warning."}.
and
{"type":"log","@timestamp":"2020-01-17T07:05:21Z","tags":["reporting","csv","execute-job","xxx","debug"],"pid":824,"message":"finished generating, total size in bytes: 188286137"}.
how much i must set http.max_content_length and xpack.reporting.csv.maxSizeBytes ? how much maximum size they can be?
@joelgriffith any ideas here on how to get more information about why this is failing?
i have change the xpack.reporting.csv.maxSizeBytes and http.max_content_length to match. but its still get another log like
{"type":"log","@timestamp":"2020-01-20T03:34:19Z","tags":["debug","http"],"pid":610,"message":"Kibana server is not ready yet get:/api/reporting/jobs/list?page=0."}
{"type":"log","@timestamp":"2020-01-20T03:34:20Z","tags":["plugins","debug"],"pid":610,"plugin":{"name":"x-pack","version":"7.4.2","author":"Elastic","private":true,"license":"Elastic-License","kibana":{"build":{"intermediateBuildDirectory":"build/plugin/kibana/x-pack","oss":false},"version":"7.4.2"},"resolutions":{"/@types/node":"10.12.27"},"dependencies":{"@babel/core":"^7.5.5","@babel/register":"^7.5.5","@babel/runtime":"^7.5.5","@elastic/ctags-langserver":"^0.1.9","@elastic/datemath":"5.0.2","@elastic/eui":"13.6.1","@elastic/javascript-typescript-langserver":"^0.2.2","@elastic/lsp-extension":"^0.1.2","@elastic/maki":"6.1.0","@elastic/node-crypto":"^1.0.0","@elastic/nodegit":"0.25.0-alpha.23","@elastic/numeral":"2.3.3","@elastic/request-crypto":"^1.0.2","@kbn/babel-preset":"1.0.0","@kbn/config-schema":"1.0.0","@kbn/elastic-idx":"1.0.0","@kbn/es-query":"1.0.0","@kbn/i18n":"1.0.0","@kbn/interpreter":"1.0.0","@kbn/ui-framework":"1.0.0","@mapbox/mapbox-gl-draw":"^1.1.1","@samverschueren/stream-to-observable":"^0.3.0","@scant/router":"^0.1.0","@slack/client":"^4.8.0","@slack/webhook":"^5.0.0","@turf/boolean-contains":"6.0.1","angular-resource":"1.7.8","angular-sanitize":"1.7.8","angular-ui-ace":"0.2.3","apollo-cache-inmemory":"1.6.2","apollo-client":"^2.3.8","apollo-link":"^1.2.3","apollo-link-error":"^1.1.7","apollo-link-http":"^1.5.4","apollo-link-schema":"^1.1.0","apollo-link-state":"^0.4.1","apollo-server-errors":"^2.0.2","apollo-server-hapi":"^1.3.6","axios":"^0.19.0","bluebird":"3.5.5","boom":"^7.2.0","brace":"0.11.1","chroma-js":"^1.4.1","classnames":"2.2.6","concat-stream":"1.6.2","constate":"^0.9.0","constate-latest":"npm:constate@^1.2.0","content-disposition":"0.5.3","copy-to-clipboard":"^3.0.8","cronstrue":"^1.51.0","d3":"3.5.17","d3-scale":"1.0.7","dataloader":"^1.4.0","dedent":"^0.7.0","dragselect":"1.13.1","elasticsearch":"^16.4.0","extract-zip":"1.6.7","file-saver":"^1.3.8","file-type":"^10.9.0","font-awesome":"4.7.0","formsy-react":"^1.1.5","geojson-rewind":"^0.3.1","get-port":"4.2.0","getos":"^3.1.0","git-url-parse":"11.1.2","github-markdown-css":"^2.10.0","glob":"^7.1.2","graphql":"^0.13.2","graphql-fields":"^1.0.2","graphql-tag":"^2.9.2","graphql-tools":"^3.0.2","h2o2":"^8.1.2","handlebars":"4.3.5","history":"4.9.0","history-extra":"^5.0.1","humps":"2.0.1","i18n-iso-countries":"^4.3.1","icalendar":"0.7.1","idx":"^2.5.2","immer":"^1.5.0","inline-style":"^2.0.0","intl":"^1.2.5","io-ts":"^1.4.2","isbinaryfile":"4.0.2","isomorphic-git":"0.55.5","joi":"^13.5.2","jquery":"^3.4.1","js-yaml":"3.13.1","json-stable-stringify":"^1.0.1","jsonwebtoken":"^8.3.0","jsts":"^1.6.2","lodash":"npm:@elastic/lodash@3.10.1-kibana3","lodash.keyby":"^4.6.0","lodash.lowercase":"^4.3.0","lodash.mean":"^4.1.0","lodash.omitby":"^4.6.0","lodash.orderby":"4.6.0","lodash.pickby":"^4.6.0","lodash.topath":"^4.5.2","lodash.uniqby":"^4.7.0","lz-string":"^1.4.4","mapbox-gl":"0.54.0","mapbox-gl-draw-rectangle-mode":"^1.0.4","markdown-it":"^8.4.1","memoize-one":"^5.0.0","mime":"^2.2.2","mkdirp":"0.5.1","moment":"^2.20.1","moment-duration-format":"^1.3.0","moment-timezone":"^0.5.14","monaco-editor":"^0.17.0","ngreact":"^0.5.1","nock":"10.0.6","node-fetch":"^2.6.0","nodemailer":"^4.7.0","object-hash":"^1.3.1","object-path-immutable":"^3.1.1","oppsy":"^2.0.0","papaparse":"^4.6.3","pdfmake":"0.1.57","pluralize":"3.1.0","pngjs":"3.4.0","polished":"^1.9.2","popper.js":"^1.14.3","prop-types":"^15.6.0","proper-lockfile":"^3.2.0","puid":"1.0.7","puppeteer-core":"^1.19.0","raw-loader":"3.1.0","react":"^16.8.0","react-apollo":"^2.1.4","react-beautiful-dnd":"^8.0.7","react-clipboard.js":"^1.1.2","react-datetime":"^2.14.0","react-dom":"^16.8.0","react-dropzone":"^4.2.9","react-fast-compare":"^2.0.4","react-markdown":"^3.4.1","react-markdown-renderer":"^1.4.0","react-moment-proptypes":"^1.6.0","react-monaco-editor":"^0.26.2","react-portal":"^3.2.0","react-redux":"^5.1.1","react-redux-request":"^1.5.6","react-resize-detector":"^4.2.0","react-router-dom":"^4.3.1","react-select":"^1.2.1","react-shortcuts":"^2.0.0","react-sticky":"^6.0.3","react-syntax-highlighter":"^5.7.0","react-vis":"^1.8.1","react-visibility-sensor":"^5.1.1","recompose":"^0.26.0","reduce-reducers":"^0.4.3","redux":"4.0.0","redux-actions":"2.2.1","redux-observable":"^1.0.0","redux-saga":"^0.16.0","redux-thunk":"2.3.0","redux-thunks":"^1.0.0","request":"^2.88.0","reselect":"3.0.1","resize-observer-polyfill":"^1.5.0","rimraf":"^2.6.2","rison-node":"0.3.1","rxjs":"^6.2.1","semver":"5.7.0","squel":"^5.12.2","stats-lite":"^2.2.0","style-it":"^2.1.3","styled-components":"3.4.10","suricata-sid-db":"^1.0.2","tinycolor2":"1.4.1","tinymath":"1.2.1","topojson-client":"3.0.0","tslib":"^1.9.3","turf":"3.0.14","typescript-fsa":"^2.5.0","typescript-fsa-reducers":"^0.4.5","ui-select":"0.19.8","unstated":"^2.1.1","uuid":"3.3.2","venn.js":"0.2.20","vscode-languageserver":"^5.2.1","wellknown":"^0.5.0","xml2js":"^0.4.19","xregexp":"4.2.4"},"engines":{"yarn":"^1.10.1"},"workspaces":{"nohoist":["/mochawesome","/mochawesome/","/mocha-multi-reporters","/mocha-multi-reporters/**"]},"build":{"git":{"count":"26506","sha":"d7f3941","date":"Mon, 28 Oct 2019 19:44:57 +0100"},"date":"Mon Oct 28 2019 21:09:35 GMT+0000 (Coordinated Universal Time)"}},"message":"Initializing plugin reporting@7.4.2"}
{"type":"log","@timestamp":"2020-01-20T03:34:20Z","tags":["reporting","exportTypes","debug"],"pid":610,"message":"Found exportType at /usr/share/kibana/x-pack/legacy/plugins/reporting/export_types/csv_from_savedobject/server/index.js"}
{"type":"log","@timestamp":"2020-01-20T03:34:20Z","tags":["reporting","exportTypes","debug"],"pid":610,"message":"Found exportType at /usr/share/kibana/x-pack/legacy/plugins/reporting/export_types/csv/server/index.js"}
{"type":"log","@timestamp":"2020-01-20T03:34:20Z","tags":["reporting","exportTypes","debug"],"pid":610,"message":"Found exportType at /usr/share/kibana/x-pack/legacy/plugins/reporting/export_types/png/server/index.js"}
{"type":"log","@timestamp":"2020-01-20T03:34:20Z","tags":["reporting","exportTypes","debug"],"pid":610,"message":"Found exportType at /usr/share/kibana/x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/index.js"}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","browser-driver","debug"],"pid":610,"message":"Browser installed at /var/lib/kibana/headless_shell-linux/headless_shell"}{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","debug"],"pid":610,"message":"Browser type: chromium"}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","debug"],"pid":610,"message":"Chromium sandbox disabled: false"}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["status","plugin:reporting@7.4.2","info"],"pid":610,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","esqueue","queue-worker","debug"],"pid":610,"message":"k5lwbd9d00gyfd3d76a6vgfh - Created worker for reporting jobs"}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","debug"],"pid":610,"message":"Running on os "linux", distribution "Ubuntu Linux", release "16.04""}
{"type":"log","@timestamp":"2020-01-20T03:34:21Z","tags":["reporting","debug"],"pid":610,"message":"Reporting plugin self-check ok!"}
{"type":"log","@timestamp":"2020-01-20T03:34:30Z","tags":["debug","stats-collection"],"pid":610,"message":"Fetching data from reporting collector"}
what mush i do to slove this problem?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.