How to increase line_merge_size_limit when importing from Visualizer

[Fosiul_Alam]

Hi,
I am trying to import IIS log file manually via [Data Visualizer]
however I am keep getting bellow message.

Now one solution is to increase "line_merge_size_limit"
Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-find-file-structure.html
however, I am not understanding how to do it,

Can you please help me, do i need to change any setting in yml file or will it by by POST Api

some samples lines are

2020-08-11 14:53:18 GET /resttst1/longlinesREMOVED=20200601 200 3082 -
2020-08-11 14:53:17 POST /resttst1/longlinesREMOVED=20200601 500 1686 -

[illegal_argument_exception] Merging lines into messages resulted in an unacceptably long message. Merged message would have [21] lines and [10535] characters (limit [10000]). If you have messages this big please increase the value of [line_merge_size_limit]. Otherwise it probably means the timestamp has been incorrectly detected, so try overriding that.

Explanation so far:

[Using character encoding [UTF-8], which matched the input with [15%] confidence - first [8kB] of input was pure ASCII]

[Not NDJSON because there was a parsing exception: [Unexpected character ('#' (code 35)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (org.elasticsearch.xpack.ml.filestructurefinder.NdJsonFileStructureFinderFactory$ContextPrintingStringReader); line: 1, column: 2]]]

[Not XML because there was a parsing exception: [ParseError at [row,col]:[1,1] Message: Content is not allowed in prolog.]]

[Not CSV because the first row has fewer than [2] fields: [1]]

[Not TSV because the first row has fewer than [2] fields: [1]]

[Not semicolon delimited values because the first row has fewer than [4] fields: [1]]

[Not vertical line delimited values because the first row has fewer than [5] fields: [1]]

[Deciding sample is text]

[Most likely timestamp format is [yyyy-MM-dd HH:mm:ss]]

[wei.wang]

Hi there,

The first thing to check is the log file itself.

• Is there any long line(s) which doesn't have date-time timestamp at the beginning?

or

• Is there any long line(s) which has different format or invalid date-time timestamp at the beginning, which is not the same as the one in your sample 2020-08-11 14:53:17 ?

If the answer is yes , the log file might have to be corrected before uploading through the Data Visualizer UI.

Cheers.
Wei

[Fosiul_Alam]

Hi Wei.wang
Thanks for repling
i have tested, if i shorted the log lines it works, so which means we have limitation how long the lines can be imported
so date time stamp is not a prolem , is the length of the lines

Thanks

[wei.wang]

Hi,

If it is confirmed that was caused by 10000+ character lines, unfortunately, there’s no way to import it using the Data Visualizer UI at this moment. Feel free to open an enhancement request, we will evaluate and see if we could support it in the future release.

cheers.
Wei