Indexing data by getting substring of each row

Elastic Stack Logstash
1

Hi All,

I am new to logstash, I would like to index attached format of data into Elasticsearch using logstash. Could you please provide some samples, so that I can work on them.

The Attached file doesn't contains headers, we need to extract each field value from a row using column index.

For example, column 5 to column 10 - is value for a particular field.

Sample Data:

`

0000C28417401370200000690856000TTEYAA KAKU IND CO.,LTD.                                                                                                                                                                                                                                                                       TAEYMA KAKU IND CO.,LTD.               13, SHIMAN                                                                                                                  TOYMA                                                                                              00105200YAMA                                            TYM       930-1305        PAN                                             3692                                                                                                                                                                                                                                                                                            000                                                                                                                              000 NKATSITO MIZCHI                                         Pres                                                   00005                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          363936793676                    0001958 000000090135954 0000000096032340000 000000058390306 0000000062210000000 000000000715035 00000000007618100000000060200000852Y   --  0000000000020180828           G  003002N17230001001637       00099764834012       311NMNNANP                      NNN006908560003690069085600000690856000369Y00015         01201809260 N20180926                                                                          043000YN0081Mfg household appliances                 0280 TAMA KAGU IND CO.,LTD.                                                                                       30, SHOBAN                                                                                                                  TOYA                                                          001052TOYA                                            TYM       930-1305        000PAN                                             2TATMA INDU CO.,LTD.                                                                                       30, SHIMAN                                                                                                                  TOYA                                                          001052TOA                                            TYM       930-1305        000PAN                                             2TATGA IND CO.,LTD.                                                                                       30, SHIMAN                                                                                                                  TOYA                                                          001052TAMA                                            TYM       930-1305        369  1958M20331 N  11

Above provided is a sample single row, similarly I will be having multiple rows. Each row contains value for multiple fields.

Thanks,
Ram Prasad G

2

What is the expected output?

3

Hi Christian,

Expected output would be like, customer_id: C2841, customer_nameL KAKU, etc.

Thanks,
Ram Prasad G

4

Can you please show how the full event should be parsed and look?

5

Hi Christian,

We need to read the file from logstash, each row should be splited based on the fixed column array index value, below i have given example.
For example,

0000C28417401370200000690856000TTEYAA KAKU IND CO.,LTD. - this is the sample data which contains in a file.
From the above data, we have to index into ES using logstash like,
customer_id : column[5] to column[10] -> C2841
customer_idproof_number: column[11] to column[15] -> 74013, etc

which means we need to extract the data for each field based on the given column index.

Kindly let me know if you need further more information.

Thanks,
Ram Prasad G

6

Any Updates please.

7

Use a grok filter as this support specification of fixed length patterns.

1 Like
8

Thank you very much for your suggestion.
Could you please provide me some example, as I am new to this.

9

Any updates please.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.