Nested List Ingest Pipeline

[dwissm1 (Dan)]

Hey Folks.

I have a nested list inside of a list (obs - below) in some data from my tempest weather station. I want to split out the csv and I have the field names. I tried a foreach and a foreach with an append... no joy. Im pretty awful at painless. Anyone see a straight forward way to do this that I am missing?

Its quite simple to do in python outside elastic but want to figure out how to do it.

Elasticsearch: 8.5.2

Sample:

    {
        "serial_number": "ST-00000512",
        "type": "obs_st",
        "hub_sn": "HB-00013030",
        "obs": [
            [1588948614,0.18,0.22,0.27,144,6,1017.57,22.37,50.26,328,0.03,3,0.000000,0,0,0,2.410,1]
        ],
        "firmware_revision": 129
    }

[dwissm1 (Dan)]

Figured it out… was not pretty and used a combo of painless to reassign the value to a string and then used CSV to break it out.

{
            "script": {
                "lang": "painless",
                "source": "ctx.temp_obs = ctx.obs[0].toString();"
            }
        },
        {
            "csv": {
                "field": "temp_obs",
                "target_fields": [
                    "observations.station_timestamp",
                    "observations.wind_lull",
                    "observations.wind_avg",
                    "observations.wind_gust",
                    "observations.wind_direction",
                    "observations.wind_sample_interval",
                    "observations.station_pressure",
                    "observations.air_temperature",
                    "observations.relative_humidity",
                    "observations.illuminance_lux",
                    "observations.uv_index",
                    "observations.solar_radiation",
                    "observations.rain_over_previous_min",
                    "observations.precipitation_type",
                    "observations.lightning_strike_avg_dist",
                    "observations.lightning_strike_count",
                    "observations.battery_volts",
                    "observations.report_interval"
                ],
                "separator": ",",
                "trim": true
            }