Trying to transform data from one index to another by applying pipeline in transform,But pipeline is executing only first half of it till Android filter

Sharing my transform alongwith Pipeline.

  1. Transform
POST _transform/_preview
{
  "source": {
    "index": [
      "events.test"
    ]
  },
  "pivot": {
    "group_by": {
      "session_id": {
        "terms": {
          "field": "session_id"
        }
      },
      "device_id": {
        "terms": {
          "field": "device_id"
        }
      }
    },
    "aggregations": {
      "session_start": {
        "min": {
          "field": "server_time"
        }
      },
      "session_end": {
        "max": {
          "field": "server_time"
        }
      },
  
      "time_spent": {
        "bucket_script": {
          "buckets_path": {
            "start": "session_start.value",
            "stop": "session_end.value"
          },
          "script": """
            return (params.stop - params.start)/1000;
          """
        }
      }
    }
  },
  "frequency": "15m",
  "dest": {
    "index": "analysis.test",
    "pipeline":"transform_user"
    
  },
  "settings": {
    "max_page_search_size": 500
  }
}


2.Pipeline

PUT _ingest/pipeline/transform_user
{
  "processors": [
    {
      "grok": {
        "field": "session_id",
        "patterns": [
          "(?<param19>...)$"
        ]
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.param19 == 'and') 
          {
            ctx.param19 = 'Android';
          }
        """
      }
	  },
	  {
      "grok": {
        "field": "param20",
        "patterns": [
          "(?<param20>%{GREEDYDATA})$"
          ]
      }
    },
    {
     "script": {
        "source": """
		      ctx.user_type = "Free"
          if (ctx.param20 == "NearRenewalUser" || ctx.param20 == "NearRenewalUserPP"
         || ctx.param20 == "SubUser" || ctx.param20 == "SubUserPP"
         || ctx.param20 == "RenewedUser" || ctx.param20 == "RenewedUserPP"
         || ctx.param20 == "PremiumSubUserPP" || ctx.param20 == "PremiumSubUser")
          {
            ctx.user_type = 'Paid';
          }
        """
      }
    }
  ]
 }

Blockquote

Hi,

  1. Could you share Elasticsearch server logs from around the time you've experienced this issue?

  2. Have you tested the pipeline separately from the transform, i.e.: by issuing a number of index requests with ?pipeline=transform_user in the URL? This way we could see if the problem lies in the pipeline or in the transform.

  3. Should there be a semicolon (;) after the line: ctx.user_type = "Free" ?

Hi My pipeline is showing results in destination index but only till ctx.param19 = 'Android ' , and for another part i.e

ctx.user_type = "Free";
          if (ctx.param20 == "NearRenewalUser" || ctx.param20 == "NearRenewalUserPP"
         || ctx.param20 == "SubUser" || ctx.param20 == "SubUserPP"
         || ctx.param20 == "RenewedUser" || ctx.param20 == "RenewedUserPP"
         || ctx.param20 == "PremiumSubUserPP" || ctx.param20 == "PremiumSubUser")
          {
            ctx.user_type = 'Paid';
          }
        """

It is showing all user_type = Free . So that means it is not going inside if condition.

Kindly suggest any solution on this issue.

This is the response for your refrence

"preview" : [
    {
      "device_id" : "0000000000000000",
      "session_id" : "16613182337680000000000000000_and",
      "session_start" : "2022-08-24 10:47:15",
      "platform" : "Android",
      "param20" : "Free",
      "session_end" : "2022-08-24 10:47:15",
      "time_spent" : 0.0
    },
    {
      "device_id" : "00000ca06f93d39e",
      "session_id" : "166959155804900000ca06f93d39e_and",
      "session_start" : "2022-11-28 23:14:34",
      "platform" : "Android",
      "param20" : "Free",
      "session_end" : "2022-11-28 23:14:34",
      "time_spent" : 0.0
    },

Hi .... any update on this?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.