crp-order-quantity-kpi-service - Details for Query 100339

Submitted Time: 2026/06/09 15:48:41
Duration: 2 s
Succeeded Jobs: 154548 154549 154550 154551 154552 154553 154554 154555

Plan Visualization

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [id="node0" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; 1 [id="node1" labelType="html" label="CollectLimit shuffle records written: 22 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 744.0 B merged fetch fallback count: 0 local blocks read: 12 remote merged chunks fetched: 0 remote blocks read: 0 local merged bytes read: 0.0 B remote reqs duration: 0 ms local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 6 ms (0 ms, 0 ms, 2 ms (stage 445503.0: task 5173512)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time: 0 ms remote bytes read: 0.0 B remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 1364.0 B (62.0 B, 62.0 B, 62.0 B (stage 445503.0: task 5173494))" tooltip="CollectLimit 1"]; 2 [id="node2" labelType="html" label="InMemoryTableScan number of output rows: 64" tooltip="InMemoryTableScan"]; 3 [id="node3" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; subgraph cluster4 { isCluster="true"; id="cluster4"; label="WholeStageCodegen (7)\n \nduration: total (min, med, max (stageId: taskId))\n2.0 s (0 ms, 106 ms, 284 ms (stage 445501.0: task 5173488))"; tooltip="WholeStageCodegen (7)"; 5 [id="node5" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 59 ms (0 ms, 0 ms, 6 ms (stage 445501.0: task 5173470)) peak memory total (min, med, max (stageId: taskId)) 6.3 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 445501.0: task 5173464)) number of output rows: 64 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[product#21517975, plant#21517976], functions=[])"]; 6 [id="node6" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 25 ms (0 ms, 0 ms, 3 ms (stage 445501.0: task 5173473)) peak memory total (min, med, max (stageId: taskId)) 6.3 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 445501.0: task 5173464)) number of output rows: 64 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[product#21517975, plant#21517976], functions=[])"]; 7 [id="node7" labelType="html" label=" Project " tooltip="Project [product#21517975, plant#21517976]"]; 8 [id="node8" labelType="html" label="Filter number of output rows: 64" tooltip="Filter (((size(demandTimeBuckets#21518404, true) = 0) OR isnull(demandTimeBuckets#21518404)) AND ((size(demandPointInTime#21518407, true) = 0) OR isnull(demandPointInTime#21518407)))"]; 9 [id="node9" labelType="html" label="BroadcastHashJoin number of output rows: 64" tooltip="BroadcastHashJoin [coalesce(plant#21517976, ), isnull(plant#21517976), coalesce(product#21517975, ), isnull(product#21517975)], [coalesce(plant#21518400, ), isnull(plant#21518400), coalesce(product#21518399, ), isnull(product#21518399)], LeftOuter, BuildRight, false"]; 10 [id="node10" labelType="html" label=" Project " tooltip="Project [product#21517975, plant#21517976]"]; 11 [id="node11" labelType="html" label="Generate number of output rows: 64" tooltip="Generate explode(_extract_quantityDenominator#21518434), [product#21517975, plant#21517976], true, [quantityDenominator#21518058]"]; 12 [id="node12" labelType="html" label=" Project " tooltip="Project [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047.quantityDenominator AS _extract_quantityDenominator#21518434]"]; 13 [id="node13" labelType="html" label="Generate number of output rows: 64" tooltip="Generate explode(filteredUnitOfMeasureConversions#21518047.quantityNumerator), [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047], true, [quantityNumerator#21518055]"]; } 14 [id="node14" labelType="html" label=" Project " tooltip="Project [product#21517975, plant#21517976, filter(unitOfMeasureConversions#21518041, lambdafunction((((lambda u#21518048.measurementUnit1.code = _extract_code#21518513) AND (lambda u#21518048.measurementUnit1.internalRefUUID = _extract_internalRefUUID#21518514)) AND ((lambda u#21518048.measurementUnit2.code = baseUnitOfMeasure#21518039.code) AND (lambda u#21518048.measurementUnit2.internalRefUUID = baseUnitOfMeasure#21518039.internalRefUUID))), lambda u#21518048, false)) AS filteredUnitOfMeasureConversions#21518047]"]; subgraph cluster15 { isCluster="true"; id="cluster15"; label="WholeStageCodegen (6)\n \nduration: total (min, med, max (stageId: taskId))\n2.4 s (1 ms, 143 ms, 197 ms (stage 445501.0: task 5173481))"; tooltip="WholeStageCodegen (6)"; 16 [id="node16" labelType="html" label="BroadcastHashJoin number of output rows: 64" tooltip="BroadcastHashJoin [product#21517975], [internalUUID#21518035], LeftOuter, BuildRight, false"]; 17 [id="node17" labelType="html" label=" Project " tooltip="Project [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514]"]; 18 [id="node18" labelType="html" label="BroadcastHashJoin number of output rows: 64" tooltip="BroadcastHashJoin [coalesce(product#21517975, ), isnull(product#21517975), coalesce(plant#21517976, ), isnull(plant#21517976)], [coalesce(_extract_internalRefUUID#21518515, ), isnull(_extract_internalRefUUID#21518515), coalesce(_extract_internalRefUUID#21518516, ), isnull(_extract_internalRefUUID#21518516)], LeftOuter, BuildRight, false"]; } 19 [id="node19" labelType="html" label="Exchange shuffle records written: 64 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 64 local bytes read total (min, med, max (stageId: taskId)) 5.0 KiB (144.0 B, 213.0 B, 370.0 B (stage 445501.0: task 5173475)) merged fetch fallback count: 0 local blocks read: 22 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 5.5 KiB (88.0 B, 176.0 B, 528.0 B (stage 445493.0: task 5173390)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 8 ms (0 ms, 0 ms, 0 ms (stage 445493.0: task 5173407)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 445501.0: task 5173464)) remote bytes read: 0.0 B number of partitions: 25 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 5.0 KiB (144.0 B, 213.0 B, 370.0 B (stage 445493.0: task 5173392))" tooltip="Exchange hashpartitioning(product#21517975, plant#21517976, 25), REPARTITION_BY_NUM, [plan_id=21256258]"]; subgraph cluster20 { isCluster="true"; id="cluster20"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n2.5 s (62 ms, 94 ms, 293 ms (stage 445493.0: task 5173394))"; tooltip="WholeStageCodegen (1)"; 21 [id="node21" labelType="html" label="ColumnarToRow number of output rows: 64 number of input batches: 22" tooltip="ColumnarToRow"]; } 22 [id="node22" labelType="html" label="Scan parquet number of files read: 22 scan time total (min, med, max (stageId: taskId)) 2.3 s (62 ms, 93 ms, 264 ms (stage 445493.0: task 5173394)) metadata time: 0 ms size of files read: 38.4 KiB number of output rows: 64" tooltip="FileScan parquet [product#21517975,plant#21517976] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<product:string,plant:string>"]; 23 [id="node23" labelType="html" label="BroadcastExchange time to broadcast: 5 ms time to build: 20 ms time to collect: 102 ms number of output rows: 12,856 data size: 4.5 MiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[2, string, true], ), isnull(input[2, string, true]), coalesce(input[3, string, true], ), isnull(input[3, string, true])),false), [plan_id=21256493]"]; subgraph cluster24 { isCluster="true"; id="cluster24"; label="WholeStageCodegen (5)\n \nduration: total (min, med, max (stageId: taskId))\n18 ms (0 ms, 1 ms, 2 ms (stage 445499.0: task 5173440))"; tooltip="WholeStageCodegen (5)"; 25 [id="node25" labelType="html" label=" Project " tooltip="Project [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]"]; } 26 [id="node26" labelType="html" label="Exchange shuffle records written: 12,856 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 12,856 local bytes read total (min, med, max (stageId: taskId)) 512.5 KiB (20.2 KiB, 20.5 KiB, 21.0 KiB (stage 445499.0: task 5173449)) merged fetch fallback count: 0 local blocks read: 25 remote merged chunks fetched: 0 remote blocks read: 0 data size: 1805.1 KiB local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time: 2 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 445499.0: task 5173439)) remote bytes read: 0.0 B number of partitions: 25 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 512.5 KiB" tooltip="Exchange RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256279]"]; subgraph cluster27 { isCluster="true"; id="cluster27"; label="WholeStageCodegen (2)\n \nduration: 88 ms"; tooltip="WholeStageCodegen (2)"; 28 [id="node28" labelType="html" label=" Project " tooltip="Project [plant#21517982.internalRefUUID AS _extract_internalRefUUID#21518516, product#21517983.internalRefUUID AS _extract_internalRefUUID#21518515, productionAspect#21517985.productMovementPlants.unitOfIssue.code AS _extract_code#21518513, productionAspect#21517985.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#21518514]"]; 29 [id="node29" labelType="html" label="ColumnarToRow number of output rows: 12,856 number of input batches: 4" tooltip="ColumnarToRow"]; } 30 [id="node30" labelType="html" label="Scan parquet number of files read: 1 scan time: 79 ms dynamic partition pruning time: 0 ms metadata time: 0 ms size of files read: 588.0 KiB number of output rows: 12,856 number of partitions read: 1" tooltip="FileScan parquet [plant#21517982,product#21517983,productionAspect#21517985,systemId#21517980] Batched: true, DataFilters: [], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<plant:struct<internalRefUUID:string>,product:struct<internalRefUUID:string>,productionAspe..."]; 31 [id="node31" labelType="html" label="BroadcastExchange time to broadcast: 1 ms time to build: 8 ms time to collect: 101 ms number of output rows: 150 data size: 2.0 MiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=21256317]"]; 32 [id="node32" labelType="html" label="Exchange shuffle records written: 150 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 150 local bytes read total (min, med, max (stageId: taskId)) 25.5 KiB (869.0 B, 1046.0 B, 1310.0 B (stage 445497.0: task 5173412)) merged fetch fallback count: 0 local blocks read: 25 remote merged chunks fetched: 0 remote blocks read: 0 data size: 89.8 KiB local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 445497.0: task 5173408)) remote bytes read: 0.0 B number of partitions: 25 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 25.5 KiB" tooltip="Exchange RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256307]"]; subgraph cluster33 { isCluster="true"; id="cluster33"; label="WholeStageCodegen (3)\n \nduration: 123 ms"; tooltip="WholeStageCodegen (3)"; 34 [id="node34" labelType="html" label=" Project " tooltip="Project [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]"]; 35 [id="node35" labelType="html" label="Filter number of output rows: 150" tooltip="Filter isnotnull(internalUUID#21518035)"]; 36 [id="node36" labelType="html" label="ColumnarToRow number of output rows: 150 number of input batches: 1" tooltip="ColumnarToRow"]; } 37 [id="node37" labelType="html" label="Scan parquet number of files read: 1 scan time: 119 ms dynamic partition pruning time: 0 ms metadata time: 0 ms size of files read: 34.1 KiB number of output rows: 150 number of partitions read: 1" tooltip="FileScan parquet [internalUUID#21518035,baseUnitOfMeasure#21518039,unitOfMeasureConversions#21518041,systemId#21518034] Batched: true, DataFilters: [isnotnull(internalUUID#21518035)], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20...., PartitionFilters: [], PushedFilters: [IsNotNull(internalUUID)], ReadSchema: struct<internalUUID:string,baseUnitOfMeasure:struct<internalRefUUID:string,code:string>,unitOfMea..."]; 38 [id="node38" labelType="html" label="BroadcastExchange time to broadcast: 0 ms time to build: 5 ms time to collect: 490 ms number of output rows: 64 data size: 2.0 MiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=21256331]"]; subgraph cluster39 { isCluster="true"; id="cluster39"; label="WholeStageCodegen (4)\n \nduration: 99 ms"; tooltip="WholeStageCodegen (4)"; 40 [id="node40" labelType="html" label="ColumnarToRow number of output rows: 64 number of input batches: 1" tooltip="ColumnarToRow"]; } 41 [id="node41" labelType="html" label="Scan parquet number of files read: 1 scan time: 97 ms metadata time: 0 ms size of files read: 5.8 KiB number of output rows: 64" tooltip="FileScan parquet [product#21518399,plant#21518400,demandTimeBuckets#21518404,demandPointInTime#21518407] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<product:string,plant:string,demandTimeBuckets:array<struct<demandStartDateTime:timestamp,d..."]; 1->0; 2->1; 3->2; 5->3; 6->5; 7->6; 8->7; 9->8; 10->9; 11->10; 12->11; 13->12; 14->13; 16->14; 17->16; 18->17; 19->18; 21->19; 22->21; 23->18; 25->23; 26->25; 28->26; 29->28; 30->29; 31->16; 32->31; 34->32; 35->34; 36->35; 37->36; 38->9; 40->38; 41->40; }

Details

== Physical Plan ==
AdaptiveSparkPlan (69)
+- == Final Plan ==
   ResultQueryStage (67), Statistics(sizeInBytes=8.0 EiB)
   +- CollectLimit (66)
      +- TableCacheQueryStage (65), Statistics(sizeInBytes=4.5 KiB, rowCount=64)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- AdaptiveSparkPlan (64)
                     +- == Final Plan ==
                        ResultQueryStage (40)
                        +- * HashAggregate (39)
                           +- * HashAggregate (38)
                              +- * Project (37)
                                 +- * Filter (36)
                                    +- * BroadcastHashJoin LeftOuter BuildRight (35)
                                       :- * Project (30)
                                       :  +- * Generate (29)
                                       :     +- * Project (28)
                                       :        +- * Generate (27)
                                       :           +- Project (26)
                                       :              +- * BroadcastHashJoin LeftOuter BuildRight (25)
                                       :                 :- * Project (16)
                                       :                 :  +- * BroadcastHashJoin LeftOuter BuildRight (15)
                                       :                 :     :- ShuffleQueryStage (6), Statistics(sizeInBytes=5.5 KiB, rowCount=64)
                                       :                 :     :  +- Exchange (5)
                                       :                 :     :     +- * ColumnarToRow (4)
                                       :                 :     :        +- Scan parquet  (3)
                                       :                 :     +- BroadcastQueryStage (14), Statistics(sizeInBytes=4.5 MiB, rowCount=1.29E+4)
                                       :                 :        +- BroadcastExchange (13)
                                       :                 :           +- * Project (12)
                                       :                 :              +- ShuffleQueryStage (11), Statistics(sizeInBytes=1805.1 KiB, rowCount=1.29E+4)
                                       :                 :                 +- Exchange (10)
                                       :                 :                    +- * Project (9)
                                       :                 :                       +- * ColumnarToRow (8)
                                       :                 :                          +- Scan parquet  (7)
                                       :                 +- BroadcastQueryStage (24), Statistics(sizeInBytes=2.0 MiB, rowCount=150)
                                       :                    +- BroadcastExchange (23)
                                       :                       +- ShuffleQueryStage (22), Statistics(sizeInBytes=89.8 KiB, rowCount=150)
                                       :                          +- Exchange (21)
                                       :                             +- * Project (20)
                                       :                                +- * Filter (19)
                                       :                                   +- * ColumnarToRow (18)
                                       :                                      +- Scan parquet  (17)
                                       +- BroadcastQueryStage (34), Statistics(sizeInBytes=2.0 MiB, rowCount=64)
                                          +- BroadcastExchange (33)
                                             +- * ColumnarToRow (32)
                                                +- Scan parquet  (31)
                     +- == Initial Plan ==
                        HashAggregate (63)
                        +- HashAggregate (62)
                           +- Project (61)
                              +- Filter (60)
                                 +- BroadcastHashJoin LeftOuter BuildRight (59)
                                    :- Project (57)
                                    :  +- Generate (56)
                                    :     +- Project (55)
                                    :        +- Generate (54)
                                    :           +- Project (53)
                                    :              +- BroadcastHashJoin LeftOuter BuildRight (52)
                                    :                 :- Project (47)
                                    :                 :  +- BroadcastHashJoin LeftOuter BuildRight (46)
                                    :                 :     :- Exchange (41)
                                    :                 :     :  +- Scan parquet  (3)
                                    :                 :     +- BroadcastExchange (45)
                                    :                 :        +- Project (44)
                                    :                 :           +- Exchange (43)
                                    :                 :              +- Project (42)
                                    :                 :                 +- Scan parquet  (7)
                                    :                 +- BroadcastExchange (51)
                                    :                    +- Exchange (50)
                                    :                       +- Project (49)
                                    :                          +- Filter (48)
                                    :                             +- Scan parquet  (17)
                                    +- BroadcastExchange (58)
                                       +- Scan parquet  (31)
+- == Initial Plan ==
   CollectLimit (68)
   +- InMemoryTableScan (1)
         +- InMemoryRelation (2)
               +- AdaptiveSparkPlan (64)
               +- == Final Plan ==
                  ResultQueryStage (40)
                  +- * HashAggregate (39)
                     +- * HashAggregate (38)
                        +- * Project (37)
                           +- * Filter (36)
                              +- * BroadcastHashJoin LeftOuter BuildRight (35)
                                 :- * Project (30)
                                 :  +- * Generate (29)
                                 :     +- * Project (28)
                                 :        +- * Generate (27)
                                 :           +- Project (26)
                                 :              +- * BroadcastHashJoin LeftOuter BuildRight (25)
                                 :                 :- * Project (16)
                                 :                 :  +- * BroadcastHashJoin LeftOuter BuildRight (15)
                                 :                 :     :- ShuffleQueryStage (6), Statistics(sizeInBytes=5.5 KiB, rowCount=64)
                                 :                 :     :  +- Exchange (5)
                                 :                 :     :     +- * ColumnarToRow (4)
                                 :                 :     :        +- Scan parquet  (3)
                                 :                 :     +- BroadcastQueryStage (14), Statistics(sizeInBytes=4.5 MiB, rowCount=1.29E+4)
                                 :                 :        +- BroadcastExchange (13)
                                 :                 :           +- * Project (12)
                                 :                 :              +- ShuffleQueryStage (11), Statistics(sizeInBytes=1805.1 KiB, rowCount=1.29E+4)
                                 :                 :                 +- Exchange (10)
                                 :                 :                    +- * Project (9)
                                 :                 :                       +- * ColumnarToRow (8)
                                 :                 :                          +- Scan parquet  (7)
                                 :                 +- BroadcastQueryStage (24), Statistics(sizeInBytes=2.0 MiB, rowCount=150)
                                 :                    +- BroadcastExchange (23)
                                 :                       +- ShuffleQueryStage (22), Statistics(sizeInBytes=89.8 KiB, rowCount=150)
                                 :                          +- Exchange (21)
                                 :                             +- * Project (20)
                                 :                                +- * Filter (19)
                                 :                                   +- * ColumnarToRow (18)
                                 :                                      +- Scan parquet  (17)
                                 +- BroadcastQueryStage (34), Statistics(sizeInBytes=2.0 MiB, rowCount=64)
                                    +- BroadcastExchange (33)
                                       +- * ColumnarToRow (32)
                                          +- Scan parquet  (31)
               +- == Initial Plan ==
                  HashAggregate (63)
                  +- HashAggregate (62)
                     +- Project (61)
                        +- Filter (60)
                           +- BroadcastHashJoin LeftOuter BuildRight (59)
                              :- Project (57)
                              :  +- Generate (56)
                              :     +- Project (55)
                              :        +- Generate (54)
                              :           +- Project (53)
                              :              +- BroadcastHashJoin LeftOuter BuildRight (52)
                              :                 :- Project (47)
                              :                 :  +- BroadcastHashJoin LeftOuter BuildRight (46)
                              :                 :     :- Exchange (41)
                              :                 :     :  +- Scan parquet  (3)
                              :                 :     +- BroadcastExchange (45)
                              :                 :        +- Project (44)
                              :                 :           +- Exchange (43)
                              :                 :              +- Project (42)
                              :                 :                 +- Scan parquet  (7)
                              :                 +- BroadcastExchange (51)
                              :                    +- Exchange (50)
                              :                       +- Project (49)
                              :                          +- Filter (48)
                              :                             +- Scan parquet  (17)
                              +- BroadcastExchange (58)
                                 +- Scan parquet  (31)


(1) InMemoryTableScan
Output: []

(2) InMemoryRelation
Arguments: [product#21517975, plant#21517976], StorageLevel(disk, memory, 2 replicas)

(3) Scan parquet 
Output [2]: [product#21517975, plant#21517976]
Batched: true
Location: InMemoryFileIndex [hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/product-plant-list/10000001651/shardId=0_1_10000001651]
ReadSchema: struct<product:string,plant:string>

(4) ColumnarToRow [codegen id : 1]
Input [2]: [product#21517975, plant#21517976]

(5) Exchange
Input [2]: [product#21517975, plant#21517976]
Arguments: hashpartitioning(product#21517975, plant#21517976, 25), REPARTITION_BY_NUM, [plan_id=21256258]

(6) ShuffleQueryStage
Output [2]: [product#21517975, plant#21517976]
Arguments: 0

(7) Scan parquet 
Output [4]: [plant#21517982, product#21517983, productionAspect#21517985, systemId#21517980]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant]
ReadSchema: struct<plant:struct<internalRefUUID:string>,product:struct<internalRefUUID:string>,productionAspect:struct<productMovementPlants:struct<unitOfIssue:struct<internalRefUUID:string,code:string>>>>

(8) ColumnarToRow [codegen id : 2]
Input [4]: [plant#21517982, product#21517983, productionAspect#21517985, systemId#21517980]

(9) Project [codegen id : 2]
Output [4]: [plant#21517982.internalRefUUID AS _extract_internalRefUUID#21518516, product#21517983.internalRefUUID AS _extract_internalRefUUID#21518515, productionAspect#21517985.productMovementPlants.unitOfIssue.code AS _extract_code#21518513, productionAspect#21517985.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#21518514]
Input [4]: [plant#21517982, product#21517983, productionAspect#21517985, systemId#21517980]

(10) Exchange
Input [4]: [_extract_internalRefUUID#21518516, _extract_internalRefUUID#21518515, _extract_code#21518513, _extract_internalRefUUID#21518514]
Arguments: RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256279]

(11) ShuffleQueryStage
Output [4]: [_extract_internalRefUUID#21518516, _extract_internalRefUUID#21518515, _extract_code#21518513, _extract_internalRefUUID#21518514]
Arguments: 1

(12) Project [codegen id : 5]
Output [4]: [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]
Input [4]: [_extract_internalRefUUID#21518516, _extract_internalRefUUID#21518515, _extract_code#21518513, _extract_internalRefUUID#21518514]

(13) BroadcastExchange
Input [4]: [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[2, string, true], ), isnull(input[2, string, true]), coalesce(input[3, string, true], ), isnull(input[3, string, true])),false), [plan_id=21256493]

(14) BroadcastQueryStage
Output [4]: [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]
Arguments: 5

(15) BroadcastHashJoin [codegen id : 6]
Left keys [4]: [coalesce(product#21517975, ), isnull(product#21517975), coalesce(plant#21517976, ), isnull(plant#21517976)]
Right keys [4]: [coalesce(_extract_internalRefUUID#21518515, ), isnull(_extract_internalRefUUID#21518515), coalesce(_extract_internalRefUUID#21518516, ), isnull(_extract_internalRefUUID#21518516)]
Join type: LeftOuter
Join condition: None

(16) Project [codegen id : 6]
Output [4]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514]
Input [6]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]

(17) Scan parquet 
Output [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-product-v2]
PushedFilters: [IsNotNull(internalUUID)]
ReadSchema: struct<internalUUID:string,baseUnitOfMeasure:struct<internalRefUUID:string,code:string>,unitOfMeasureConversions:array<struct<measurementUnit1:struct<internalRefUUID:string,code:string>,measurementUnit2:struct<internalRefUUID:string,code:string>,quantityNumerator:int,quantityDenominator:int>>>

(18) ColumnarToRow [codegen id : 3]
Input [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]

(19) Filter [codegen id : 3]
Input [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]
Condition : isnotnull(internalUUID#21518035)

(20) Project [codegen id : 3]
Output [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Input [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]

(21) Exchange
Input [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256307]

(22) ShuffleQueryStage
Output [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: 2

(23) BroadcastExchange
Input [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=21256317]

(24) BroadcastQueryStage
Output [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: 4

(25) BroadcastHashJoin [codegen id : 6]
Left keys [1]: [product#21517975]
Right keys [1]: [internalUUID#21518035]
Join type: LeftOuter
Join condition: None

(26) Project
Output [3]: [product#21517975, plant#21517976, filter(unitOfMeasureConversions#21518041, lambdafunction((((lambda u#21518048.measurementUnit1.code = _extract_code#21518513) AND (lambda u#21518048.measurementUnit1.internalRefUUID = _extract_internalRefUUID#21518514)) AND ((lambda u#21518048.measurementUnit2.code = baseUnitOfMeasure#21518039.code) AND (lambda u#21518048.measurementUnit2.internalRefUUID = baseUnitOfMeasure#21518039.internalRefUUID))), lambda u#21518048, false)) AS filteredUnitOfMeasureConversions#21518047]
Input [7]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514, internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]

(27) Generate [codegen id : 7]
Input [3]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047]
Arguments: explode(filteredUnitOfMeasureConversions#21518047.quantityNumerator), [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047], true, [quantityNumerator#21518055]

(28) Project [codegen id : 7]
Output [3]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047.quantityDenominator AS _extract_quantityDenominator#21518434]
Input [4]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047, quantityNumerator#21518055]

(29) Generate [codegen id : 7]
Input [3]: [product#21517975, plant#21517976, _extract_quantityDenominator#21518434]
Arguments: explode(_extract_quantityDenominator#21518434), [product#21517975, plant#21517976], true, [quantityDenominator#21518058]

(30) Project [codegen id : 7]
Output [2]: [product#21517975, plant#21517976]
Input [3]: [product#21517975, plant#21517976, quantityDenominator#21518058]

(31) Scan parquet 
Output [4]: [product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Batched: true
Location: InMemoryFileIndex [hdlfs://f1cc4c74-cdd4-440e-abc7-4e0157628b42.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/out/demand/10000001651/0_1_10000001651]
ReadSchema: struct<product:string,plant:string,demandTimeBuckets:array<struct<demandStartDateTime:timestamp,demandQuantity:double,demandVariance:double>>,demandPointInTime:array<struct<demandDateTime:timestamp,demandQuantity:double,demandVariance:double>>>

(32) ColumnarToRow [codegen id : 4]
Input [4]: [product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]

(33) BroadcastExchange
Input [4]: [product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=21256331]

(34) BroadcastQueryStage
Output [4]: [product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Arguments: 3

(35) BroadcastHashJoin [codegen id : 7]
Left keys [4]: [coalesce(plant#21517976, ), isnull(plant#21517976), coalesce(product#21517975, ), isnull(product#21517975)]
Right keys [4]: [coalesce(plant#21518400, ), isnull(plant#21518400), coalesce(product#21518399, ), isnull(product#21518399)]
Join type: LeftOuter
Join condition: None

(36) Filter [codegen id : 7]
Input [6]: [product#21517975, plant#21517976, product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Condition : (((size(demandTimeBuckets#21518404, true) = 0) OR isnull(demandTimeBuckets#21518404)) AND ((size(demandPointInTime#21518407, true) = 0) OR isnull(demandPointInTime#21518407)))

(37) Project [codegen id : 7]
Output [2]: [product#21517975, plant#21517976]
Input [6]: [product#21517975, plant#21517976, product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]

(38) HashAggregate [codegen id : 7]
Input [2]: [product#21517975, plant#21517976]
Keys [2]: [product#21517975, plant#21517976]
Functions: []
Aggregate Attributes: []
Results [2]: [product#21517975, plant#21517976]

(39) HashAggregate [codegen id : 7]
Input [2]: [product#21517975, plant#21517976]
Keys [2]: [product#21517975, plant#21517976]
Functions: []
Aggregate Attributes: []
Results [2]: [product#21517975, plant#21517976]

(40) ResultQueryStage
Output [2]: [product#21517975, plant#21517976]
Arguments: 6

(41) Exchange
Input [2]: [product#21517975, plant#21517976]
Arguments: hashpartitioning(product#21517975, plant#21517976, 25), REPARTITION_BY_NUM, [plan_id=21256199]

(42) Project
Output [4]: [plant#21517982.internalRefUUID AS _extract_internalRefUUID#21518516, product#21517983.internalRefUUID AS _extract_internalRefUUID#21518515, productionAspect#21517985.productMovementPlants.unitOfIssue.code AS _extract_code#21518513, productionAspect#21517985.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#21518514]
Input [4]: [plant#21517982, product#21517983, productionAspect#21517985, systemId#21517980]

(43) Exchange
Input [4]: [_extract_internalRefUUID#21518516, _extract_internalRefUUID#21518515, _extract_code#21518513, _extract_internalRefUUID#21518514]
Arguments: RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256202]

(44) Project
Output [4]: [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]
Input [4]: [_extract_internalRefUUID#21518516, _extract_internalRefUUID#21518515, _extract_code#21518513, _extract_internalRefUUID#21518514]

(45) BroadcastExchange
Input [4]: [_extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[2, string, true], ), isnull(input[2, string, true]), coalesce(input[3, string, true], ), isnull(input[3, string, true])),false), [plan_id=21256223]

(46) BroadcastHashJoin
Left keys [4]: [coalesce(product#21517975, ), isnull(product#21517975), coalesce(plant#21517976, ), isnull(plant#21517976)]
Right keys [4]: [coalesce(_extract_internalRefUUID#21518515, ), isnull(_extract_internalRefUUID#21518515), coalesce(_extract_internalRefUUID#21518516, ), isnull(_extract_internalRefUUID#21518516)]
Join type: LeftOuter
Join condition: None

(47) Project
Output [4]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514]
Input [6]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514, _extract_internalRefUUID#21518515, _extract_internalRefUUID#21518516]

(48) Filter
Input [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]
Condition : isnotnull(internalUUID#21518035)

(49) Project
Output [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Input [4]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041, systemId#21518034]

(50) Exchange
Input [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: RoundRobinPartitioning(25), REPARTITION_BY_NUM, [plan_id=21256209]

(51) BroadcastExchange
Input [3]: [internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true]),false), [plan_id=21256227]

(52) BroadcastHashJoin
Left keys [1]: [product#21517975]
Right keys [1]: [internalUUID#21518035]
Join type: LeftOuter
Join condition: None

(53) Project
Output [3]: [product#21517975, plant#21517976, filter(unitOfMeasureConversions#21518041, lambdafunction((((lambda u#21518048.measurementUnit1.code = _extract_code#21518513) AND (lambda u#21518048.measurementUnit1.internalRefUUID = _extract_internalRefUUID#21518514)) AND ((lambda u#21518048.measurementUnit2.code = baseUnitOfMeasure#21518039.code) AND (lambda u#21518048.measurementUnit2.internalRefUUID = baseUnitOfMeasure#21518039.internalRefUUID))), lambda u#21518048, false)) AS filteredUnitOfMeasureConversions#21518047]
Input [7]: [product#21517975, plant#21517976, _extract_code#21518513, _extract_internalRefUUID#21518514, internalUUID#21518035, baseUnitOfMeasure#21518039, unitOfMeasureConversions#21518041]

(54) Generate
Input [3]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047]
Arguments: explode(filteredUnitOfMeasureConversions#21518047.quantityNumerator), [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047], true, [quantityNumerator#21518055]

(55) Project
Output [3]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047.quantityDenominator AS _extract_quantityDenominator#21518434]
Input [4]: [product#21517975, plant#21517976, filteredUnitOfMeasureConversions#21518047, quantityNumerator#21518055]

(56) Generate
Input [3]: [product#21517975, plant#21517976, _extract_quantityDenominator#21518434]
Arguments: explode(_extract_quantityDenominator#21518434), [product#21517975, plant#21517976], true, [quantityDenominator#21518058]

(57) Project
Output [2]: [product#21517975, plant#21517976]
Input [3]: [product#21517975, plant#21517976, quantityDenominator#21518058]

(58) BroadcastExchange
Input [4]: [product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=21256235]

(59) BroadcastHashJoin
Left keys [4]: [coalesce(plant#21517976, ), isnull(plant#21517976), coalesce(product#21517975, ), isnull(product#21517975)]
Right keys [4]: [coalesce(plant#21518400, ), isnull(plant#21518400), coalesce(product#21518399, ), isnull(product#21518399)]
Join type: LeftOuter
Join condition: None

(60) Filter
Input [6]: [product#21517975, plant#21517976, product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]
Condition : (((size(demandTimeBuckets#21518404, true) = 0) OR isnull(demandTimeBuckets#21518404)) AND ((size(demandPointInTime#21518407, true) = 0) OR isnull(demandPointInTime#21518407)))

(61) Project
Output [2]: [product#21517975, plant#21517976]
Input [6]: [product#21517975, plant#21517976, product#21518399, plant#21518400, demandTimeBuckets#21518404, demandPointInTime#21518407]

(62) HashAggregate
Input [2]: [product#21517975, plant#21517976]
Keys [2]: [product#21517975, plant#21517976]
Functions: []
Aggregate Attributes: []
Results [2]: [product#21517975, plant#21517976]

(63) HashAggregate
Input [2]: [product#21517975, plant#21517976]
Keys [2]: [product#21517975, plant#21517976]
Functions: []
Aggregate Attributes: []
Results [2]: [product#21517975, plant#21517976]

(64) AdaptiveSparkPlan
Output [2]: [product#21517975, plant#21517976]
Arguments: isFinalPlan=true

(65) TableCacheQueryStage
Output: []
Arguments: 0

(66) CollectLimit
Input: []
Arguments: 1

(67) ResultQueryStage
Output: []
Arguments: 1

(68) CollectLimit
Input: []
Arguments: 1

(69) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.activity.hdlfsEnabled	true
spark.activity.isolateSession	true
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true
temporal.activity.id	6d8a08e4-7f20-3db7-b055-c6b5fee432b1
temporal.activity.type	BufferDataSnapShot