== Physical Plan ==
AdaptiveSparkPlan (155)
+- == Final Plan ==
* HashAggregate (151)
+- AQEShuffleRead (150)
+- ShuffleQueryStage (149), Statistics(sizeInBytes=240.0 B, rowCount=6)
+- Exchange (148)
+- * HashAggregate (147)
+- TableCacheQueryStage (146), Statistics(sizeInBytes=1424.0 B, rowCount=8)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (145)
+- Union (144)
:- Project (23)
: +- BroadcastHashJoin Inner BuildLeft (22)
: :- BroadcastExchange (12)
: : +- Filter (11)
: : +- InMemoryTableScan (3)
: : +- InMemoryRelation (4)
: : +- AdaptiveSparkPlan (10)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (21)
: +- Filter (20)
: +- InMemoryTableScan (13)
: +- InMemoryRelation (14)
: +- AdaptiveSparkPlan (19)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
:- Project (72)
: +- Filter (71)
: +- Window (70)
: +- WindowGroupLimit (69)
: +- Sort (68)
: +- Exchange (67)
: +- WindowGroupLimit (66)
: +- Sort (65)
: +- Project (64)
: +- BatchEvalPython (63)
: +- Project (62)
: +- Filter (61)
: +- BatchEvalPython (60)
: +- Project (59)
: +- BroadcastHashJoin Inner BuildLeft (58)
: :- BroadcastExchange (51)
: : +- Project (50)
: : +- SortMergeJoin LeftAnti (49)
: : :- Sort (34)
: : : +- Exchange (33)
: : : +- Project (32)
: : : +- BatchEvalPython (31)
: : : +- Project (30)
: : : +- Filter (29)
: : : +- BatchEvalPython (28)
: : : +- Filter (27)
: : : +- InMemoryTableScan (24)
: : : +- InMemoryRelation (25)
: : : +- AdaptiveSparkPlan (26)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Sort (48)
: : +- Exchange (47)
: : +- Project (46)
: : +- BroadcastHashJoin Inner BuildLeft (45)
: : :- BroadcastExchange (39)
: : : +- Filter (38)
: : : +- InMemoryTableScan (35)
: : : +- InMemoryRelation (36)
: : : +- AdaptiveSparkPlan (37)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Project (44)
: : +- Filter (43)
: : +- InMemoryTableScan (40)
: : +- InMemoryRelation (41)
: : +- AdaptiveSparkPlan (42)
: : +- Filter (18)
: : +- InMemoryTableScan (15)
: : +- InMemoryRelation (16)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
: +- Project (57)
: +- BatchEvalPython (56)
: +- Filter (55)
: +- InMemoryTableScan (52)
: +- InMemoryRelation (53)
: +- AdaptiveSparkPlan (54)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (143)
+- SortMergeJoin LeftAnti (142)
:- SortMergeJoin LeftAnti (92)
: :- Sort (77)
: : +- Exchange (76)
: : +- InMemoryTableScan (73)
: : +- InMemoryRelation (74)
: : +- AdaptiveSparkPlan (75)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (91)
: +- Exchange (90)
: +- Project (89)
: +- BroadcastHashJoin Inner BuildLeft (88)
: :- BroadcastExchange (82)
: : +- Filter (81)
: : +- InMemoryTableScan (78)
: : +- InMemoryRelation (79)
: : +- AdaptiveSparkPlan (80)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (87)
: +- Filter (86)
: +- InMemoryTableScan (83)
: +- InMemoryRelation (84)
: +- AdaptiveSparkPlan (85)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (141)
+- Filter (140)
+- Window (139)
+- WindowGroupLimit (138)
+- Sort (137)
+- Exchange (136)
+- WindowGroupLimit (135)
+- Sort (134)
+- Project (133)
+- BatchEvalPython (132)
+- Project (131)
+- Filter (130)
+- BatchEvalPython (129)
+- Project (128)
+- BroadcastHashJoin Inner BuildLeft (127)
:- BroadcastExchange (120)
: +- Project (119)
: +- SortMergeJoin LeftAnti (118)
: :- Sort (103)
: : +- Exchange (102)
: : +- Project (101)
: : +- BatchEvalPython (100)
: : +- Project (99)
: : +- Filter (98)
: : +- BatchEvalPython (97)
: : +- Filter (96)
: : +- InMemoryTableScan (93)
: : +- InMemoryRelation (94)
: : +- AdaptiveSparkPlan (95)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (117)
: +- Exchange (116)
: +- Project (115)
: +- BroadcastHashJoin Inner BuildLeft (114)
: :- BroadcastExchange (108)
: : +- Filter (107)
: : +- InMemoryTableScan (104)
: : +- InMemoryRelation (105)
: : +- AdaptiveSparkPlan (106)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (113)
: +- Filter (112)
: +- InMemoryTableScan (109)
: +- InMemoryRelation (110)
: +- AdaptiveSparkPlan (111)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (126)
+- BatchEvalPython (125)
+- Filter (124)
+- InMemoryTableScan (121)
+- InMemoryRelation (122)
+- AdaptiveSparkPlan (123)
+- Filter (18)
+- InMemoryTableScan (15)
+- InMemoryRelation (16)
+- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- == Initial Plan ==
HashAggregate (154)
+- Exchange (153)
+- HashAggregate (152)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (145)
+- Union (144)
:- Project (23)
: +- BroadcastHashJoin Inner BuildLeft (22)
: :- BroadcastExchange (12)
: : +- Filter (11)
: : +- InMemoryTableScan (3)
: : +- InMemoryRelation (4)
: : +- AdaptiveSparkPlan (10)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (21)
: +- Filter (20)
: +- InMemoryTableScan (13)
: +- InMemoryRelation (14)
: +- AdaptiveSparkPlan (19)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
:- Project (72)
: +- Filter (71)
: +- Window (70)
: +- WindowGroupLimit (69)
: +- Sort (68)
: +- Exchange (67)
: +- WindowGroupLimit (66)
: +- Sort (65)
: +- Project (64)
: +- BatchEvalPython (63)
: +- Project (62)
: +- Filter (61)
: +- BatchEvalPython (60)
: +- Project (59)
: +- BroadcastHashJoin Inner BuildLeft (58)
: :- BroadcastExchange (51)
: : +- Project (50)
: : +- SortMergeJoin LeftAnti (49)
: : :- Sort (34)
: : : +- Exchange (33)
: : : +- Project (32)
: : : +- BatchEvalPython (31)
: : : +- Project (30)
: : : +- Filter (29)
: : : +- BatchEvalPython (28)
: : : +- Filter (27)
: : : +- InMemoryTableScan (24)
: : : +- InMemoryRelation (25)
: : : +- AdaptiveSparkPlan (26)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Sort (48)
: : +- Exchange (47)
: : +- Project (46)
: : +- BroadcastHashJoin Inner BuildLeft (45)
: : :- BroadcastExchange (39)
: : : +- Filter (38)
: : : +- InMemoryTableScan (35)
: : : +- InMemoryRelation (36)
: : : +- AdaptiveSparkPlan (37)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Project (44)
: : +- Filter (43)
: : +- InMemoryTableScan (40)
: : +- InMemoryRelation (41)
: : +- AdaptiveSparkPlan (42)
: : +- Filter (18)
: : +- InMemoryTableScan (15)
: : +- InMemoryRelation (16)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
: +- Project (57)
: +- BatchEvalPython (56)
: +- Filter (55)
: +- InMemoryTableScan (52)
: +- InMemoryRelation (53)
: +- AdaptiveSparkPlan (54)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (143)
+- SortMergeJoin LeftAnti (142)
:- SortMergeJoin LeftAnti (92)
: :- Sort (77)
: : +- Exchange (76)
: : +- InMemoryTableScan (73)
: : +- InMemoryRelation (74)
: : +- AdaptiveSparkPlan (75)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (91)
: +- Exchange (90)
: +- Project (89)
: +- BroadcastHashJoin Inner BuildLeft (88)
: :- BroadcastExchange (82)
: : +- Filter (81)
: : +- InMemoryTableScan (78)
: : +- InMemoryRelation (79)
: : +- AdaptiveSparkPlan (80)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (87)
: +- Filter (86)
: +- InMemoryTableScan (83)
: +- InMemoryRelation (84)
: +- AdaptiveSparkPlan (85)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (141)
+- Filter (140)
+- Window (139)
+- WindowGroupLimit (138)
+- Sort (137)
+- Exchange (136)
+- WindowGroupLimit (135)
+- Sort (134)
+- Project (133)
+- BatchEvalPython (132)
+- Project (131)
+- Filter (130)
+- BatchEvalPython (129)
+- Project (128)
+- BroadcastHashJoin Inner BuildLeft (127)
:- BroadcastExchange (120)
: +- Project (119)
: +- SortMergeJoin LeftAnti (118)
: :- Sort (103)
: : +- Exchange (102)
: : +- Project (101)
: : +- BatchEvalPython (100)
: : +- Project (99)
: : +- Filter (98)
: : +- BatchEvalPython (97)
: : +- Filter (96)
: : +- InMemoryTableScan (93)
: : +- InMemoryRelation (94)
: : +- AdaptiveSparkPlan (95)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (117)
: +- Exchange (116)
: +- Project (115)
: +- BroadcastHashJoin Inner BuildLeft (114)
: :- BroadcastExchange (108)
: : +- Filter (107)
: : +- InMemoryTableScan (104)
: : +- InMemoryRelation (105)
: : +- AdaptiveSparkPlan (106)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (113)
: +- Filter (112)
: +- InMemoryTableScan (109)
: +- InMemoryRelation (110)
: +- AdaptiveSparkPlan (111)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (126)
+- BatchEvalPython (125)
+- Filter (124)
+- InMemoryTableScan (121)
+- InMemoryRelation (122)
+- AdaptiveSparkPlan (123)
+- Filter (18)
+- InMemoryTableScan (15)
+- InMemoryRelation (16)
+- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
(1) InMemoryTableScan
Output [1]: [match_reason#18146951]
Arguments: [match_reason#18146951]
(2) InMemoryRelation
Arguments: [brand_context_id#18144926, supplier_id#18144927, declaring_supplier_id#18144933, supplier_type#18144932, canonical_id#18146883, is_primary#18146947, configured_threshold#18146948, match_percentage#18146949, confidence_score#18146950, match_reason#18146951], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
Union
:- *(26) Project [brand_context_id#18144926, supplier_id#18144927, declaring_supplier_id#18144933, supplier_type#18144932, canonical_id#18146883, 0 AS is_primary#18146947, threshold#18144936 AS configured_threshold#18146948, 100.0 AS match_percentage#18146949, 1.0 AS confidence_score#18146950, exact_id_match AS match_reason#18146951]
: +- *(26) BroadcastHashJoin [brand_context_id#18144926, supplier_id#18144927], [brand_context_id#18146875, supplier_id#18146876], Inner, BuildLeft, false
: :- BroadcastQueryStage 13
: : +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4620893]
: : +- *(1) Filter (isnotnull(brand_context_id#18144926) AND isnotnull(supplier_id#18144927))
: : +- TableCacheQueryStage 0
: : +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936], [isnotnull(brand_context_id#18144926), isnotnull(supplier_id#18144927)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- *(26) Project [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883]
: +- *(26) Filter ((((isnotnull(is_primary#18146884) AND is_primary#18146884) AND isnotnull(canonical_id#18146883)) AND isnotnull(brand_context_id#18146875)) AND isnotnull(supplier_id#18146876))
: +- TableCacheQueryStage 1
: +- InMemoryTableScan [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876], [isnotnull(is_primary#18146884), is_primary#18146884, isnotnull(canonical_id#18146883), isnotnull(brand_context_id#18146875), isnotnull(supplier_id#18146876)]
: +- InMemoryRelation [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883, is_primary#18146884, entity_name#18146877, city#18146878, state#18146879, country#18146880, threshold#18146885, create_date#18146886], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
:- *(28) Project [u_brand_context_id#18146991 AS brand_context_id#18147112, u_supplier_id#18146992 AS supplier_id#18147113, u_declaring_supplier_id#18146993 AS declaring_supplier_id#18147114, u_supplier_type#18146994 AS supplier_type#18147115, e_canonical_id#18147015 AS canonical_id#18147116, 0 AS is_primary#18147117, u_threshold#18146995 AS configured_threshold#18147118, cast((similarity#18147072 * 100.0) as double) AS match_percentage#18147225, cast(similarity#18147072 as double) AS confidence_score#18147226, matched_existing AS match_reason#18147121]
: +- *(28) Filter (rank#18147092 = 1)
: +- Window [row_number() windowspecdefinition(u_brand_context_id#18146991, u_supplier_id#18146992, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#18147092], [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST]
: +- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Final
: +- *(27) Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
: +- ShuffleQueryStage 25
: +- Exchange hashpartitioning(u_brand_context_id#18146991, u_supplier_id#18146992, 10), ENSURE_REQUIREMENTS, [plan_id=4623863]
: +- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Partial
: +- *(19) Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
: +- *(19) Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, pythonUDF0#18148607 AS similarity#18147072]
: +- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148607]
: +- *(18) Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
: +- *(18) Filter (cast(pythonUDF0#18148606 as double) >= u_threshold#18146995)
: +- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148606]
: +- *(17) Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
: +- *(17) BroadcastHashJoin [u_brand_context_id#18146991, u_location_key#18147000], [e_brand_context_id#18147013, e_location_key#18147017], Inner, BuildLeft, false
: :- BroadcastQueryStage 23
: : +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[6, string, false]),false), [plan_id=4623233]
: : +- *(14) Project [brand_context_id#18147201 AS u_brand_context_id#18146991, supplier_id#18147202 AS u_supplier_id#18146992, declaring_supplier_id#18147208 AS u_declaring_supplier_id#18146993, supplier_type#18147207 AS u_supplier_type#18146994, threshold#18147211 AS u_threshold#18146995, normalized_name#18146458 AS u_normalized_name#18146996, location_key#18146515 AS u_location_key#18147000]
: : +- AQEShuffleRead coalesced
: : +- ShuffleQueryStage 14
: : +- Exchange hashpartitioning(brand_context_id#18147201, supplier_id#18147202, 10), ENSURE_REQUIREMENTS, [plan_id=4620975]
: : +- *(4) Project [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148598 AS normalized_name#18146458, concat_ws(|, coalesce(pythonUDF1#18148599, ), coalesce(pythonUDF2#18148600, ), coalesce(pythonUDF3#18148601, )) AS location_key#18146515]
: : +- BatchEvalPython [normalize_name(entity_name#18147203)#18146457, normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148598, pythonUDF1#18148599, pythonUDF2#18148600, pythonUDF3#18148601]
: : +- *(3) Project [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
: : +- *(3) Filter ((isnotnull(pythonUDF0#18148595) AND isnotnull(pythonUDF1#18148596)) AND isnotnull(pythonUDF2#18148597))
: : +- BatchEvalPython [normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148595, pythonUDF1#18148596, pythonUDF2#18148597]
: : +- *(2) Filter (isnotnull(threshold#18147211) AND isnotnull(brand_context_id#18147201))
: : +- TableCacheQueryStage 2
: : +- InMemoryTableScan [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211], [isnotnull(threshold#18147211), isnotnull(brand_context_id#18147201)]
: : +- InMemoryRelation [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, create_date#18147212], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- *(17) Project [brand_context_id#18147025 AS e_brand_context_id#18147013, supplier_id#18147026 AS e_supplier_id#18147014, canonical_id#18147033 AS e_canonical_id#18147015, pythonUDF0#18148602 AS e_normalized_name#18147016, concat_ws(|, coalesce(pythonUDF1#18148603, ), coalesce(pythonUDF2#18148604, ), coalesce(pythonUDF3#18148605, )) AS e_location_key#18147017, create_date#18147036 AS e_create_date#18147018]
: +- BatchEvalPython [normalize_name(entity_name#18147027)#18146531, normalize_location(city#18147028)#18146544, normalize_location(state#18147029)#18146558, normalize_location(country#18147030)#18146573], [pythonUDF0#18148602, pythonUDF1#18148603, pythonUDF2#18148604, pythonUDF3#18148605]
: +- *(16) Filter isnotnull(brand_context_id#18147025)
: +- TableCacheQueryStage 5
: +- InMemoryTableScan [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036], [isnotnull(brand_context_id#18147025)]
: +- InMemoryRelation [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, is_primary#18147034, entity_name#18147027, city#18147028, state#18147029, country#18147030, threshold#18147035, create_date#18147036], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- *(29) Project [brand_context_id#18147237, supplier_id#18147238, declaring_supplier_id#18147244, supplier_type#18147243, supplier_id#18147238 AS canonical_id#18147185, 1 AS is_primary#18147186, threshold#18147247 AS configured_threshold#18147187, 0.0 AS match_percentage#18147188, 0.0 AS confidence_score#18147189, new_canonical AS match_reason#18147190]
+- ShuffleQueryStage 16
+- Exchange hashpartitioning(brand_context_id#18147237, supplier_id#18147238, 10), ENSURE_REQUIREMENTS, [plan_id=4620487]
+- TableCacheQueryStage 6
+- InMemoryTableScan [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
+- InMemoryRelation [brand_context_id#18147237, supplier_id#18147238, entity_name#18147239, city#18147240, state#18147241, country#18147242, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247, create_date#18147248], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Union
:- Project [brand_context_id#18144926, supplier_id#18144927, declaring_supplier_id#18144933, supplier_type#18144932, canonical_id#18146883, 0 AS is_primary#18146947, threshold#18144936 AS configured_threshold#18146948, 100.0 AS match_percentage#18146949, 1.0 AS confidence_score#18146950, exact_id_match AS match_reason#18146951]
: +- BroadcastHashJoin [brand_context_id#18144926, supplier_id#18144927], [brand_context_id#18146875, supplier_id#18146876], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4620175]
: : +- Filter (isnotnull(brand_context_id#18144926) AND isnotnull(supplier_id#18144927))
: : +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936], [isnotnull(brand_context_id#18144926), isnotnull(supplier_id#18144927)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883]
: +- Filter ((((isnotnull(is_primary#18146884) AND is_primary#18146884) AND isnotnull(canonical_id#18146883)) AND isnotnull(brand_context_id#18146875)) AND isnotnull(supplier_id#18146876))
: +- InMemoryTableScan [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876], [isnotnull(is_primary#18146884), is_primary#18146884, isnotnull(canonical_id#18146883), isnotnull(brand_context_id#18146875), isnotnull(supplier_id#18146876)]
: +- InMemoryRelation [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883, is_primary#18146884, entity_name#18146877, city#18146878, state#18146879, country#18146880, threshold#18146885, create_date#18146886], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
:- Project [u_brand_context_id#18146991 AS brand_context_id#18147112, u_supplier_id#18146992 AS supplier_id#18147113, u_declaring_supplier_id#18146993 AS declaring_supplier_id#18147114, u_supplier_type#18146994 AS supplier_type#18147115, e_canonical_id#18147015 AS canonical_id#18147116, 0 AS is_primary#18147117, u_threshold#18146995 AS configured_threshold#18147118, cast((similarity#18147072 * 100.0) as double) AS match_percentage#18147225, cast(similarity#18147072 as double) AS confidence_score#18147226, matched_existing AS match_reason#18147121]
: +- Filter (rank#18147092 = 1)
: +- Window [row_number() windowspecdefinition(u_brand_context_id#18146991, u_supplier_id#18146992, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#18147092], [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST]
: +- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Final
: +- Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(u_brand_context_id#18146991, u_supplier_id#18146992, 10), ENSURE_REQUIREMENTS, [plan_id=4620202]
: +- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Partial
: +- Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
: +- Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, pythonUDF0#18148607 AS similarity#18147072]
: +- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148607]
: +- Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
: +- Filter (cast(pythonUDF0#18148606 as double) >= u_threshold#18146995)
: +- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148606]
: +- Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
: +- BroadcastHashJoin [u_brand_context_id#18146991, u_location_key#18147000], [e_brand_context_id#18147013, e_location_key#18147017], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[6, string, false]),false), [plan_id=4620190]
: : +- Project [brand_context_id#18147201 AS u_brand_context_id#18146991, supplier_id#18147202 AS u_supplier_id#18146992, declaring_supplier_id#18147208 AS u_declaring_supplier_id#18146993, supplier_type#18147207 AS u_supplier_type#18146994, threshold#18147211 AS u_threshold#18146995, normalized_name#18146458 AS u_normalized_name#18146996, location_key#18146515 AS u_location_key#18147000]
: : +- SortMergeJoin [brand_context_id#18147201, supplier_id#18147202], [brand_context_id#18146964, supplier_id#18146965], LeftAnti
: : :- Sort [brand_context_id#18147201 ASC NULLS FIRST, supplier_id#18147202 ASC NULLS FIRST], false, 0
: : : +- Exchange hashpartitioning(brand_context_id#18147201, supplier_id#18147202, 10), ENSURE_REQUIREMENTS, [plan_id=4620183]
: : : +- Project [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148598 AS normalized_name#18146458, concat_ws(|, coalesce(pythonUDF1#18148599, ), coalesce(pythonUDF2#18148600, ), coalesce(pythonUDF3#18148601, )) AS location_key#18146515]
: : : +- BatchEvalPython [normalize_name(entity_name#18147203)#18146457, normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148598, pythonUDF1#18148599, pythonUDF2#18148600, pythonUDF3#18148601]
: : : +- Project [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
: : : +- Filter ((isnotnull(pythonUDF0#18148595) AND isnotnull(pythonUDF1#18148596)) AND isnotnull(pythonUDF2#18148597))
: : : +- BatchEvalPython [normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148595, pythonUDF1#18148596, pythonUDF2#18148597]
: : : +- Filter (isnotnull(threshold#18147211) AND isnotnull(brand_context_id#18147201))
: : : +- InMemoryTableScan [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211], [isnotnull(threshold#18147211), isnotnull(brand_context_id#18147201)]
: : : +- InMemoryRelation [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, create_date#18147212], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- AdaptiveSparkPlan isFinalPlan=false
: : : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : : +- Filter isnull(canonical_id#18144934)
: : : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: : +- Sort [brand_context_id#18146964 ASC NULLS FIRST, supplier_id#18146965 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#18146964, supplier_id#18146965, 10), ENSURE_REQUIREMENTS, [plan_id=4620184]
: : +- Project [brand_context_id#18146964, supplier_id#18146965]
: : +- BroadcastHashJoin [brand_context_id#18146964, supplier_id#18146965], [brand_context_id#18147213, supplier_id#18147214], Inner, BuildLeft, false
: : :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4620178]
: : : +- Filter (isnotnull(brand_context_id#18146964) AND isnotnull(supplier_id#18146965))
: : : +- InMemoryTableScan [brand_context_id#18146964, supplier_id#18146965], [isnotnull(brand_context_id#18146964), isnotnull(supplier_id#18146965)]
: : : +- InMemoryRelation [brand_context_id#18146964, supplier_id#18146965, entity_name#18146966, city#18146967, state#18146968, country#18146969, supplier_type#18146970, declaring_supplier_id#18146971, threshold#18146974, create_date#18146975], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- AdaptiveSparkPlan isFinalPlan=false
: : : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : : +- Filter isnull(canonical_id#18144934)
: : : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: : +- Project [brand_context_id#18147213, supplier_id#18147214]
: : +- Filter ((((isnotnull(is_primary#18147222) AND is_primary#18147222) AND isnotnull(canonical_id#18147221)) AND isnotnull(brand_context_id#18147213)) AND isnotnull(supplier_id#18147214))
: : +- InMemoryTableScan [brand_context_id#18147213, canonical_id#18147221, is_primary#18147222, supplier_id#18147214], [isnotnull(is_primary#18147222), is_primary#18147222, isnotnull(canonical_id#18147221), isnotnull(brand_context_id#18147213), isnotnull(supplier_id#18147214)]
: : +- InMemoryRelation [brand_context_id#18147213, supplier_id#18147214, canonical_id#18147221, is_primary#18147222, entity_name#18147215, city#18147216, state#18147217, country#18147218, threshold#18147223, create_date#18147224], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Filter isnotnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#18147025 AS e_brand_context_id#18147013, supplier_id#18147026 AS e_supplier_id#18147014, canonical_id#18147033 AS e_canonical_id#18147015, pythonUDF0#18148602 AS e_normalized_name#18147016, concat_ws(|, coalesce(pythonUDF1#18148603, ), coalesce(pythonUDF2#18148604, ), coalesce(pythonUDF3#18148605, )) AS e_location_key#18147017, create_date#18147036 AS e_create_date#18147018]
: +- BatchEvalPython [normalize_name(entity_name#18147027)#18146531, normalize_location(city#18147028)#18146544, normalize_location(state#18147029)#18146558, normalize_location(country#18147030)#18146573], [pythonUDF0#18148602, pythonUDF1#18148603, pythonUDF2#18148604, pythonUDF3#18148605]
: +- Filter isnotnull(brand_context_id#18147025)
: +- InMemoryTableScan [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036], [isnotnull(brand_context_id#18147025)]
: +- InMemoryRelation [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, is_primary#18147034, entity_name#18147027, city#18147028, state#18147029, country#18147030, threshold#18147035, create_date#18147036], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- Project [brand_context_id#18147237, supplier_id#18147238, declaring_supplier_id#18147244, supplier_type#18147243, supplier_id#18147238 AS canonical_id#18147185, 1 AS is_primary#18147186, threshold#18147247 AS configured_threshold#18147187, 0.0 AS match_percentage#18147188, 0.0 AS confidence_score#18147189, new_canonical AS match_reason#18147190]
+- SortMergeJoin [brand_context_id#18147237, supplier_id#18147238], [brand_context_id#18147112, supplier_id#18147113], LeftAnti
:- SortMergeJoin [brand_context_id#18147237, supplier_id#18147238], [brand_context_id#18147249, supplier_id#18147250], LeftAnti
: :- Sort [brand_context_id#18147237 ASC NULLS FIRST, supplier_id#18147238 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#18147237, supplier_id#18147238, 10), ENSURE_REQUIREMENTS, [plan_id=4620213]
: : +- InMemoryTableScan [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
: : +- InMemoryRelation [brand_context_id#18147237, supplier_id#18147238, entity_name#18147239, city#18147240, state#18147241, country#18147242, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247, create_date#18147248], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Sort [brand_context_id#18147249 ASC NULLS FIRST, supplier_id#18147250 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(brand_context_id#18147249, supplier_id#18147250, 10), ENSURE_REQUIREMENTS, [plan_id=4620214]
: +- Project [brand_context_id#18147249, supplier_id#18147250]
: +- BroadcastHashJoin [brand_context_id#18147249, supplier_id#18147250], [brand_context_id#18147261, supplier_id#18147262], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4620208]
: : +- Filter (isnotnull(brand_context_id#18147249) AND isnotnull(supplier_id#18147250))
: : +- InMemoryTableScan [brand_context_id#18147249, supplier_id#18147250], [isnotnull(brand_context_id#18147249), isnotnull(supplier_id#18147250)]
: : +- InMemoryRelation [brand_context_id#18147249, supplier_id#18147250, entity_name#18147251, city#18147252, state#18147253, country#18147254, supplier_type#18147255, declaring_supplier_id#18147256, threshold#18147259, create_date#18147260], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#18147261, supplier_id#18147262]
: +- Filter ((((isnotnull(is_primary#18147270) AND is_primary#18147270) AND isnotnull(canonical_id#18147269)) AND isnotnull(brand_context_id#18147261)) AND isnotnull(supplier_id#18147262))
: +- InMemoryTableScan [brand_context_id#18147261, canonical_id#18147269, is_primary#18147270, supplier_id#18147262], [isnotnull(is_primary#18147270), is_primary#18147270, isnotnull(canonical_id#18147269), isnotnull(brand_context_id#18147261), isnotnull(supplier_id#18147262)]
: +- InMemoryRelation [brand_context_id#18147261, supplier_id#18147262, canonical_id#18147269, is_primary#18147270, entity_name#18147263, city#18147264, state#18147265, country#18147266, threshold#18147271, create_date#18147272], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- Project [u_brand_context_id#18146991 AS brand_context_id#18147112, u_supplier_id#18146992 AS supplier_id#18147113]
+- Filter (rank#18147092 = 1)
+- Window [row_number() windowspecdefinition(u_brand_context_id#18146991, u_supplier_id#18146992, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#18147092], [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST]
+- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Final
+- Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(u_brand_context_id#18146991, u_supplier_id#18146992, 10), ENSURE_REQUIREMENTS, [plan_id=4620242]
+- WindowGroupLimit [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Partial
+- Sort [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
+- Project [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, pythonUDF0#18148620 AS similarity#18147072]
+- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148620]
+- Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
+- Filter (cast(pythonUDF0#18148619 as double) >= u_threshold#18146995)
+- BatchEvalPython [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148619]
+- Project [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
+- BroadcastHashJoin [u_brand_context_id#18146991, u_location_key#18147000], [e_brand_context_id#18147013, e_location_key#18147017], Inner, BuildLeft, false
:- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[4, string, false]),false), [plan_id=4620230]
: +- Project [brand_context_id#18147134 AS u_brand_context_id#18146991, supplier_id#18147135 AS u_supplier_id#18146992, threshold#18147144 AS u_threshold#18146995, normalized_name#18146458 AS u_normalized_name#18146996, location_key#18146515 AS u_location_key#18147000]
: +- SortMergeJoin [brand_context_id#18147134, supplier_id#18147135], [brand_context_id#18147146, supplier_id#18147147], LeftAnti
: :- Sort [brand_context_id#18147134 ASC NULLS FIRST, supplier_id#18147135 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#18147134, supplier_id#18147135, 10), ENSURE_REQUIREMENTS, [plan_id=4620223]
: : +- Project [brand_context_id#18147134, supplier_id#18147135, threshold#18147144, pythonUDF0#18148611 AS normalized_name#18146458, concat_ws(|, coalesce(pythonUDF1#18148612, ), coalesce(pythonUDF2#18148613, ), coalesce(pythonUDF3#18148614, )) AS location_key#18146515]
: : +- BatchEvalPython [normalize_name(entity_name#18147136)#18146457, normalize_location(city#18147137)#18146470, normalize_location(state#18147138)#18146484, normalize_location(country#18147139)#18146499], [pythonUDF0#18148611, pythonUDF1#18148612, pythonUDF2#18148613, pythonUDF3#18148614]
: : +- Project [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
: : +- Filter ((isnotnull(pythonUDF0#18148608) AND isnotnull(pythonUDF1#18148609)) AND isnotnull(pythonUDF2#18148610))
: : +- BatchEvalPython [normalize_location(city#18147137)#18146470, normalize_location(state#18147138)#18146484, normalize_location(country#18147139)#18146499], [pythonUDF0#18148608, pythonUDF1#18148609, pythonUDF2#18148610]
: : +- Filter ((isnotnull(threshold#18147144) AND isnotnull(brand_context_id#18147134)) AND isnotnull(supplier_id#18147135))
: : +- InMemoryTableScan [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144], [isnotnull(threshold#18147144), isnotnull(brand_context_id#18147134), isnotnull(supplier_id#18147135)]
: : +- InMemoryRelation [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, supplier_type#18147140, declaring_supplier_id#18147141, threshold#18147144, create_date#18147145], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Sort [brand_context_id#18147146 ASC NULLS FIRST, supplier_id#18147147 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(brand_context_id#18147146, supplier_id#18147147, 10), ENSURE_REQUIREMENTS, [plan_id=4620224]
: +- Project [brand_context_id#18147146, supplier_id#18147147]
: +- BroadcastHashJoin [brand_context_id#18147146, supplier_id#18147147], [brand_context_id#18147158, supplier_id#18147159], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4620218]
: : +- Filter (isnotnull(brand_context_id#18147146) AND isnotnull(supplier_id#18147147))
: : +- InMemoryTableScan [brand_context_id#18147146, supplier_id#18147147], [isnotnull(brand_context_id#18147146), isnotnull(supplier_id#18147147)]
: : +- InMemoryRelation [brand_context_id#18147146, supplier_id#18147147, entity_name#18147148, city#18147149, state#18147150, country#18147151, supplier_type#18147152, declaring_supplier_id#18147153, threshold#18147156, create_date#18147157], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
: : +- Filter isnull(canonical_id#18144934)
: : +- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
: : +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#18147158, supplier_id#18147159]
: +- Filter ((((isnotnull(is_primary#18147167) AND is_primary#18147167) AND isnotnull(canonical_id#18147166)) AND isnotnull(brand_context_id#18147158)) AND isnotnull(supplier_id#18147159))
: +- InMemoryTableScan [brand_context_id#18147158, canonical_id#18147166, is_primary#18147167, supplier_id#18147159], [isnotnull(is_primary#18147167), is_primary#18147167, isnotnull(canonical_id#18147166), isnotnull(brand_context_id#18147158), isnotnull(supplier_id#18147159)]
: +- InMemoryRelation [brand_context_id#18147158, supplier_id#18147159, canonical_id#18147166, is_primary#18147167, entity_name#18147160, city#18147161, state#18147162, country#18147163, threshold#18147168, create_date#18147169], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#18144934)
: +- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
: +- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- Project [brand_context_id#18147273 AS e_brand_context_id#18147013, supplier_id#18147274 AS e_supplier_id#18147014, pythonUDF0#18148615 AS e_normalized_name#18147016, concat_ws(|, coalesce(pythonUDF1#18148616, ), coalesce(pythonUDF2#18148617, ), coalesce(pythonUDF3#18148618, )) AS e_location_key#18147017, create_date#18147284 AS e_create_date#18147018]
+- BatchEvalPython [normalize_name(entity_name#18147275)#18146531, normalize_location(city#18147276)#18146544, normalize_location(state#18147277)#18146558, normalize_location(country#18147278)#18146573], [pythonUDF0#18148615, pythonUDF1#18148616, pythonUDF2#18148617, pythonUDF3#18148618]
+- Filter isnotnull(brand_context_id#18147273)
+- InMemoryTableScan [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284], [isnotnull(brand_context_id#18147273)]
+- InMemoryRelation [brand_context_id#18147273, supplier_id#18147274, canonical_id#18147281, is_primary#18147282, entity_name#18147275, city#18147276, state#18147277, country#18147278, threshold#18147283, create_date#18147284], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(3) InMemoryTableScan
Output [5]: [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936]
Arguments: [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936], [isnotnull(brand_context_id#18144926), isnotnull(supplier_id#18144927)]
(4) InMemoryRelation
Arguments: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(5) InMemoryTableScan
Output [11]: [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936]
Arguments: [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
(6) InMemoryRelation
Arguments: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(7) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [codegen id : 1]
Output [12]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937]
ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,country:string,supplier_type:string,declaring_supplier_id:string,canonical_id:string,is_primary:boolean,threshold:double,create_date:date>
(8) Filter
Input [11]: [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936]
Condition : isnull(canonical_id#18144934)
(9) Project
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Input [11]: [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936]
(10) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(11) Filter
Input [5]: [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936]
Condition : (isnotnull(brand_context_id#18144926) AND isnotnull(supplier_id#18144927))
(12) BroadcastExchange
Input [5]: [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4624289]
(13) InMemoryTableScan
Output [4]: [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876]
Arguments: [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876], [isnotnull(is_primary#18146884), is_primary#18146884, isnotnull(canonical_id#18146883), isnotnull(brand_context_id#18146875), isnotnull(supplier_id#18146876)]
(14) InMemoryRelation
Arguments: [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883, is_primary#18146884, entity_name#18146877, city#18146878, state#18146879, country#18146880, threshold#18146885, create_date#18146886], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(15) InMemoryTableScan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
(16) InMemoryRelation
Arguments: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(17) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [codegen id : 1]
Output [12]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937]
ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,country:string,supplier_type:string,declaring_supplier_id:string,canonical_id:string,is_primary:boolean,threshold:double,create_date:date>
(18) Filter
Input [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Condition : isnotnull(canonical_id#18144934)
(19) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(20) Filter
Input [4]: [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876]
Condition : ((((isnotnull(is_primary#18146884) AND is_primary#18146884) AND isnotnull(canonical_id#18146883)) AND isnotnull(brand_context_id#18146875)) AND isnotnull(supplier_id#18146876))
(21) Project
Output [3]: [brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883]
Input [4]: [brand_context_id#18146875, canonical_id#18146883, is_primary#18146884, supplier_id#18146876]
(22) BroadcastHashJoin
Left keys [2]: [brand_context_id#18144926, supplier_id#18144927]
Right keys [2]: [brand_context_id#18146875, supplier_id#18146876]
Join type: Inner
Join condition: None
(23) Project
Output [10]: [brand_context_id#18144926, supplier_id#18144927, declaring_supplier_id#18144933, supplier_type#18144932, canonical_id#18146883, 0 AS is_primary#18146947, threshold#18144936 AS configured_threshold#18146948, 100.0 AS match_percentage#18146949, 1.0 AS confidence_score#18146950, exact_id_match AS match_reason#18146951]
Input [8]: [brand_context_id#18144926, supplier_id#18144927, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, brand_context_id#18146875, supplier_id#18146876, canonical_id#18146883]
(24) InMemoryTableScan
Output [9]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
Arguments: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211], [isnotnull(threshold#18147211), isnotnull(brand_context_id#18147201)]
(25) InMemoryRelation
Arguments: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, create_date#18147212], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(26) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(27) Filter
Input [9]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
Condition : (isnotnull(threshold#18147211) AND isnotnull(brand_context_id#18147201))
(28) BatchEvalPython
Input [9]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
Arguments: [normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148595, pythonUDF1#18148596, pythonUDF2#18148597]
(29) Filter
Input [12]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148595, pythonUDF1#18148596, pythonUDF2#18148597]
Condition : ((isnotnull(pythonUDF0#18148595) AND isnotnull(pythonUDF1#18148596)) AND isnotnull(pythonUDF2#18148597))
(30) Project
Output [9]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
Input [12]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148595, pythonUDF1#18148596, pythonUDF2#18148597]
(31) BatchEvalPython
Input [9]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211]
Arguments: [normalize_name(entity_name#18147203)#18146457, normalize_location(city#18147204)#18146470, normalize_location(state#18147205)#18146484, normalize_location(country#18147206)#18146499], [pythonUDF0#18148598, pythonUDF1#18148599, pythonUDF2#18148600, pythonUDF3#18148601]
(32) Project
Output [7]: [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148598 AS normalized_name#18146458, concat_ws(|, coalesce(pythonUDF1#18148599, ), coalesce(pythonUDF2#18148600, ), coalesce(pythonUDF3#18148601, )) AS location_key#18146515]
Input [13]: [brand_context_id#18147201, supplier_id#18147202, entity_name#18147203, city#18147204, state#18147205, country#18147206, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, pythonUDF0#18148598, pythonUDF1#18148599, pythonUDF2#18148600, pythonUDF3#18148601]
(33) Exchange
Input [7]: [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, normalized_name#18146458, location_key#18146515]
Arguments: hashpartitioning(brand_context_id#18147201, supplier_id#18147202, 10), ENSURE_REQUIREMENTS, [plan_id=4624297]
(34) Sort
Input [7]: [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, normalized_name#18146458, location_key#18146515]
Arguments: [brand_context_id#18147201 ASC NULLS FIRST, supplier_id#18147202 ASC NULLS FIRST], false, 0
(35) InMemoryTableScan
Output [2]: [brand_context_id#18146964, supplier_id#18146965]
Arguments: [brand_context_id#18146964, supplier_id#18146965], [isnotnull(brand_context_id#18146964), isnotnull(supplier_id#18146965)]
(36) InMemoryRelation
Arguments: [brand_context_id#18146964, supplier_id#18146965, entity_name#18146966, city#18146967, state#18146968, country#18146969, supplier_type#18146970, declaring_supplier_id#18146971, threshold#18146974, create_date#18146975], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(37) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(38) Filter
Input [2]: [brand_context_id#18146964, supplier_id#18146965]
Condition : (isnotnull(brand_context_id#18146964) AND isnotnull(supplier_id#18146965))
(39) BroadcastExchange
Input [2]: [brand_context_id#18146964, supplier_id#18146965]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4624292]
(40) InMemoryTableScan
Output [4]: [brand_context_id#18147213, canonical_id#18147221, is_primary#18147222, supplier_id#18147214]
Arguments: [brand_context_id#18147213, canonical_id#18147221, is_primary#18147222, supplier_id#18147214], [isnotnull(is_primary#18147222), is_primary#18147222, isnotnull(canonical_id#18147221), isnotnull(brand_context_id#18147213), isnotnull(supplier_id#18147214)]
(41) InMemoryRelation
Arguments: [brand_context_id#18147213, supplier_id#18147214, canonical_id#18147221, is_primary#18147222, entity_name#18147215, city#18147216, state#18147217, country#18147218, threshold#18147223, create_date#18147224], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(42) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(43) Filter
Input [4]: [brand_context_id#18147213, canonical_id#18147221, is_primary#18147222, supplier_id#18147214]
Condition : ((((isnotnull(is_primary#18147222) AND is_primary#18147222) AND isnotnull(canonical_id#18147221)) AND isnotnull(brand_context_id#18147213)) AND isnotnull(supplier_id#18147214))
(44) Project
Output [2]: [brand_context_id#18147213, supplier_id#18147214]
Input [4]: [brand_context_id#18147213, canonical_id#18147221, is_primary#18147222, supplier_id#18147214]
(45) BroadcastHashJoin
Left keys [2]: [brand_context_id#18146964, supplier_id#18146965]
Right keys [2]: [brand_context_id#18147213, supplier_id#18147214]
Join type: Inner
Join condition: None
(46) Project
Output [2]: [brand_context_id#18146964, supplier_id#18146965]
Input [4]: [brand_context_id#18146964, supplier_id#18146965, brand_context_id#18147213, supplier_id#18147214]
(47) Exchange
Input [2]: [brand_context_id#18146964, supplier_id#18146965]
Arguments: hashpartitioning(brand_context_id#18146964, supplier_id#18146965, 10), ENSURE_REQUIREMENTS, [plan_id=4624298]
(48) Sort
Input [2]: [brand_context_id#18146964, supplier_id#18146965]
Arguments: [brand_context_id#18146964 ASC NULLS FIRST, supplier_id#18146965 ASC NULLS FIRST], false, 0
(49) SortMergeJoin
Left keys [2]: [brand_context_id#18147201, supplier_id#18147202]
Right keys [2]: [brand_context_id#18146964, supplier_id#18146965]
Join type: LeftAnti
Join condition: None
(50) Project
Output [7]: [brand_context_id#18147201 AS u_brand_context_id#18146991, supplier_id#18147202 AS u_supplier_id#18146992, declaring_supplier_id#18147208 AS u_declaring_supplier_id#18146993, supplier_type#18147207 AS u_supplier_type#18146994, threshold#18147211 AS u_threshold#18146995, normalized_name#18146458 AS u_normalized_name#18146996, location_key#18146515 AS u_location_key#18147000]
Input [7]: [brand_context_id#18147201, supplier_id#18147202, supplier_type#18147207, declaring_supplier_id#18147208, threshold#18147211, normalized_name#18146458, location_key#18146515]
(51) BroadcastExchange
Input [7]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, u_location_key#18147000]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true], input[6, string, false]),false), [plan_id=4624304]
(52) InMemoryTableScan
Output [8]: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036]
Arguments: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036], [isnotnull(brand_context_id#18147025)]
(53) InMemoryRelation
Arguments: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, is_primary#18147034, entity_name#18147027, city#18147028, state#18147029, country#18147030, threshold#18147035, create_date#18147036], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(54) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(55) Filter
Input [8]: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036]
Condition : isnotnull(brand_context_id#18147025)
(56) BatchEvalPython
Input [8]: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036]
Arguments: [normalize_name(entity_name#18147027)#18146531, normalize_location(city#18147028)#18146544, normalize_location(state#18147029)#18146558, normalize_location(country#18147030)#18146573], [pythonUDF0#18148602, pythonUDF1#18148603, pythonUDF2#18148604, pythonUDF3#18148605]
(57) Project
Output [6]: [brand_context_id#18147025 AS e_brand_context_id#18147013, supplier_id#18147026 AS e_supplier_id#18147014, canonical_id#18147033 AS e_canonical_id#18147015, pythonUDF0#18148602 AS e_normalized_name#18147016, concat_ws(|, coalesce(pythonUDF1#18148603, ), coalesce(pythonUDF2#18148604, ), coalesce(pythonUDF3#18148605, )) AS e_location_key#18147017, create_date#18147036 AS e_create_date#18147018]
Input [12]: [brand_context_id#18147025, supplier_id#18147026, canonical_id#18147033, entity_name#18147027, city#18147028, state#18147029, country#18147030, create_date#18147036, pythonUDF0#18148602, pythonUDF1#18148603, pythonUDF2#18148604, pythonUDF3#18148605]
(58) BroadcastHashJoin
Left keys [2]: [u_brand_context_id#18146991, u_location_key#18147000]
Right keys [2]: [e_brand_context_id#18147013, e_location_key#18147017]
Join type: Inner
Join condition: None
(59) Project
Output [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
Input [13]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, u_location_key#18147000, e_brand_context_id#18147013, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_location_key#18147017, e_create_date#18147018]
(60) BatchEvalPython
Input [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
Arguments: [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148606]
(61) Filter
Input [11]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148606]
Condition : (cast(pythonUDF0#18148606 as double) >= u_threshold#18146995)
(62) Project
Output [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
Input [11]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148606]
(63) BatchEvalPython
Input [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018]
Arguments: [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148607]
(64) Project
Output [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, pythonUDF0#18148607 AS similarity#18147072]
Input [11]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_canonical_id#18147015, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148607]
(65) Sort
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
(66) WindowGroupLimit
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Partial
(67) Exchange
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: hashpartitioning(u_brand_context_id#18146991, u_supplier_id#18146992, 10), ENSURE_REQUIREMENTS, [plan_id=4624316]
(68) Sort
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
(69) WindowGroupLimit
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Final
(70) Window
Input [9]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072]
Arguments: [row_number() windowspecdefinition(u_brand_context_id#18146991, u_supplier_id#18146992, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#18147092], [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST]
(71) Filter
Input [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072, rank#18147092]
Condition : (rank#18147092 = 1)
(72) Project
Output [10]: [u_brand_context_id#18146991 AS brand_context_id#18147112, u_supplier_id#18146992 AS supplier_id#18147113, u_declaring_supplier_id#18146993 AS declaring_supplier_id#18147114, u_supplier_type#18146994 AS supplier_type#18147115, e_canonical_id#18147015 AS canonical_id#18147116, 0 AS is_primary#18147117, u_threshold#18146995 AS configured_threshold#18147118, cast((similarity#18147072 * 100.0) as double) AS match_percentage#18147225, cast(similarity#18147072 as double) AS confidence_score#18147226, matched_existing AS match_reason#18147121]
Input [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_declaring_supplier_id#18146993, u_supplier_type#18146994, u_threshold#18146995, e_supplier_id#18147014, e_canonical_id#18147015, e_create_date#18147018, similarity#18147072, rank#18147092]
(73) InMemoryTableScan
Output [5]: [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
Arguments: [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
(74) InMemoryRelation
Arguments: [brand_context_id#18147237, supplier_id#18147238, entity_name#18147239, city#18147240, state#18147241, country#18147242, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247, create_date#18147248], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(75) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(76) Exchange
Input [5]: [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
Arguments: hashpartitioning(brand_context_id#18147237, supplier_id#18147238, 10), ENSURE_REQUIREMENTS, [plan_id=4624327]
(77) Sort
Input [5]: [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
Arguments: [brand_context_id#18147237 ASC NULLS FIRST, supplier_id#18147238 ASC NULLS FIRST], false, 0
(78) InMemoryTableScan
Output [2]: [brand_context_id#18147249, supplier_id#18147250]
Arguments: [brand_context_id#18147249, supplier_id#18147250], [isnotnull(brand_context_id#18147249), isnotnull(supplier_id#18147250)]
(79) InMemoryRelation
Arguments: [brand_context_id#18147249, supplier_id#18147250, entity_name#18147251, city#18147252, state#18147253, country#18147254, supplier_type#18147255, declaring_supplier_id#18147256, threshold#18147259, create_date#18147260], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(80) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(81) Filter
Input [2]: [brand_context_id#18147249, supplier_id#18147250]
Condition : (isnotnull(brand_context_id#18147249) AND isnotnull(supplier_id#18147250))
(82) BroadcastExchange
Input [2]: [brand_context_id#18147249, supplier_id#18147250]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4624322]
(83) InMemoryTableScan
Output [4]: [brand_context_id#18147261, canonical_id#18147269, is_primary#18147270, supplier_id#18147262]
Arguments: [brand_context_id#18147261, canonical_id#18147269, is_primary#18147270, supplier_id#18147262], [isnotnull(is_primary#18147270), is_primary#18147270, isnotnull(canonical_id#18147269), isnotnull(brand_context_id#18147261), isnotnull(supplier_id#18147262)]
(84) InMemoryRelation
Arguments: [brand_context_id#18147261, supplier_id#18147262, canonical_id#18147269, is_primary#18147270, entity_name#18147263, city#18147264, state#18147265, country#18147266, threshold#18147271, create_date#18147272], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(85) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(86) Filter
Input [4]: [brand_context_id#18147261, canonical_id#18147269, is_primary#18147270, supplier_id#18147262]
Condition : ((((isnotnull(is_primary#18147270) AND is_primary#18147270) AND isnotnull(canonical_id#18147269)) AND isnotnull(brand_context_id#18147261)) AND isnotnull(supplier_id#18147262))
(87) Project
Output [2]: [brand_context_id#18147261, supplier_id#18147262]
Input [4]: [brand_context_id#18147261, canonical_id#18147269, is_primary#18147270, supplier_id#18147262]
(88) BroadcastHashJoin
Left keys [2]: [brand_context_id#18147249, supplier_id#18147250]
Right keys [2]: [brand_context_id#18147261, supplier_id#18147262]
Join type: Inner
Join condition: None
(89) Project
Output [2]: [brand_context_id#18147249, supplier_id#18147250]
Input [4]: [brand_context_id#18147249, supplier_id#18147250, brand_context_id#18147261, supplier_id#18147262]
(90) Exchange
Input [2]: [brand_context_id#18147249, supplier_id#18147250]
Arguments: hashpartitioning(brand_context_id#18147249, supplier_id#18147250, 10), ENSURE_REQUIREMENTS, [plan_id=4624328]
(91) Sort
Input [2]: [brand_context_id#18147249, supplier_id#18147250]
Arguments: [brand_context_id#18147249 ASC NULLS FIRST, supplier_id#18147250 ASC NULLS FIRST], false, 0
(92) SortMergeJoin
Left keys [2]: [brand_context_id#18147237, supplier_id#18147238]
Right keys [2]: [brand_context_id#18147249, supplier_id#18147250]
Join type: LeftAnti
Join condition: None
(93) InMemoryTableScan
Output [7]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
Arguments: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144], [isnotnull(threshold#18147144), isnotnull(brand_context_id#18147134), isnotnull(supplier_id#18147135)]
(94) InMemoryRelation
Arguments: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, supplier_type#18147140, declaring_supplier_id#18147141, threshold#18147144, create_date#18147145], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(95) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(96) Filter
Input [7]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
Condition : ((isnotnull(threshold#18147144) AND isnotnull(brand_context_id#18147134)) AND isnotnull(supplier_id#18147135))
(97) BatchEvalPython
Input [7]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
Arguments: [normalize_location(city#18147137)#18146470, normalize_location(state#18147138)#18146484, normalize_location(country#18147139)#18146499], [pythonUDF0#18148608, pythonUDF1#18148609, pythonUDF2#18148610]
(98) Filter
Input [10]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144, pythonUDF0#18148608, pythonUDF1#18148609, pythonUDF2#18148610]
Condition : ((isnotnull(pythonUDF0#18148608) AND isnotnull(pythonUDF1#18148609)) AND isnotnull(pythonUDF2#18148610))
(99) Project
Output [7]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
Input [10]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144, pythonUDF0#18148608, pythonUDF1#18148609, pythonUDF2#18148610]
(100) BatchEvalPython
Input [7]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144]
Arguments: [normalize_name(entity_name#18147136)#18146457, normalize_location(city#18147137)#18146470, normalize_location(state#18147138)#18146484, normalize_location(country#18147139)#18146499], [pythonUDF0#18148611, pythonUDF1#18148612, pythonUDF2#18148613, pythonUDF3#18148614]
(101) Project
Output [5]: [brand_context_id#18147134, supplier_id#18147135, threshold#18147144, pythonUDF0#18148611 AS normalized_name#18146458, concat_ws(|, coalesce(pythonUDF1#18148612, ), coalesce(pythonUDF2#18148613, ), coalesce(pythonUDF3#18148614, )) AS location_key#18146515]
Input [11]: [brand_context_id#18147134, supplier_id#18147135, entity_name#18147136, city#18147137, state#18147138, country#18147139, threshold#18147144, pythonUDF0#18148611, pythonUDF1#18148612, pythonUDF2#18148613, pythonUDF3#18148614]
(102) Exchange
Input [5]: [brand_context_id#18147134, supplier_id#18147135, threshold#18147144, normalized_name#18146458, location_key#18146515]
Arguments: hashpartitioning(brand_context_id#18147134, supplier_id#18147135, 10), ENSURE_REQUIREMENTS, [plan_id=4624337]
(103) Sort
Input [5]: [brand_context_id#18147134, supplier_id#18147135, threshold#18147144, normalized_name#18146458, location_key#18146515]
Arguments: [brand_context_id#18147134 ASC NULLS FIRST, supplier_id#18147135 ASC NULLS FIRST], false, 0
(104) InMemoryTableScan
Output [2]: [brand_context_id#18147146, supplier_id#18147147]
Arguments: [brand_context_id#18147146, supplier_id#18147147], [isnotnull(brand_context_id#18147146), isnotnull(supplier_id#18147147)]
(105) InMemoryRelation
Arguments: [brand_context_id#18147146, supplier_id#18147147, entity_name#18147148, city#18147149, state#18147150, country#18147151, supplier_type#18147152, declaring_supplier_id#18147153, threshold#18147156, create_date#18147157], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- *(1) Filter isnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
+- Filter isnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, canonical_id#18144934, city#18144929, country#18144931, create_date#18144937, declaring_supplier_id#18144933, entity_name#18144928, state#18144930, supplier_id#18144927, supplier_type#18144932, threshold#18144936], [isnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(106) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(107) Filter
Input [2]: [brand_context_id#18147146, supplier_id#18147147]
Condition : (isnotnull(brand_context_id#18147146) AND isnotnull(supplier_id#18147147))
(108) BroadcastExchange
Input [2]: [brand_context_id#18147146, supplier_id#18147147]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4624332]
(109) InMemoryTableScan
Output [4]: [brand_context_id#18147158, canonical_id#18147166, is_primary#18147167, supplier_id#18147159]
Arguments: [brand_context_id#18147158, canonical_id#18147166, is_primary#18147167, supplier_id#18147159], [isnotnull(is_primary#18147167), is_primary#18147167, isnotnull(canonical_id#18147166), isnotnull(brand_context_id#18147158), isnotnull(supplier_id#18147159)]
(110) InMemoryRelation
Arguments: [brand_context_id#18147158, supplier_id#18147159, canonical_id#18147166, is_primary#18147167, entity_name#18147160, city#18147161, state#18147162, country#18147163, threshold#18147168, create_date#18147169], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(111) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(112) Filter
Input [4]: [brand_context_id#18147158, canonical_id#18147166, is_primary#18147167, supplier_id#18147159]
Condition : ((((isnotnull(is_primary#18147167) AND is_primary#18147167) AND isnotnull(canonical_id#18147166)) AND isnotnull(brand_context_id#18147158)) AND isnotnull(supplier_id#18147159))
(113) Project
Output [2]: [brand_context_id#18147158, supplier_id#18147159]
Input [4]: [brand_context_id#18147158, canonical_id#18147166, is_primary#18147167, supplier_id#18147159]
(114) BroadcastHashJoin
Left keys [2]: [brand_context_id#18147146, supplier_id#18147147]
Right keys [2]: [brand_context_id#18147158, supplier_id#18147159]
Join type: Inner
Join condition: None
(115) Project
Output [2]: [brand_context_id#18147146, supplier_id#18147147]
Input [4]: [brand_context_id#18147146, supplier_id#18147147, brand_context_id#18147158, supplier_id#18147159]
(116) Exchange
Input [2]: [brand_context_id#18147146, supplier_id#18147147]
Arguments: hashpartitioning(brand_context_id#18147146, supplier_id#18147147, 10), ENSURE_REQUIREMENTS, [plan_id=4624338]
(117) Sort
Input [2]: [brand_context_id#18147146, supplier_id#18147147]
Arguments: [brand_context_id#18147146 ASC NULLS FIRST, supplier_id#18147147 ASC NULLS FIRST], false, 0
(118) SortMergeJoin
Left keys [2]: [brand_context_id#18147134, supplier_id#18147135]
Right keys [2]: [brand_context_id#18147146, supplier_id#18147147]
Join type: LeftAnti
Join condition: None
(119) Project
Output [5]: [brand_context_id#18147134 AS u_brand_context_id#18146991, supplier_id#18147135 AS u_supplier_id#18146992, threshold#18147144 AS u_threshold#18146995, normalized_name#18146458 AS u_normalized_name#18146996, location_key#18146515 AS u_location_key#18147000]
Input [5]: [brand_context_id#18147134, supplier_id#18147135, threshold#18147144, normalized_name#18146458, location_key#18146515]
(120) BroadcastExchange
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, u_location_key#18147000]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true], input[4, string, false]),false), [plan_id=4624344]
(121) InMemoryTableScan
Output [7]: [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284]
Arguments: [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284], [isnotnull(brand_context_id#18147273)]
(122) InMemoryRelation
Arguments: [brand_context_id#18147273, supplier_id#18147274, canonical_id#18147281, is_primary#18147282, entity_name#18147275, city#18147276, state#18147277, country#18147278, threshold#18147283, create_date#18147284], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#18144934)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#18144934)
+- InMemoryTableScan [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937], [isnotnull(canonical_id#18144934)]
+- InMemoryRelation [brand_context_id#18144926, supplier_id#18144927, entity_name#18144928, city#18144929, state#18144930, country#18144931, supplier_type#18144932, declaring_supplier_id#18144933, canonical_id#18144934, is_primary#18144935, threshold#18144936, create_date#18144937], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#18144926,supplier_id#18144927,entity_name#18144928,city#18144929,state#18144930,country#18144931,supplier_type#18144932,declaring_supplier_id#18144933,canonical_id#18144934,is_primary#18144935,threshold#18144936,create_date#18144937] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(123) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, canonical_id#18144934, is_primary#18144935, entity_name#18144928, city#18144929, state#18144930, country#18144931, threshold#18144936, create_date#18144937]
Arguments: isFinalPlan=false
(124) Filter
Input [7]: [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284]
Condition : isnotnull(brand_context_id#18147273)
(125) BatchEvalPython
Input [7]: [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284]
Arguments: [normalize_name(entity_name#18147275)#18146531, normalize_location(city#18147276)#18146544, normalize_location(state#18147277)#18146558, normalize_location(country#18147278)#18146573], [pythonUDF0#18148615, pythonUDF1#18148616, pythonUDF2#18148617, pythonUDF3#18148618]
(126) Project
Output [5]: [brand_context_id#18147273 AS e_brand_context_id#18147013, supplier_id#18147274 AS e_supplier_id#18147014, pythonUDF0#18148615 AS e_normalized_name#18147016, concat_ws(|, coalesce(pythonUDF1#18148616, ), coalesce(pythonUDF2#18148617, ), coalesce(pythonUDF3#18148618, )) AS e_location_key#18147017, create_date#18147284 AS e_create_date#18147018]
Input [11]: [brand_context_id#18147273, supplier_id#18147274, entity_name#18147275, city#18147276, state#18147277, country#18147278, create_date#18147284, pythonUDF0#18148615, pythonUDF1#18148616, pythonUDF2#18148617, pythonUDF3#18148618]
(127) BroadcastHashJoin
Left keys [2]: [u_brand_context_id#18146991, u_location_key#18147000]
Right keys [2]: [e_brand_context_id#18147013, e_location_key#18147017]
Join type: Inner
Join condition: None
(128) Project
Output [7]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
Input [10]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, u_location_key#18147000, e_brand_context_id#18147013, e_supplier_id#18147014, e_normalized_name#18147016, e_location_key#18147017, e_create_date#18147018]
(129) BatchEvalPython
Input [7]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
Arguments: [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148619]
(130) Filter
Input [8]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148619]
Condition : (cast(pythonUDF0#18148619 as double) >= u_threshold#18146995)
(131) Project
Output [6]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
Input [8]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_threshold#18146995, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148619]
(132) BatchEvalPython
Input [6]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018]
Arguments: [calculate_similarity(u_normalized_name#18146996, e_normalized_name#18147016)#18147071], [pythonUDF0#18148620]
(133) Project
Output [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, pythonUDF0#18148620 AS similarity#18147072]
Input [7]: [u_brand_context_id#18146991, u_supplier_id#18146992, u_normalized_name#18146996, e_supplier_id#18147014, e_normalized_name#18147016, e_create_date#18147018, pythonUDF0#18148620]
(134) Sort
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
(135) WindowGroupLimit
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Partial
(136) Exchange
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: hashpartitioning(u_brand_context_id#18146991, u_supplier_id#18146992, 10), ENSURE_REQUIREMENTS, [plan_id=4624356]
(137) Sort
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991 ASC NULLS FIRST, u_supplier_id#18146992 ASC NULLS FIRST, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], false, 0
(138) WindowGroupLimit
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST], row_number(), 1, Final
(139) Window
Input [5]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072]
Arguments: [row_number() windowspecdefinition(u_brand_context_id#18146991, u_supplier_id#18146992, similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#18147092], [u_brand_context_id#18146991, u_supplier_id#18146992], [similarity#18147072 DESC NULLS LAST, e_create_date#18147018 ASC NULLS FIRST, e_supplier_id#18147014 ASC NULLS FIRST]
(140) Filter
Input [6]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072, rank#18147092]
Condition : (rank#18147092 = 1)
(141) Project
Output [2]: [u_brand_context_id#18146991 AS brand_context_id#18147112, u_supplier_id#18146992 AS supplier_id#18147113]
Input [6]: [u_brand_context_id#18146991, u_supplier_id#18146992, e_supplier_id#18147014, e_create_date#18147018, similarity#18147072, rank#18147092]
(142) SortMergeJoin
Left keys [2]: [brand_context_id#18147237, supplier_id#18147238]
Right keys [2]: [brand_context_id#18147112, supplier_id#18147113]
Join type: LeftAnti
Join condition: None
(143) Project
Output [10]: [brand_context_id#18147237, supplier_id#18147238, declaring_supplier_id#18147244, supplier_type#18147243, supplier_id#18147238 AS canonical_id#18147185, 1 AS is_primary#18147186, threshold#18147247 AS configured_threshold#18147187, 0.0 AS match_percentage#18147188, 0.0 AS confidence_score#18147189, new_canonical AS match_reason#18147190]
Input [5]: [brand_context_id#18147237, supplier_id#18147238, supplier_type#18147243, declaring_supplier_id#18147244, threshold#18147247]
(144) Union
(145) AdaptiveSparkPlan
Output [10]: [brand_context_id#18144926, supplier_id#18144927, declaring_supplier_id#18144933, supplier_type#18144932, canonical_id#18146883, is_primary#18146947, configured_threshold#18146948, match_percentage#18146949, confidence_score#18146950, match_reason#18146951]
Arguments: isFinalPlan=false
(146) TableCacheQueryStage
Output [1]: [match_reason#18146951]
Arguments: 0
(147) HashAggregate [codegen id : 1]
Input [1]: [match_reason#18146951]
Keys [1]: [match_reason#18146951]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#18151000L]
Results [2]: [match_reason#18146951, count#18151001L]
(148) Exchange
Input [2]: [match_reason#18146951, count#18151001L]
Arguments: hashpartitioning(match_reason#18146951, 10), ENSURE_REQUIREMENTS, [plan_id=4624390]
(149) ShuffleQueryStage
Output [2]: [match_reason#18146951, count#18151001L]
Arguments: 1
(150) AQEShuffleRead
Input [2]: [match_reason#18146951, count#18151001L]
Arguments: coalesced
(151) HashAggregate [codegen id : 2]
Input [2]: [match_reason#18146951, count#18151001L]
Keys [1]: [match_reason#18146951]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#18150846L]
Results [2]: [match_reason#18146951, count(1)#18150846L AS count#18150847L]
(152) HashAggregate
Input [1]: [match_reason#18146951]
Keys [1]: [match_reason#18146951]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#18151000L]
Results [2]: [match_reason#18146951, count#18151001L]
(153) Exchange
Input [2]: [match_reason#18146951, count#18151001L]
Arguments: hashpartitioning(match_reason#18146951, 10), ENSURE_REQUIREMENTS, [plan_id=4624286]
(154) HashAggregate
Input [2]: [match_reason#18146951, count#18151001L]
Keys [1]: [match_reason#18146951]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#18150846L]
Results [2]: [match_reason#18146951, count(1)#18150846L AS count#18150847L]
(155) AdaptiveSparkPlan
Output [2]: [match_reason#18146951, count#18150847L]
Arguments: isFinalPlan=true