== Physical Plan ==
AdaptiveSparkPlan (156)
+- == Final Plan ==
* HashAggregate (152)
+- AQEShuffleRead (151)
+- ShuffleQueryStage (150), Statistics(sizeInBytes=40.0 B, rowCount=1)
+- Exchange (149)
+- * HashAggregate (148)
+- TableCacheQueryStage (147), Statistics(sizeInBytes=178.0 B, rowCount=1)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (146)
+- Union (145)
:- Project (23)
: +- BroadcastHashJoin Inner BuildLeft (22)
: :- BroadcastExchange (12)
: : +- Filter (11)
: : +- InMemoryTableScan (3)
: : +- InMemoryRelation (4)
: : +- AdaptiveSparkPlan (10)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (21)
: +- Filter (20)
: +- InMemoryTableScan (13)
: +- InMemoryRelation (14)
: +- AdaptiveSparkPlan (19)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
:- Project (72)
: +- Filter (71)
: +- Window (70)
: +- WindowGroupLimit (69)
: +- Sort (68)
: +- Exchange (67)
: +- WindowGroupLimit (66)
: +- Sort (65)
: +- Project (64)
: +- BatchEvalPython (63)
: +- Project (62)
: +- Filter (61)
: +- BatchEvalPython (60)
: +- Project (59)
: +- BroadcastHashJoin Inner BuildLeft (58)
: :- BroadcastExchange (51)
: : +- Project (50)
: : +- SortMergeJoin LeftAnti (49)
: : :- Sort (34)
: : : +- Exchange (33)
: : : +- Project (32)
: : : +- BatchEvalPython (31)
: : : +- Project (30)
: : : +- Filter (29)
: : : +- BatchEvalPython (28)
: : : +- Filter (27)
: : : +- InMemoryTableScan (24)
: : : +- InMemoryRelation (25)
: : : +- AdaptiveSparkPlan (26)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Sort (48)
: : +- Exchange (47)
: : +- Project (46)
: : +- BroadcastHashJoin Inner BuildLeft (45)
: : :- BroadcastExchange (39)
: : : +- Filter (38)
: : : +- InMemoryTableScan (35)
: : : +- InMemoryRelation (36)
: : : +- AdaptiveSparkPlan (37)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Project (44)
: : +- Filter (43)
: : +- InMemoryTableScan (40)
: : +- InMemoryRelation (41)
: : +- AdaptiveSparkPlan (42)
: : +- Filter (18)
: : +- InMemoryTableScan (15)
: : +- InMemoryRelation (16)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
: +- Project (57)
: +- BatchEvalPython (56)
: +- Filter (55)
: +- InMemoryTableScan (52)
: +- InMemoryRelation (53)
: +- AdaptiveSparkPlan (54)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (144)
+- BroadcastHashJoin LeftAnti BuildRight (143)
:- SortMergeJoin LeftAnti (92)
: :- Sort (77)
: : +- Exchange (76)
: : +- InMemoryTableScan (73)
: : +- InMemoryRelation (74)
: : +- AdaptiveSparkPlan (75)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (91)
: +- Exchange (90)
: +- Project (89)
: +- BroadcastHashJoin Inner BuildLeft (88)
: :- BroadcastExchange (82)
: : +- Filter (81)
: : +- InMemoryTableScan (78)
: : +- InMemoryRelation (79)
: : +- AdaptiveSparkPlan (80)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (87)
: +- Filter (86)
: +- InMemoryTableScan (83)
: +- InMemoryRelation (84)
: +- AdaptiveSparkPlan (85)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- BroadcastExchange (142)
+- Project (141)
+- Filter (140)
+- Window (139)
+- WindowGroupLimit (138)
+- Sort (137)
+- Exchange (136)
+- WindowGroupLimit (135)
+- Sort (134)
+- Project (133)
+- BatchEvalPython (132)
+- Project (131)
+- Filter (130)
+- BatchEvalPython (129)
+- Project (128)
+- BroadcastHashJoin Inner BuildLeft (127)
:- BroadcastExchange (120)
: +- Project (119)
: +- SortMergeJoin LeftAnti (118)
: :- Sort (103)
: : +- Exchange (102)
: : +- Project (101)
: : +- BatchEvalPython (100)
: : +- Project (99)
: : +- Filter (98)
: : +- BatchEvalPython (97)
: : +- Filter (96)
: : +- InMemoryTableScan (93)
: : +- InMemoryRelation (94)
: : +- AdaptiveSparkPlan (95)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (117)
: +- Exchange (116)
: +- Project (115)
: +- BroadcastHashJoin Inner BuildLeft (114)
: :- BroadcastExchange (108)
: : +- Filter (107)
: : +- InMemoryTableScan (104)
: : +- InMemoryRelation (105)
: : +- AdaptiveSparkPlan (106)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (113)
: +- Filter (112)
: +- InMemoryTableScan (109)
: +- InMemoryRelation (110)
: +- AdaptiveSparkPlan (111)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (126)
+- BatchEvalPython (125)
+- Filter (124)
+- InMemoryTableScan (121)
+- InMemoryRelation (122)
+- AdaptiveSparkPlan (123)
+- Filter (18)
+- InMemoryTableScan (15)
+- InMemoryRelation (16)
+- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- == Initial Plan ==
HashAggregate (155)
+- Exchange (154)
+- HashAggregate (153)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (146)
+- Union (145)
:- Project (23)
: +- BroadcastHashJoin Inner BuildLeft (22)
: :- BroadcastExchange (12)
: : +- Filter (11)
: : +- InMemoryTableScan (3)
: : +- InMemoryRelation (4)
: : +- AdaptiveSparkPlan (10)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (21)
: +- Filter (20)
: +- InMemoryTableScan (13)
: +- InMemoryRelation (14)
: +- AdaptiveSparkPlan (19)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
:- Project (72)
: +- Filter (71)
: +- Window (70)
: +- WindowGroupLimit (69)
: +- Sort (68)
: +- Exchange (67)
: +- WindowGroupLimit (66)
: +- Sort (65)
: +- Project (64)
: +- BatchEvalPython (63)
: +- Project (62)
: +- Filter (61)
: +- BatchEvalPython (60)
: +- Project (59)
: +- BroadcastHashJoin Inner BuildLeft (58)
: :- BroadcastExchange (51)
: : +- Project (50)
: : +- SortMergeJoin LeftAnti (49)
: : :- Sort (34)
: : : +- Exchange (33)
: : : +- Project (32)
: : : +- BatchEvalPython (31)
: : : +- Project (30)
: : : +- Filter (29)
: : : +- BatchEvalPython (28)
: : : +- Filter (27)
: : : +- InMemoryTableScan (24)
: : : +- InMemoryRelation (25)
: : : +- AdaptiveSparkPlan (26)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Sort (48)
: : +- Exchange (47)
: : +- Project (46)
: : +- BroadcastHashJoin Inner BuildLeft (45)
: : :- BroadcastExchange (39)
: : : +- Filter (38)
: : : +- InMemoryTableScan (35)
: : : +- InMemoryRelation (36)
: : : +- AdaptiveSparkPlan (37)
: : : +- Project (9)
: : : +- Filter (8)
: : : +- InMemoryTableScan (5)
: : : +- InMemoryRelation (6)
: : : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: : +- Project (44)
: : +- Filter (43)
: : +- InMemoryTableScan (40)
: : +- InMemoryRelation (41)
: : +- AdaptiveSparkPlan (42)
: : +- Filter (18)
: : +- InMemoryTableScan (15)
: : +- InMemoryRelation (16)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
: +- Project (57)
: +- BatchEvalPython (56)
: +- Filter (55)
: +- InMemoryTableScan (52)
: +- InMemoryRelation (53)
: +- AdaptiveSparkPlan (54)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (144)
+- BroadcastHashJoin LeftAnti BuildRight (143)
:- SortMergeJoin LeftAnti (92)
: :- Sort (77)
: : +- Exchange (76)
: : +- InMemoryTableScan (73)
: : +- InMemoryRelation (74)
: : +- AdaptiveSparkPlan (75)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (91)
: +- Exchange (90)
: +- Project (89)
: +- BroadcastHashJoin Inner BuildLeft (88)
: :- BroadcastExchange (82)
: : +- Filter (81)
: : +- InMemoryTableScan (78)
: : +- InMemoryRelation (79)
: : +- AdaptiveSparkPlan (80)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (87)
: +- Filter (86)
: +- InMemoryTableScan (83)
: +- InMemoryRelation (84)
: +- AdaptiveSparkPlan (85)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- BroadcastExchange (142)
+- Project (141)
+- Filter (140)
+- Window (139)
+- WindowGroupLimit (138)
+- Sort (137)
+- Exchange (136)
+- WindowGroupLimit (135)
+- Sort (134)
+- Project (133)
+- BatchEvalPython (132)
+- Project (131)
+- Filter (130)
+- BatchEvalPython (129)
+- Project (128)
+- BroadcastHashJoin Inner BuildLeft (127)
:- BroadcastExchange (120)
: +- Project (119)
: +- SortMergeJoin LeftAnti (118)
: :- Sort (103)
: : +- Exchange (102)
: : +- Project (101)
: : +- BatchEvalPython (100)
: : +- Project (99)
: : +- Filter (98)
: : +- BatchEvalPython (97)
: : +- Filter (96)
: : +- InMemoryTableScan (93)
: : +- InMemoryRelation (94)
: : +- AdaptiveSparkPlan (95)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Sort (117)
: +- Exchange (116)
: +- Project (115)
: +- BroadcastHashJoin Inner BuildLeft (114)
: :- BroadcastExchange (108)
: : +- Filter (107)
: : +- InMemoryTableScan (104)
: : +- InMemoryRelation (105)
: : +- AdaptiveSparkPlan (106)
: : +- Project (9)
: : +- Filter (8)
: : +- InMemoryTableScan (5)
: : +- InMemoryRelation (6)
: : +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (7)
: +- Project (113)
: +- Filter (112)
: +- InMemoryTableScan (109)
: +- InMemoryRelation (110)
: +- AdaptiveSparkPlan (111)
: +- Filter (18)
: +- InMemoryTableScan (15)
: +- InMemoryRelation (16)
: +- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
+- Project (126)
+- BatchEvalPython (125)
+- Filter (124)
+- InMemoryTableScan (121)
+- InMemoryRelation (122)
+- AdaptiveSparkPlan (123)
+- Filter (18)
+- InMemoryTableScan (15)
+- InMemoryRelation (16)
+- * Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] (17)
(1) InMemoryTableScan
Output [1]: [match_reason#17310608]
Arguments: [match_reason#17310608]
(2) InMemoryRelation
Arguments: [brand_context_id#17308583, supplier_id#17308584, declaring_supplier_id#17308590, supplier_type#17308589, canonical_id#17310540, is_primary#17310604, configured_threshold#17310605, match_percentage#17310606, confidence_score#17310607, match_reason#17310608], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
Union
:- *(20) Project [brand_context_id#17308583, supplier_id#17308584, declaring_supplier_id#17308590, supplier_type#17308589, canonical_id#17310540, 0 AS is_primary#17310604, threshold#17308593 AS configured_threshold#17310605, 100.0 AS match_percentage#17310606, 1.0 AS confidence_score#17310607, exact_id_match AS match_reason#17310608]
: +- *(20) BroadcastHashJoin [brand_context_id#17308583, supplier_id#17308584], [brand_context_id#17310532, supplier_id#17310533], Inner, BuildLeft, false
: :- BroadcastQueryStage 13
: : +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4566374]
: : +- *(1) Filter (isnotnull(brand_context_id#17308583) AND isnotnull(supplier_id#17308584))
: : +- TableCacheQueryStage 0
: : +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593], [isnotnull(brand_context_id#17308583), isnotnull(supplier_id#17308584)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- *(20) Project [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540]
: +- *(20) Filter ((((isnotnull(is_primary#17310541) AND is_primary#17310541) AND isnotnull(canonical_id#17310540)) AND isnotnull(brand_context_id#17310532)) AND isnotnull(supplier_id#17310533))
: +- TableCacheQueryStage 1
: +- InMemoryTableScan [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533], [isnotnull(is_primary#17310541), is_primary#17310541, isnotnull(canonical_id#17310540), isnotnull(brand_context_id#17310532), isnotnull(supplier_id#17310533)]
: +- InMemoryRelation [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540, is_primary#17310541, entity_name#17310534, city#17310535, state#17310536, country#17310537, threshold#17310542, create_date#17310543], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#17308591)
: +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
:- *(22) Project [u_brand_context_id#17310648 AS brand_context_id#17310769, u_supplier_id#17310649 AS supplier_id#17310770, u_declaring_supplier_id#17310650 AS declaring_supplier_id#17310771, u_supplier_type#17310651 AS supplier_type#17310772, e_canonical_id#17310672 AS canonical_id#17310773, 0 AS is_primary#17310774, u_threshold#17310652 AS configured_threshold#17310775, cast((similarity#17310729 * 100.0) as double) AS match_percentage#17310882, cast(similarity#17310729 as double) AS confidence_score#17310883, matched_existing AS match_reason#17310778]
: +- *(22) Filter (rank#17310749 = 1)
: +- Window [row_number() windowspecdefinition(u_brand_context_id#17310648, u_supplier_id#17310649, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#17310749], [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST]
: +- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Final
: +- *(21) Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
: +- ShuffleQueryStage 23
: +- Exchange hashpartitioning(u_brand_context_id#17310648, u_supplier_id#17310649, 10), ENSURE_REQUIREMENTS, [plan_id=4567517]
: +- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Partial
: +- *(15) Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
: +- *(15) Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, pythonUDF0#17312264 AS similarity#17310729]
: +- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312264]
: +- *(14) Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
: +- *(14) Filter (cast(pythonUDF0#17312263 as double) >= u_threshold#17310652)
: +- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312263]
: +- LocalTableScan <empty>, [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
+- *(23) Project [brand_context_id#17310894, supplier_id#17310895, declaring_supplier_id#17310901, supplier_type#17310900, supplier_id#17310895 AS canonical_id#17310842, 1 AS is_primary#17310843, threshold#17310904 AS configured_threshold#17310844, 0.0 AS match_percentage#17310845, 0.0 AS confidence_score#17310846, new_canonical AS match_reason#17310847]
+- ShuffleQueryStage 16
+- Exchange hashpartitioning(brand_context_id#17310894, supplier_id#17310895, 10), ENSURE_REQUIREMENTS, [plan_id=4565955]
+- TableCacheQueryStage 6
+- InMemoryTableScan [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
+- InMemoryRelation [brand_context_id#17310894, supplier_id#17310895, entity_name#17310896, city#17310897, state#17310898, country#17310899, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904, create_date#17310905], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Union
:- Project [brand_context_id#17308583, supplier_id#17308584, declaring_supplier_id#17308590, supplier_type#17308589, canonical_id#17310540, 0 AS is_primary#17310604, threshold#17308593 AS configured_threshold#17310605, 100.0 AS match_percentage#17310606, 1.0 AS confidence_score#17310607, exact_id_match AS match_reason#17310608]
: +- BroadcastHashJoin [brand_context_id#17308583, supplier_id#17308584], [brand_context_id#17310532, supplier_id#17310533], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4565641]
: : +- Filter (isnotnull(brand_context_id#17308583) AND isnotnull(supplier_id#17308584))
: : +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593], [isnotnull(brand_context_id#17308583), isnotnull(supplier_id#17308584)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540]
: +- Filter ((((isnotnull(is_primary#17310541) AND is_primary#17310541) AND isnotnull(canonical_id#17310540)) AND isnotnull(brand_context_id#17310532)) AND isnotnull(supplier_id#17310533))
: +- InMemoryTableScan [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533], [isnotnull(is_primary#17310541), is_primary#17310541, isnotnull(canonical_id#17310540), isnotnull(brand_context_id#17310532), isnotnull(supplier_id#17310533)]
: +- InMemoryRelation [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540, is_primary#17310541, entity_name#17310534, city#17310535, state#17310536, country#17310537, threshold#17310542, create_date#17310543], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#17308591)
: +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
:- Project [u_brand_context_id#17310648 AS brand_context_id#17310769, u_supplier_id#17310649 AS supplier_id#17310770, u_declaring_supplier_id#17310650 AS declaring_supplier_id#17310771, u_supplier_type#17310651 AS supplier_type#17310772, e_canonical_id#17310672 AS canonical_id#17310773, 0 AS is_primary#17310774, u_threshold#17310652 AS configured_threshold#17310775, cast((similarity#17310729 * 100.0) as double) AS match_percentage#17310882, cast(similarity#17310729 as double) AS confidence_score#17310883, matched_existing AS match_reason#17310778]
: +- Filter (rank#17310749 = 1)
: +- Window [row_number() windowspecdefinition(u_brand_context_id#17310648, u_supplier_id#17310649, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#17310749], [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST]
: +- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Final
: +- Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(u_brand_context_id#17310648, u_supplier_id#17310649, 10), ENSURE_REQUIREMENTS, [plan_id=4565668]
: +- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Partial
: +- Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
: +- Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, pythonUDF0#17312264 AS similarity#17310729]
: +- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312264]
: +- Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
: +- Filter (cast(pythonUDF0#17312263 as double) >= u_threshold#17310652)
: +- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312263]
: +- Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
: +- BroadcastHashJoin [u_brand_context_id#17310648, u_location_key#17310657], [e_brand_context_id#17310670, e_location_key#17310674], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[6, string, false]),false), [plan_id=4565656]
: : +- Project [brand_context_id#17310858 AS u_brand_context_id#17310648, supplier_id#17310859 AS u_supplier_id#17310649, declaring_supplier_id#17310865 AS u_declaring_supplier_id#17310650, supplier_type#17310864 AS u_supplier_type#17310651, threshold#17310868 AS u_threshold#17310652, normalized_name#17310115 AS u_normalized_name#17310653, location_key#17310172 AS u_location_key#17310657]
: : +- SortMergeJoin [brand_context_id#17310858, supplier_id#17310859], [brand_context_id#17310621, supplier_id#17310622], LeftAnti
: : :- Sort [brand_context_id#17310858 ASC NULLS FIRST, supplier_id#17310859 ASC NULLS FIRST], false, 0
: : : +- Exchange hashpartitioning(brand_context_id#17310858, supplier_id#17310859, 10), ENSURE_REQUIREMENTS, [plan_id=4565649]
: : : +- Project [brand_context_id#17310858, supplier_id#17310859, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, pythonUDF0#17312255 AS normalized_name#17310115, concat_ws(|, coalesce(pythonUDF1#17312256, ), coalesce(pythonUDF2#17312257, ), coalesce(pythonUDF3#17312258, )) AS location_key#17310172]
: : : +- BatchEvalPython [normalize_name(entity_name#17310860)#17310114, normalize_location(city#17310861)#17310127, normalize_location(state#17310862)#17310141, normalize_location(country#17310863)#17310156], [pythonUDF0#17312255, pythonUDF1#17312256, pythonUDF2#17312257, pythonUDF3#17312258]
: : : +- Project [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
: : : +- Filter ((isnotnull(pythonUDF0#17312252) AND isnotnull(pythonUDF1#17312253)) AND isnotnull(pythonUDF2#17312254))
: : : +- BatchEvalPython [normalize_location(city#17310861)#17310127, normalize_location(state#17310862)#17310141, normalize_location(country#17310863)#17310156], [pythonUDF0#17312252, pythonUDF1#17312253, pythonUDF2#17312254]
: : : +- Filter (isnotnull(threshold#17310868) AND isnotnull(brand_context_id#17310858))
: : : +- InMemoryTableScan [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868], [isnotnull(threshold#17310868), isnotnull(brand_context_id#17310858)]
: : : +- InMemoryRelation [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, create_date#17310869], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- AdaptiveSparkPlan isFinalPlan=false
: : : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : : +- Filter isnull(canonical_id#17308591)
: : : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: : +- Sort [brand_context_id#17310621 ASC NULLS FIRST, supplier_id#17310622 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#17310621, supplier_id#17310622, 10), ENSURE_REQUIREMENTS, [plan_id=4565650]
: : +- Project [brand_context_id#17310621, supplier_id#17310622]
: : +- BroadcastHashJoin [brand_context_id#17310621, supplier_id#17310622], [brand_context_id#17310870, supplier_id#17310871], Inner, BuildLeft, false
: : :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4565644]
: : : +- Filter (isnotnull(brand_context_id#17310621) AND isnotnull(supplier_id#17310622))
: : : +- InMemoryTableScan [brand_context_id#17310621, supplier_id#17310622], [isnotnull(brand_context_id#17310621), isnotnull(supplier_id#17310622)]
: : : +- InMemoryRelation [brand_context_id#17310621, supplier_id#17310622, entity_name#17310623, city#17310624, state#17310625, country#17310626, supplier_type#17310627, declaring_supplier_id#17310628, threshold#17310631, create_date#17310632], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- AdaptiveSparkPlan isFinalPlan=false
: : : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : : +- Filter isnull(canonical_id#17308591)
: : : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: : +- Project [brand_context_id#17310870, supplier_id#17310871]
: : +- Filter ((((isnotnull(is_primary#17310879) AND is_primary#17310879) AND isnotnull(canonical_id#17310878)) AND isnotnull(brand_context_id#17310870)) AND isnotnull(supplier_id#17310871))
: : +- InMemoryTableScan [brand_context_id#17310870, canonical_id#17310878, is_primary#17310879, supplier_id#17310871], [isnotnull(is_primary#17310879), is_primary#17310879, isnotnull(canonical_id#17310878), isnotnull(brand_context_id#17310870), isnotnull(supplier_id#17310871)]
: : +- InMemoryRelation [brand_context_id#17310870, supplier_id#17310871, canonical_id#17310878, is_primary#17310879, entity_name#17310872, city#17310873, state#17310874, country#17310875, threshold#17310880, create_date#17310881], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Filter isnotnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#17310682 AS e_brand_context_id#17310670, supplier_id#17310683 AS e_supplier_id#17310671, canonical_id#17310690 AS e_canonical_id#17310672, pythonUDF0#17312259 AS e_normalized_name#17310673, concat_ws(|, coalesce(pythonUDF1#17312260, ), coalesce(pythonUDF2#17312261, ), coalesce(pythonUDF3#17312262, )) AS e_location_key#17310674, create_date#17310693 AS e_create_date#17310675]
: +- BatchEvalPython [normalize_name(entity_name#17310684)#17310188, normalize_location(city#17310685)#17310201, normalize_location(state#17310686)#17310215, normalize_location(country#17310687)#17310230], [pythonUDF0#17312259, pythonUDF1#17312260, pythonUDF2#17312261, pythonUDF3#17312262]
: +- Filter isnotnull(brand_context_id#17310682)
: +- InMemoryTableScan [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693], [isnotnull(brand_context_id#17310682)]
: +- InMemoryRelation [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, is_primary#17310691, entity_name#17310684, city#17310685, state#17310686, country#17310687, threshold#17310692, create_date#17310693], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#17308591)
: +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- Project [brand_context_id#17310894, supplier_id#17310895, declaring_supplier_id#17310901, supplier_type#17310900, supplier_id#17310895 AS canonical_id#17310842, 1 AS is_primary#17310843, threshold#17310904 AS configured_threshold#17310844, 0.0 AS match_percentage#17310845, 0.0 AS confidence_score#17310846, new_canonical AS match_reason#17310847]
+- BroadcastHashJoin [brand_context_id#17310894, supplier_id#17310895], [brand_context_id#17310769, supplier_id#17310770], LeftAnti, BuildRight, false
:- SortMergeJoin [brand_context_id#17310894, supplier_id#17310895], [brand_context_id#17310906, supplier_id#17310907], LeftAnti
: :- Sort [brand_context_id#17310894 ASC NULLS FIRST, supplier_id#17310895 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#17310894, supplier_id#17310895, 10), ENSURE_REQUIREMENTS, [plan_id=4565679]
: : +- InMemoryTableScan [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
: : +- InMemoryRelation [brand_context_id#17310894, supplier_id#17310895, entity_name#17310896, city#17310897, state#17310898, country#17310899, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904, create_date#17310905], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Sort [brand_context_id#17310906 ASC NULLS FIRST, supplier_id#17310907 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(brand_context_id#17310906, supplier_id#17310907, 10), ENSURE_REQUIREMENTS, [plan_id=4565680]
: +- Project [brand_context_id#17310906, supplier_id#17310907]
: +- BroadcastHashJoin [brand_context_id#17310906, supplier_id#17310907], [brand_context_id#17310918, supplier_id#17310919], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4565674]
: : +- Filter (isnotnull(brand_context_id#17310906) AND isnotnull(supplier_id#17310907))
: : +- InMemoryTableScan [brand_context_id#17310906, supplier_id#17310907], [isnotnull(brand_context_id#17310906), isnotnull(supplier_id#17310907)]
: : +- InMemoryRelation [brand_context_id#17310906, supplier_id#17310907, entity_name#17310908, city#17310909, state#17310910, country#17310911, supplier_type#17310912, declaring_supplier_id#17310913, threshold#17310916, create_date#17310917], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#17310918, supplier_id#17310919]
: +- Filter ((((isnotnull(is_primary#17310927) AND is_primary#17310927) AND isnotnull(canonical_id#17310926)) AND isnotnull(brand_context_id#17310918)) AND isnotnull(supplier_id#17310919))
: +- InMemoryTableScan [brand_context_id#17310918, canonical_id#17310926, is_primary#17310927, supplier_id#17310919], [isnotnull(is_primary#17310927), is_primary#17310927, isnotnull(canonical_id#17310926), isnotnull(brand_context_id#17310918), isnotnull(supplier_id#17310919)]
: +- InMemoryRelation [brand_context_id#17310918, supplier_id#17310919, canonical_id#17310926, is_primary#17310927, entity_name#17310920, city#17310921, state#17310922, country#17310923, threshold#17310928, create_date#17310929], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#17308591)
: +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[1, string, true]),false), [plan_id=4565715]
+- Project [u_brand_context_id#17310648 AS brand_context_id#17310769, u_supplier_id#17310649 AS supplier_id#17310770]
+- Filter (rank#17310749 = 1)
+- Window [row_number() windowspecdefinition(u_brand_context_id#17310648, u_supplier_id#17310649, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#17310749], [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST]
+- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Final
+- Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(u_brand_context_id#17310648, u_supplier_id#17310649, 10), ENSURE_REQUIREMENTS, [plan_id=4565708]
+- WindowGroupLimit [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Partial
+- Sort [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
+- Project [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, pythonUDF0#17312277 AS similarity#17310729]
+- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312277]
+- Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
+- Filter (cast(pythonUDF0#17312276 as double) >= u_threshold#17310652)
+- BatchEvalPython [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312276]
+- Project [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
+- BroadcastHashJoin [u_brand_context_id#17310648, u_location_key#17310657], [e_brand_context_id#17310670, e_location_key#17310674], Inner, BuildLeft, false
:- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true], input[4, string, false]),false), [plan_id=4565696]
: +- Project [brand_context_id#17310791 AS u_brand_context_id#17310648, supplier_id#17310792 AS u_supplier_id#17310649, threshold#17310801 AS u_threshold#17310652, normalized_name#17310115 AS u_normalized_name#17310653, location_key#17310172 AS u_location_key#17310657]
: +- SortMergeJoin [brand_context_id#17310791, supplier_id#17310792], [brand_context_id#17310803, supplier_id#17310804], LeftAnti
: :- Sort [brand_context_id#17310791 ASC NULLS FIRST, supplier_id#17310792 ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(brand_context_id#17310791, supplier_id#17310792, 10), ENSURE_REQUIREMENTS, [plan_id=4565689]
: : +- Project [brand_context_id#17310791, supplier_id#17310792, threshold#17310801, pythonUDF0#17312268 AS normalized_name#17310115, concat_ws(|, coalesce(pythonUDF1#17312269, ), coalesce(pythonUDF2#17312270, ), coalesce(pythonUDF3#17312271, )) AS location_key#17310172]
: : +- BatchEvalPython [normalize_name(entity_name#17310793)#17310114, normalize_location(city#17310794)#17310127, normalize_location(state#17310795)#17310141, normalize_location(country#17310796)#17310156], [pythonUDF0#17312268, pythonUDF1#17312269, pythonUDF2#17312270, pythonUDF3#17312271]
: : +- Project [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
: : +- Filter ((isnotnull(pythonUDF0#17312265) AND isnotnull(pythonUDF1#17312266)) AND isnotnull(pythonUDF2#17312267))
: : +- BatchEvalPython [normalize_location(city#17310794)#17310127, normalize_location(state#17310795)#17310141, normalize_location(country#17310796)#17310156], [pythonUDF0#17312265, pythonUDF1#17312266, pythonUDF2#17312267]
: : +- Filter ((isnotnull(threshold#17310801) AND isnotnull(brand_context_id#17310791)) AND isnotnull(supplier_id#17310792))
: : +- InMemoryTableScan [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801], [isnotnull(threshold#17310801), isnotnull(brand_context_id#17310791), isnotnull(supplier_id#17310792)]
: : +- InMemoryRelation [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, supplier_type#17310797, declaring_supplier_id#17310798, threshold#17310801, create_date#17310802], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Sort [brand_context_id#17310803 ASC NULLS FIRST, supplier_id#17310804 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(brand_context_id#17310803, supplier_id#17310804, 10), ENSURE_REQUIREMENTS, [plan_id=4565690]
: +- Project [brand_context_id#17310803, supplier_id#17310804]
: +- BroadcastHashJoin [brand_context_id#17310803, supplier_id#17310804], [brand_context_id#17310815, supplier_id#17310816], Inner, BuildLeft, false
: :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4565684]
: : +- Filter (isnotnull(brand_context_id#17310803) AND isnotnull(supplier_id#17310804))
: : +- InMemoryTableScan [brand_context_id#17310803, supplier_id#17310804], [isnotnull(brand_context_id#17310803), isnotnull(supplier_id#17310804)]
: : +- InMemoryRelation [brand_context_id#17310803, supplier_id#17310804, entity_name#17310805, city#17310806, state#17310807, country#17310808, supplier_type#17310809, declaring_supplier_id#17310810, threshold#17310813, create_date#17310814], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- AdaptiveSparkPlan isFinalPlan=false
: : +- Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
: : +- Filter isnull(canonical_id#17308591)
: : +- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
: : +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
: +- Project [brand_context_id#17310815, supplier_id#17310816]
: +- Filter ((((isnotnull(is_primary#17310824) AND is_primary#17310824) AND isnotnull(canonical_id#17310823)) AND isnotnull(brand_context_id#17310815)) AND isnotnull(supplier_id#17310816))
: +- InMemoryTableScan [brand_context_id#17310815, canonical_id#17310823, is_primary#17310824, supplier_id#17310816], [isnotnull(is_primary#17310824), is_primary#17310824, isnotnull(canonical_id#17310823), isnotnull(brand_context_id#17310815), isnotnull(supplier_id#17310816)]
: +- InMemoryRelation [brand_context_id#17310815, supplier_id#17310816, canonical_id#17310823, is_primary#17310824, entity_name#17310817, city#17310818, state#17310819, country#17310820, threshold#17310825, create_date#17310826], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Filter isnotnull(canonical_id#17308591)
: +- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
: +- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- Project [brand_context_id#17310930 AS e_brand_context_id#17310670, supplier_id#17310931 AS e_supplier_id#17310671, pythonUDF0#17312272 AS e_normalized_name#17310673, concat_ws(|, coalesce(pythonUDF1#17312273, ), coalesce(pythonUDF2#17312274, ), coalesce(pythonUDF3#17312275, )) AS e_location_key#17310674, create_date#17310941 AS e_create_date#17310675]
+- BatchEvalPython [normalize_name(entity_name#17310932)#17310188, normalize_location(city#17310933)#17310201, normalize_location(state#17310934)#17310215, normalize_location(country#17310935)#17310230], [pythonUDF0#17312272, pythonUDF1#17312273, pythonUDF2#17312274, pythonUDF3#17312275]
+- Filter isnotnull(brand_context_id#17310930)
+- InMemoryTableScan [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941], [isnotnull(brand_context_id#17310930)]
+- InMemoryRelation [brand_context_id#17310930, supplier_id#17310931, canonical_id#17310938, is_primary#17310939, entity_name#17310932, city#17310933, state#17310934, country#17310935, threshold#17310940, create_date#17310941], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(3) InMemoryTableScan
Output [5]: [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593]
Arguments: [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593], [isnotnull(brand_context_id#17308583), isnotnull(supplier_id#17308584)]
(4) InMemoryRelation
Arguments: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(5) InMemoryTableScan
Output [11]: [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593]
Arguments: [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
(6) InMemoryRelation
Arguments: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(7) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [codegen id : 1]
Output [12]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594]
ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,country:string,supplier_type:string,declaring_supplier_id:string,canonical_id:string,is_primary:boolean,threshold:double,create_date:date>
(8) Filter
Input [11]: [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593]
Condition : isnull(canonical_id#17308591)
(9) Project
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Input [11]: [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593]
(10) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(11) Filter
Input [5]: [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593]
Condition : (isnotnull(brand_context_id#17308583) AND isnotnull(supplier_id#17308584))
(12) BroadcastExchange
Input [5]: [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4568298]
(13) InMemoryTableScan
Output [4]: [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533]
Arguments: [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533], [isnotnull(is_primary#17310541), is_primary#17310541, isnotnull(canonical_id#17310540), isnotnull(brand_context_id#17310532), isnotnull(supplier_id#17310533)]
(14) InMemoryRelation
Arguments: [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540, is_primary#17310541, entity_name#17310534, city#17310535, state#17310536, country#17310537, threshold#17310542, create_date#17310543], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(15) InMemoryTableScan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
(16) InMemoryRelation
Arguments: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(17) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [codegen id : 1]
Output [12]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594]
ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,country:string,supplier_type:string,declaring_supplier_id:string,canonical_id:string,is_primary:boolean,threshold:double,create_date:date>
(18) Filter
Input [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Condition : isnotnull(canonical_id#17308591)
(19) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(20) Filter
Input [4]: [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533]
Condition : ((((isnotnull(is_primary#17310541) AND is_primary#17310541) AND isnotnull(canonical_id#17310540)) AND isnotnull(brand_context_id#17310532)) AND isnotnull(supplier_id#17310533))
(21) Project
Output [3]: [brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540]
Input [4]: [brand_context_id#17310532, canonical_id#17310540, is_primary#17310541, supplier_id#17310533]
(22) BroadcastHashJoin
Left keys [2]: [brand_context_id#17308583, supplier_id#17308584]
Right keys [2]: [brand_context_id#17310532, supplier_id#17310533]
Join type: Inner
Join condition: None
(23) Project
Output [10]: [brand_context_id#17308583, supplier_id#17308584, declaring_supplier_id#17308590, supplier_type#17308589, canonical_id#17310540, 0 AS is_primary#17310604, threshold#17308593 AS configured_threshold#17310605, 100.0 AS match_percentage#17310606, 1.0 AS confidence_score#17310607, exact_id_match AS match_reason#17310608]
Input [8]: [brand_context_id#17308583, supplier_id#17308584, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, brand_context_id#17310532, supplier_id#17310533, canonical_id#17310540]
(24) InMemoryTableScan
Output [9]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
Arguments: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868], [isnotnull(threshold#17310868), isnotnull(brand_context_id#17310858)]
(25) InMemoryRelation
Arguments: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, create_date#17310869], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(26) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(27) Filter
Input [9]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
Condition : (isnotnull(threshold#17310868) AND isnotnull(brand_context_id#17310858))
(28) BatchEvalPython
Input [9]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
Arguments: [normalize_location(city#17310861)#17310127, normalize_location(state#17310862)#17310141, normalize_location(country#17310863)#17310156], [pythonUDF0#17312252, pythonUDF1#17312253, pythonUDF2#17312254]
(29) Filter
Input [12]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, pythonUDF0#17312252, pythonUDF1#17312253, pythonUDF2#17312254]
Condition : ((isnotnull(pythonUDF0#17312252) AND isnotnull(pythonUDF1#17312253)) AND isnotnull(pythonUDF2#17312254))
(30) Project
Output [9]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
Input [12]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, pythonUDF0#17312252, pythonUDF1#17312253, pythonUDF2#17312254]
(31) BatchEvalPython
Input [9]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868]
Arguments: [normalize_name(entity_name#17310860)#17310114, normalize_location(city#17310861)#17310127, normalize_location(state#17310862)#17310141, normalize_location(country#17310863)#17310156], [pythonUDF0#17312255, pythonUDF1#17312256, pythonUDF2#17312257, pythonUDF3#17312258]
(32) Project
Output [7]: [brand_context_id#17310858, supplier_id#17310859, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, pythonUDF0#17312255 AS normalized_name#17310115, concat_ws(|, coalesce(pythonUDF1#17312256, ), coalesce(pythonUDF2#17312257, ), coalesce(pythonUDF3#17312258, )) AS location_key#17310172]
Input [13]: [brand_context_id#17310858, supplier_id#17310859, entity_name#17310860, city#17310861, state#17310862, country#17310863, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, pythonUDF0#17312255, pythonUDF1#17312256, pythonUDF2#17312257, pythonUDF3#17312258]
(33) Exchange
Input [7]: [brand_context_id#17310858, supplier_id#17310859, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, normalized_name#17310115, location_key#17310172]
Arguments: hashpartitioning(brand_context_id#17310858, supplier_id#17310859, 10), ENSURE_REQUIREMENTS, [plan_id=4568306]
(34) Sort
Input [7]: [brand_context_id#17310858, supplier_id#17310859, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, normalized_name#17310115, location_key#17310172]
Arguments: [brand_context_id#17310858 ASC NULLS FIRST, supplier_id#17310859 ASC NULLS FIRST], false, 0
(35) InMemoryTableScan
Output [2]: [brand_context_id#17310621, supplier_id#17310622]
Arguments: [brand_context_id#17310621, supplier_id#17310622], [isnotnull(brand_context_id#17310621), isnotnull(supplier_id#17310622)]
(36) InMemoryRelation
Arguments: [brand_context_id#17310621, supplier_id#17310622, entity_name#17310623, city#17310624, state#17310625, country#17310626, supplier_type#17310627, declaring_supplier_id#17310628, threshold#17310631, create_date#17310632], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(37) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(38) Filter
Input [2]: [brand_context_id#17310621, supplier_id#17310622]
Condition : (isnotnull(brand_context_id#17310621) AND isnotnull(supplier_id#17310622))
(39) BroadcastExchange
Input [2]: [brand_context_id#17310621, supplier_id#17310622]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4568301]
(40) InMemoryTableScan
Output [4]: [brand_context_id#17310870, canonical_id#17310878, is_primary#17310879, supplier_id#17310871]
Arguments: [brand_context_id#17310870, canonical_id#17310878, is_primary#17310879, supplier_id#17310871], [isnotnull(is_primary#17310879), is_primary#17310879, isnotnull(canonical_id#17310878), isnotnull(brand_context_id#17310870), isnotnull(supplier_id#17310871)]
(41) InMemoryRelation
Arguments: [brand_context_id#17310870, supplier_id#17310871, canonical_id#17310878, is_primary#17310879, entity_name#17310872, city#17310873, state#17310874, country#17310875, threshold#17310880, create_date#17310881], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(42) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(43) Filter
Input [4]: [brand_context_id#17310870, canonical_id#17310878, is_primary#17310879, supplier_id#17310871]
Condition : ((((isnotnull(is_primary#17310879) AND is_primary#17310879) AND isnotnull(canonical_id#17310878)) AND isnotnull(brand_context_id#17310870)) AND isnotnull(supplier_id#17310871))
(44) Project
Output [2]: [brand_context_id#17310870, supplier_id#17310871]
Input [4]: [brand_context_id#17310870, canonical_id#17310878, is_primary#17310879, supplier_id#17310871]
(45) BroadcastHashJoin
Left keys [2]: [brand_context_id#17310621, supplier_id#17310622]
Right keys [2]: [brand_context_id#17310870, supplier_id#17310871]
Join type: Inner
Join condition: None
(46) Project
Output [2]: [brand_context_id#17310621, supplier_id#17310622]
Input [4]: [brand_context_id#17310621, supplier_id#17310622, brand_context_id#17310870, supplier_id#17310871]
(47) Exchange
Input [2]: [brand_context_id#17310621, supplier_id#17310622]
Arguments: hashpartitioning(brand_context_id#17310621, supplier_id#17310622, 10), ENSURE_REQUIREMENTS, [plan_id=4568307]
(48) Sort
Input [2]: [brand_context_id#17310621, supplier_id#17310622]
Arguments: [brand_context_id#17310621 ASC NULLS FIRST, supplier_id#17310622 ASC NULLS FIRST], false, 0
(49) SortMergeJoin
Left keys [2]: [brand_context_id#17310858, supplier_id#17310859]
Right keys [2]: [brand_context_id#17310621, supplier_id#17310622]
Join type: LeftAnti
Join condition: None
(50) Project
Output [7]: [brand_context_id#17310858 AS u_brand_context_id#17310648, supplier_id#17310859 AS u_supplier_id#17310649, declaring_supplier_id#17310865 AS u_declaring_supplier_id#17310650, supplier_type#17310864 AS u_supplier_type#17310651, threshold#17310868 AS u_threshold#17310652, normalized_name#17310115 AS u_normalized_name#17310653, location_key#17310172 AS u_location_key#17310657]
Input [7]: [brand_context_id#17310858, supplier_id#17310859, supplier_type#17310864, declaring_supplier_id#17310865, threshold#17310868, normalized_name#17310115, location_key#17310172]
(51) BroadcastExchange
Input [7]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, u_location_key#17310657]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true], input[6, string, false]),false), [plan_id=4568313]
(52) InMemoryTableScan
Output [8]: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693]
Arguments: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693], [isnotnull(brand_context_id#17310682)]
(53) InMemoryRelation
Arguments: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, is_primary#17310691, entity_name#17310684, city#17310685, state#17310686, country#17310687, threshold#17310692, create_date#17310693], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(54) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(55) Filter
Input [8]: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693]
Condition : isnotnull(brand_context_id#17310682)
(56) BatchEvalPython
Input [8]: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693]
Arguments: [normalize_name(entity_name#17310684)#17310188, normalize_location(city#17310685)#17310201, normalize_location(state#17310686)#17310215, normalize_location(country#17310687)#17310230], [pythonUDF0#17312259, pythonUDF1#17312260, pythonUDF2#17312261, pythonUDF3#17312262]
(57) Project
Output [6]: [brand_context_id#17310682 AS e_brand_context_id#17310670, supplier_id#17310683 AS e_supplier_id#17310671, canonical_id#17310690 AS e_canonical_id#17310672, pythonUDF0#17312259 AS e_normalized_name#17310673, concat_ws(|, coalesce(pythonUDF1#17312260, ), coalesce(pythonUDF2#17312261, ), coalesce(pythonUDF3#17312262, )) AS e_location_key#17310674, create_date#17310693 AS e_create_date#17310675]
Input [12]: [brand_context_id#17310682, supplier_id#17310683, canonical_id#17310690, entity_name#17310684, city#17310685, state#17310686, country#17310687, create_date#17310693, pythonUDF0#17312259, pythonUDF1#17312260, pythonUDF2#17312261, pythonUDF3#17312262]
(58) BroadcastHashJoin
Left keys [2]: [u_brand_context_id#17310648, u_location_key#17310657]
Right keys [2]: [e_brand_context_id#17310670, e_location_key#17310674]
Join type: Inner
Join condition: None
(59) Project
Output [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
Input [13]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, u_location_key#17310657, e_brand_context_id#17310670, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_location_key#17310674, e_create_date#17310675]
(60) BatchEvalPython
Input [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
Arguments: [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312263]
(61) Filter
Input [11]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312263]
Condition : (cast(pythonUDF0#17312263 as double) >= u_threshold#17310652)
(62) Project
Output [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
Input [11]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312263]
(63) BatchEvalPython
Input [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675]
Arguments: [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312264]
(64) Project
Output [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, pythonUDF0#17312264 AS similarity#17310729]
Input [11]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_canonical_id#17310672, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312264]
(65) Sort
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
(66) WindowGroupLimit
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Partial
(67) Exchange
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: hashpartitioning(u_brand_context_id#17310648, u_supplier_id#17310649, 10), ENSURE_REQUIREMENTS, [plan_id=4568325]
(68) Sort
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
(69) WindowGroupLimit
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Final
(70) Window
Input [9]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729]
Arguments: [row_number() windowspecdefinition(u_brand_context_id#17310648, u_supplier_id#17310649, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#17310749], [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST]
(71) Filter
Input [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729, rank#17310749]
Condition : (rank#17310749 = 1)
(72) Project
Output [10]: [u_brand_context_id#17310648 AS brand_context_id#17310769, u_supplier_id#17310649 AS supplier_id#17310770, u_declaring_supplier_id#17310650 AS declaring_supplier_id#17310771, u_supplier_type#17310651 AS supplier_type#17310772, e_canonical_id#17310672 AS canonical_id#17310773, 0 AS is_primary#17310774, u_threshold#17310652 AS configured_threshold#17310775, cast((similarity#17310729 * 100.0) as double) AS match_percentage#17310882, cast(similarity#17310729 as double) AS confidence_score#17310883, matched_existing AS match_reason#17310778]
Input [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_declaring_supplier_id#17310650, u_supplier_type#17310651, u_threshold#17310652, e_supplier_id#17310671, e_canonical_id#17310672, e_create_date#17310675, similarity#17310729, rank#17310749]
(73) InMemoryTableScan
Output [5]: [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
Arguments: [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
(74) InMemoryRelation
Arguments: [brand_context_id#17310894, supplier_id#17310895, entity_name#17310896, city#17310897, state#17310898, country#17310899, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904, create_date#17310905], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(75) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(76) Exchange
Input [5]: [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
Arguments: hashpartitioning(brand_context_id#17310894, supplier_id#17310895, 10), ENSURE_REQUIREMENTS, [plan_id=4568336]
(77) Sort
Input [5]: [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
Arguments: [brand_context_id#17310894 ASC NULLS FIRST, supplier_id#17310895 ASC NULLS FIRST], false, 0
(78) InMemoryTableScan
Output [2]: [brand_context_id#17310906, supplier_id#17310907]
Arguments: [brand_context_id#17310906, supplier_id#17310907], [isnotnull(brand_context_id#17310906), isnotnull(supplier_id#17310907)]
(79) InMemoryRelation
Arguments: [brand_context_id#17310906, supplier_id#17310907, entity_name#17310908, city#17310909, state#17310910, country#17310911, supplier_type#17310912, declaring_supplier_id#17310913, threshold#17310916, create_date#17310917], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(80) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(81) Filter
Input [2]: [brand_context_id#17310906, supplier_id#17310907]
Condition : (isnotnull(brand_context_id#17310906) AND isnotnull(supplier_id#17310907))
(82) BroadcastExchange
Input [2]: [brand_context_id#17310906, supplier_id#17310907]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4568331]
(83) InMemoryTableScan
Output [4]: [brand_context_id#17310918, canonical_id#17310926, is_primary#17310927, supplier_id#17310919]
Arguments: [brand_context_id#17310918, canonical_id#17310926, is_primary#17310927, supplier_id#17310919], [isnotnull(is_primary#17310927), is_primary#17310927, isnotnull(canonical_id#17310926), isnotnull(brand_context_id#17310918), isnotnull(supplier_id#17310919)]
(84) InMemoryRelation
Arguments: [brand_context_id#17310918, supplier_id#17310919, canonical_id#17310926, is_primary#17310927, entity_name#17310920, city#17310921, state#17310922, country#17310923, threshold#17310928, create_date#17310929], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(85) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(86) Filter
Input [4]: [brand_context_id#17310918, canonical_id#17310926, is_primary#17310927, supplier_id#17310919]
Condition : ((((isnotnull(is_primary#17310927) AND is_primary#17310927) AND isnotnull(canonical_id#17310926)) AND isnotnull(brand_context_id#17310918)) AND isnotnull(supplier_id#17310919))
(87) Project
Output [2]: [brand_context_id#17310918, supplier_id#17310919]
Input [4]: [brand_context_id#17310918, canonical_id#17310926, is_primary#17310927, supplier_id#17310919]
(88) BroadcastHashJoin
Left keys [2]: [brand_context_id#17310906, supplier_id#17310907]
Right keys [2]: [brand_context_id#17310918, supplier_id#17310919]
Join type: Inner
Join condition: None
(89) Project
Output [2]: [brand_context_id#17310906, supplier_id#17310907]
Input [4]: [brand_context_id#17310906, supplier_id#17310907, brand_context_id#17310918, supplier_id#17310919]
(90) Exchange
Input [2]: [brand_context_id#17310906, supplier_id#17310907]
Arguments: hashpartitioning(brand_context_id#17310906, supplier_id#17310907, 10), ENSURE_REQUIREMENTS, [plan_id=4568337]
(91) Sort
Input [2]: [brand_context_id#17310906, supplier_id#17310907]
Arguments: [brand_context_id#17310906 ASC NULLS FIRST, supplier_id#17310907 ASC NULLS FIRST], false, 0
(92) SortMergeJoin
Left keys [2]: [brand_context_id#17310894, supplier_id#17310895]
Right keys [2]: [brand_context_id#17310906, supplier_id#17310907]
Join type: LeftAnti
Join condition: None
(93) InMemoryTableScan
Output [7]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
Arguments: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801], [isnotnull(threshold#17310801), isnotnull(brand_context_id#17310791), isnotnull(supplier_id#17310792)]
(94) InMemoryRelation
Arguments: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, supplier_type#17310797, declaring_supplier_id#17310798, threshold#17310801, create_date#17310802], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(95) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(96) Filter
Input [7]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
Condition : ((isnotnull(threshold#17310801) AND isnotnull(brand_context_id#17310791)) AND isnotnull(supplier_id#17310792))
(97) BatchEvalPython
Input [7]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
Arguments: [normalize_location(city#17310794)#17310127, normalize_location(state#17310795)#17310141, normalize_location(country#17310796)#17310156], [pythonUDF0#17312265, pythonUDF1#17312266, pythonUDF2#17312267]
(98) Filter
Input [10]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801, pythonUDF0#17312265, pythonUDF1#17312266, pythonUDF2#17312267]
Condition : ((isnotnull(pythonUDF0#17312265) AND isnotnull(pythonUDF1#17312266)) AND isnotnull(pythonUDF2#17312267))
(99) Project
Output [7]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
Input [10]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801, pythonUDF0#17312265, pythonUDF1#17312266, pythonUDF2#17312267]
(100) BatchEvalPython
Input [7]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801]
Arguments: [normalize_name(entity_name#17310793)#17310114, normalize_location(city#17310794)#17310127, normalize_location(state#17310795)#17310141, normalize_location(country#17310796)#17310156], [pythonUDF0#17312268, pythonUDF1#17312269, pythonUDF2#17312270, pythonUDF3#17312271]
(101) Project
Output [5]: [brand_context_id#17310791, supplier_id#17310792, threshold#17310801, pythonUDF0#17312268 AS normalized_name#17310115, concat_ws(|, coalesce(pythonUDF1#17312269, ), coalesce(pythonUDF2#17312270, ), coalesce(pythonUDF3#17312271, )) AS location_key#17310172]
Input [11]: [brand_context_id#17310791, supplier_id#17310792, entity_name#17310793, city#17310794, state#17310795, country#17310796, threshold#17310801, pythonUDF0#17312268, pythonUDF1#17312269, pythonUDF2#17312270, pythonUDF3#17312271]
(102) Exchange
Input [5]: [brand_context_id#17310791, supplier_id#17310792, threshold#17310801, normalized_name#17310115, location_key#17310172]
Arguments: hashpartitioning(brand_context_id#17310791, supplier_id#17310792, 10), ENSURE_REQUIREMENTS, [plan_id=4568346]
(103) Sort
Input [5]: [brand_context_id#17310791, supplier_id#17310792, threshold#17310801, normalized_name#17310115, location_key#17310172]
Arguments: [brand_context_id#17310791 ASC NULLS FIRST, supplier_id#17310792 ASC NULLS FIRST], false, 0
(104) InMemoryTableScan
Output [2]: [brand_context_id#17310803, supplier_id#17310804]
Arguments: [brand_context_id#17310803, supplier_id#17310804], [isnotnull(brand_context_id#17310803), isnotnull(supplier_id#17310804)]
(105) InMemoryRelation
Arguments: [brand_context_id#17310803, supplier_id#17310804, entity_name#17310805, city#17310806, state#17310807, country#17310808, supplier_type#17310809, declaring_supplier_id#17310810, threshold#17310813, create_date#17310814], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- *(1) Filter isnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Project [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
+- Filter isnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, canonical_id#17308591, city#17308586, country#17308588, create_date#17308594, declaring_supplier_id#17308590, entity_name#17308585, state#17308587, supplier_id#17308584, supplier_type#17308589, threshold#17308593], [isnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(106) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(107) Filter
Input [2]: [brand_context_id#17310803, supplier_id#17310804]
Condition : (isnotnull(brand_context_id#17310803) AND isnotnull(supplier_id#17310804))
(108) BroadcastExchange
Input [2]: [brand_context_id#17310803, supplier_id#17310804]
Arguments: HashedRelationBroadcastMode(List(input[0, string, false], input[1, string, false]),false), [plan_id=4568341]
(109) InMemoryTableScan
Output [4]: [brand_context_id#17310815, canonical_id#17310823, is_primary#17310824, supplier_id#17310816]
Arguments: [brand_context_id#17310815, canonical_id#17310823, is_primary#17310824, supplier_id#17310816], [isnotnull(is_primary#17310824), is_primary#17310824, isnotnull(canonical_id#17310823), isnotnull(brand_context_id#17310815), isnotnull(supplier_id#17310816)]
(110) InMemoryRelation
Arguments: [brand_context_id#17310815, supplier_id#17310816, canonical_id#17310823, is_primary#17310824, entity_name#17310817, city#17310818, state#17310819, country#17310820, threshold#17310825, create_date#17310826], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(111) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(112) Filter
Input [4]: [brand_context_id#17310815, canonical_id#17310823, is_primary#17310824, supplier_id#17310816]
Condition : ((((isnotnull(is_primary#17310824) AND is_primary#17310824) AND isnotnull(canonical_id#17310823)) AND isnotnull(brand_context_id#17310815)) AND isnotnull(supplier_id#17310816))
(113) Project
Output [2]: [brand_context_id#17310815, supplier_id#17310816]
Input [4]: [brand_context_id#17310815, canonical_id#17310823, is_primary#17310824, supplier_id#17310816]
(114) BroadcastHashJoin
Left keys [2]: [brand_context_id#17310803, supplier_id#17310804]
Right keys [2]: [brand_context_id#17310815, supplier_id#17310816]
Join type: Inner
Join condition: None
(115) Project
Output [2]: [brand_context_id#17310803, supplier_id#17310804]
Input [4]: [brand_context_id#17310803, supplier_id#17310804, brand_context_id#17310815, supplier_id#17310816]
(116) Exchange
Input [2]: [brand_context_id#17310803, supplier_id#17310804]
Arguments: hashpartitioning(brand_context_id#17310803, supplier_id#17310804, 10), ENSURE_REQUIREMENTS, [plan_id=4568347]
(117) Sort
Input [2]: [brand_context_id#17310803, supplier_id#17310804]
Arguments: [brand_context_id#17310803 ASC NULLS FIRST, supplier_id#17310804 ASC NULLS FIRST], false, 0
(118) SortMergeJoin
Left keys [2]: [brand_context_id#17310791, supplier_id#17310792]
Right keys [2]: [brand_context_id#17310803, supplier_id#17310804]
Join type: LeftAnti
Join condition: None
(119) Project
Output [5]: [brand_context_id#17310791 AS u_brand_context_id#17310648, supplier_id#17310792 AS u_supplier_id#17310649, threshold#17310801 AS u_threshold#17310652, normalized_name#17310115 AS u_normalized_name#17310653, location_key#17310172 AS u_location_key#17310657]
Input [5]: [brand_context_id#17310791, supplier_id#17310792, threshold#17310801, normalized_name#17310115, location_key#17310172]
(120) BroadcastExchange
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, u_location_key#17310657]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true], input[4, string, false]),false), [plan_id=4568353]
(121) InMemoryTableScan
Output [7]: [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941]
Arguments: [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941], [isnotnull(brand_context_id#17310930)]
(122) InMemoryRelation
Arguments: [brand_context_id#17310930, supplier_id#17310931, canonical_id#17310938, is_primary#17310939, entity_name#17310932, city#17310933, state#17310934, country#17310935, threshold#17310940, create_date#17310941], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@730ae8bf,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Filter isnotnull(canonical_id#17308591)
+- TableCacheQueryStage 0
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
+- == Initial Plan ==
Filter isnotnull(canonical_id#17308591)
+- InMemoryTableScan [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594], [isnotnull(canonical_id#17308591)]
+- InMemoryRelation [brand_context_id#17308583, supplier_id#17308584, entity_name#17308585, city#17308586, state#17308587, country#17308588, supplier_type#17308589, declaring_supplier_id#17308590, canonical_id#17308591, is_primary#17308592, threshold#17308593, create_date#17308594], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Scan JDBCRelation((
SELECT DISTINCT
v.brand_context_id,
v.supplier_id,
v.entity_name,
v.city,
v.state,
v.country,
v.supplier_type,
v.declaring_supplier_id,
v.canonical_id,
v.is_primary,
COALESCE(v.threshold, 0.9) AS threshold,
v.create_date
FROM vw_all_suppliers_for_dedup v
) AS all_suppliers) [numPartitions=1] [brand_context_id#17308583,supplier_id#17308584,entity_name#17308585,city#17308586,state#17308587,country#17308588,supplier_type#17308589,declaring_supplier_id#17308590,canonical_id#17308591,is_primary#17308592,threshold#17308593,create_date#17308594] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,entity_name:string,city:string,state:string,cou...
,None)
(123) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, canonical_id#17308591, is_primary#17308592, entity_name#17308585, city#17308586, state#17308587, country#17308588, threshold#17308593, create_date#17308594]
Arguments: isFinalPlan=false
(124) Filter
Input [7]: [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941]
Condition : isnotnull(brand_context_id#17310930)
(125) BatchEvalPython
Input [7]: [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941]
Arguments: [normalize_name(entity_name#17310932)#17310188, normalize_location(city#17310933)#17310201, normalize_location(state#17310934)#17310215, normalize_location(country#17310935)#17310230], [pythonUDF0#17312272, pythonUDF1#17312273, pythonUDF2#17312274, pythonUDF3#17312275]
(126) Project
Output [5]: [brand_context_id#17310930 AS e_brand_context_id#17310670, supplier_id#17310931 AS e_supplier_id#17310671, pythonUDF0#17312272 AS e_normalized_name#17310673, concat_ws(|, coalesce(pythonUDF1#17312273, ), coalesce(pythonUDF2#17312274, ), coalesce(pythonUDF3#17312275, )) AS e_location_key#17310674, create_date#17310941 AS e_create_date#17310675]
Input [11]: [brand_context_id#17310930, supplier_id#17310931, entity_name#17310932, city#17310933, state#17310934, country#17310935, create_date#17310941, pythonUDF0#17312272, pythonUDF1#17312273, pythonUDF2#17312274, pythonUDF3#17312275]
(127) BroadcastHashJoin
Left keys [2]: [u_brand_context_id#17310648, u_location_key#17310657]
Right keys [2]: [e_brand_context_id#17310670, e_location_key#17310674]
Join type: Inner
Join condition: None
(128) Project
Output [7]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
Input [10]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, u_location_key#17310657, e_brand_context_id#17310670, e_supplier_id#17310671, e_normalized_name#17310673, e_location_key#17310674, e_create_date#17310675]
(129) BatchEvalPython
Input [7]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
Arguments: [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312276]
(130) Filter
Input [8]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312276]
Condition : (cast(pythonUDF0#17312276 as double) >= u_threshold#17310652)
(131) Project
Output [6]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
Input [8]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_threshold#17310652, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312276]
(132) BatchEvalPython
Input [6]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675]
Arguments: [calculate_similarity(u_normalized_name#17310653, e_normalized_name#17310673)#17310728], [pythonUDF0#17312277]
(133) Project
Output [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, pythonUDF0#17312277 AS similarity#17310729]
Input [7]: [u_brand_context_id#17310648, u_supplier_id#17310649, u_normalized_name#17310653, e_supplier_id#17310671, e_normalized_name#17310673, e_create_date#17310675, pythonUDF0#17312277]
(134) Sort
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
(135) WindowGroupLimit
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Partial
(136) Exchange
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: hashpartitioning(u_brand_context_id#17310648, u_supplier_id#17310649, 10), ENSURE_REQUIREMENTS, [plan_id=4568365]
(137) Sort
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648 ASC NULLS FIRST, u_supplier_id#17310649 ASC NULLS FIRST, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], false, 0
(138) WindowGroupLimit
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST], row_number(), 1, Final
(139) Window
Input [5]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729]
Arguments: [row_number() windowspecdefinition(u_brand_context_id#17310648, u_supplier_id#17310649, similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rank#17310749], [u_brand_context_id#17310648, u_supplier_id#17310649], [similarity#17310729 DESC NULLS LAST, e_create_date#17310675 ASC NULLS FIRST, e_supplier_id#17310671 ASC NULLS FIRST]
(140) Filter
Input [6]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729, rank#17310749]
Condition : (rank#17310749 = 1)
(141) Project
Output [2]: [u_brand_context_id#17310648 AS brand_context_id#17310769, u_supplier_id#17310649 AS supplier_id#17310770]
Input [6]: [u_brand_context_id#17310648, u_supplier_id#17310649, e_supplier_id#17310671, e_create_date#17310675, similarity#17310729, rank#17310749]
(142) BroadcastExchange
Input [2]: [brand_context_id#17310769, supplier_id#17310770]
Arguments: HashedRelationBroadcastMode(List(input[0, string, true], input[1, string, true]),false), [plan_id=4568372]
(143) BroadcastHashJoin
Left keys [2]: [brand_context_id#17310894, supplier_id#17310895]
Right keys [2]: [brand_context_id#17310769, supplier_id#17310770]
Join type: LeftAnti
Join condition: None
(144) Project
Output [10]: [brand_context_id#17310894, supplier_id#17310895, declaring_supplier_id#17310901, supplier_type#17310900, supplier_id#17310895 AS canonical_id#17310842, 1 AS is_primary#17310843, threshold#17310904 AS configured_threshold#17310844, 0.0 AS match_percentage#17310845, 0.0 AS confidence_score#17310846, new_canonical AS match_reason#17310847]
Input [5]: [brand_context_id#17310894, supplier_id#17310895, supplier_type#17310900, declaring_supplier_id#17310901, threshold#17310904]
(145) Union
(146) AdaptiveSparkPlan
Output [10]: [brand_context_id#17308583, supplier_id#17308584, declaring_supplier_id#17308590, supplier_type#17308589, canonical_id#17310540, is_primary#17310604, configured_threshold#17310605, match_percentage#17310606, confidence_score#17310607, match_reason#17310608]
Arguments: isFinalPlan=false
(147) TableCacheQueryStage
Output [1]: [match_reason#17310608]
Arguments: 0
(148) HashAggregate [codegen id : 1]
Input [1]: [match_reason#17310608]
Keys [1]: [match_reason#17310608]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#17314557L]
Results [2]: [match_reason#17310608, count#17314558L]
(149) Exchange
Input [2]: [match_reason#17310608, count#17314558L]
Arguments: hashpartitioning(match_reason#17310608, 10), ENSURE_REQUIREMENTS, [plan_id=4568400]
(150) ShuffleQueryStage
Output [2]: [match_reason#17310608, count#17314558L]
Arguments: 1
(151) AQEShuffleRead
Input [2]: [match_reason#17310608, count#17314558L]
Arguments: coalesced
(152) HashAggregate [codegen id : 2]
Input [2]: [match_reason#17310608, count#17314558L]
Keys [1]: [match_reason#17310608]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#17314403L]
Results [2]: [match_reason#17310608, count(1)#17314403L AS count#17314404L]
(153) HashAggregate
Input [1]: [match_reason#17310608]
Keys [1]: [match_reason#17310608]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#17314557L]
Results [2]: [match_reason#17310608, count#17314558L]
(154) Exchange
Input [2]: [match_reason#17310608, count#17314558L]
Arguments: hashpartitioning(match_reason#17310608, 10), ENSURE_REQUIREMENTS, [plan_id=4568295]
(155) HashAggregate
Input [2]: [match_reason#17310608, count#17314558L]
Keys [1]: [match_reason#17310608]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#17314403L]
Results [2]: [match_reason#17310608, count(1)#17314403L AS count#17314404L]
(156) AdaptiveSparkPlan
Output [2]: [match_reason#17310608, count#17314404L]
Arguments: isFinalPlan=true