Compare dataframes in scala and write the mismatching old and new columns to new dataframe

Kiran Kumar Source

I have two df's

df1
ID |BTH_DT |CDC_FLAG|CDC_TS |CNSM_ID
123|1986-10-07|I |2018-10-10 05:51:24.000000941|301634310
124|1973-02-15|I |2018-10-10 17:12:22.000000254|298910234

df2
ID |BTH_DT |CDC_FLAG|CDC_TS |CNSM_ID
123|1986-10-07|I |2018-10-10 05:51:24.000000941|\c
124|1973-02-15|I |2018-10-10 17:12:22.000000254|298910234

How do i compare two df's and write the mismatching columns alone to different df?

ID |CNSM_ID
123|301634310
123| \\c

df2.except(df1)  

above isn't serving the purpose

scalaapache-spark

Answers

answered 6 days ago antonioACR1 #1

How about

val diff1=df1.except(df2)
val diff2=df2.except(df1)
val join=diff1.unionAll(diff2)

Then join.select("id","CNSM_ID").

comments powered by Disqus