Spark SQL between statement

Learn Hadoop Source

How to find the salary between two ranges. Here is my spark sql dataFrame code for finding the salary between 10000 to 20000

empData.where($"Salary" > 10000 && $"Salary" < 20000).orderBy($"Salary".desc).show()

I believe have alternative solution to use between. How can i use between approach?



answered 5 days ago Navom Saxena #1

If you wish to use between, you can use sparkSQL and run logic as query. For e.g.


val filteredData = spark.sql("select * from empDataTempTable where salary between 10000 and 20000 order by salary desc")

For random lookups in a column and filter process, sparkSQL and DataFrame gives almost same performance results.

answered 5 days ago user10494677 #2

I believe have alternative solution to use between.

Just like that:

empData.where($"Salary".between(10000, 20000))

comments powered by Disqus