Spark SQL (Dataset) Joins with Scala Examples





Spark joins are used for datasets. To join one or more datasets with join() function. Spark API contains join () function using in Scala classes to join huge datasets

Here are different types of Spark join() functions in Scala:

1.join
2.rightOuterJoin()
3.leftOuterJoin()

1. Join (): In Spark simple join is used for the inner join between two RDDs
example: rdd . join ( other)

2.rightOuterJoin() : It is used for join between two RDDs. In this join key mandatory in the first RDD

example: rdd.rightOuterJoin (other)

3. leftOuterJoin(): It is used to join between two RDDs. In this join key mandatory in the other RDD

example: rdd.leftOuterJoin(other)

A brief explanation for Spark join programming example with Scala coding:

val linesdata = sc.textFile("Datalog.txt")
val linesLength = linesdata.map(_.split("\t"))
linesdata.join(linesLength).collect()

Most of the cases, Spark SQL is using joins with RDBMS data structured.

Like an employee, customer data, and etc.

For example: Take an employee database with Schema:

Employee Schema:                        Job Schema:    
emp id, emp name, emp sal              emp id, company name

Here is the Spark with Scala partial code for Spark joins

case class Employee(id: Int, name: String, sal: Float);
case class Job (id: Int, c_name: String);
emp.join(job).collect()

Leave a Reply

Your email address will not be published. Required fields are marked *