Common ERROR on spark in Scala, Java or python

1. ImportError: cannot import name SparkSession

2. java.io.EOFException

3. database not found , table not found.

Hello, today we are going to discuss few APACHE SPARK error which we mostly get when we use Scala or Python API.

As of now we are writing for only 3, but there are many more so let us kwon in comment section.

1. When we submit our Pyspark application code to run by "spark-submit" command then we get exception:

"ImportError: cannot import name SparkSession"

Solution: possibly this error comes when you are using spark 1 or any version which is less then 2.0 . to overcome this issue you need to export spark session version before running spark-submit or spark-shell command:

export SPARK_MAJOR_VERSION=2

2. This error come when you work with Pyspark. It's related to memory management and optimization in pyspark job.

Caused by: java.io.EOFException

This error come's when there is less memory on python process, spark has taken all the memory and nothing left for python. So to solve this issue you need to consider below points:

1. Add more memory to server,

2. Unpersist RDDs,DF if not needed OR (remove possible df.persist)

3. Manage memory carefully (add some constraints on Spark memory usage)

3. After submitting spark-submit with python code we generally get this exception: table not found or database not found.

This happens because application in not connected with hive meta store. So to solve this error you need to follow below point:

1. Enable hive support while creating spark session: SparkSession spark = SparkSession.builder().appName("MyAp").enableHiveSupport().getOrCreate()

2. If still issue persist then set .conf("spark.sql.warehouse.dir","hdfs://host/user/hive/warehouse")

3. If still issue persist then pass hive-site.xml file with spark submit like: spark-submit --files /usr/local/hive/conf/hive-site.xml code.py

There are lot of other issue we get, will update the post once i have list of more issues in coming days on the same page, till that thanks for visit.