Symptoms
Apache Spark Web UI URLs are currently accessible by anybody without any authentication.
It is a big problem for deployments when schema admins need to access to Spark Master/Worker/Job UIs to see MV jobs and SQLi Jobs and these ports are exposed.
Solution
Incorta Security Filter for Spark can protect Spark UIs from unauthorized access while allowing only incorta users to see job details securely.
Steps to apply incorta spark UI with user authentication :
1- Go to the following path:
Incorta installation path / IncortaNode / spark / jars
2- Uncompress and add the attached jar.
3- Vim file
Incorta installation path / IncortaNode / spark / conf /spark-defaults.conf
4- Add the following configurations in
spark.ui.filters com.incorta.BasicAuthFilter
spark.com.incorta.BasicAuthFilter.param.realm incorta
spark.com.incorta.BasicAuthFilter.param.url http://localhost:8080
5- Replace http://localhost:8080 with incorta analytic URL and port
Note:
For clustered machines with many analytics services on different machines, u can add the above to any node in the spark configurations.
For external spark add the same jar and same configurations in
external spark path / spark / jars
external spark path / spark / conf /spark-defaults.conf
.png "Employee"){kind=icon}
