feat: spark 4.1 support #761
Description
Is your feature request related to a problem? Please describe.
https://spark.apache.org/news/spark-4-1-0-released.html
Describe the solution you would like
Tests are passed with spark 4.1
Component
- Scala Core Internal
- Scala API
- Spark Connect Plugin
- Infrastructure
- PySpark Classic
- PySpark Connect
Additional context
Are you planning on creating a PR?
- I'm willing to make a pull-request