Issue while upgrading Spark Verison from 3.3 to 3.4 in Azure Synapse Studio.

[ANSHUL22122001]

While upgrading the pyspark version of sparkpools to 3.4 in Synapse Studio. We faced this error

An error occurred while calling o6063.save.\n: java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)\n\tat com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)\n\tat com.microsoft.sqlserver.jdbc.spark.DefaultSource.createRelation(DefaultSource.scala:66)\n\tat

We have updgraded the pyspark(3.4.0) and delta-spark(2.4.0) version in requirement.txt file also but issue is same.

[ashkmr574]

even I am facing same issue. looks like extra parameter was added IsTimestampNTZ in getShema method in spark 3.4.0
https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L279

I see 1.4.0-BETA release , but not available in maven