SNOW-264052: Data inconsistency when writing to Snowflake using multithreading

[vaibhavsingh007]

Hi, I heard concurrent write to snowflake using same jdbc connection is threadsafe, as it ought to be however, I encountered the following issue:

"
Hi, so I just ran into an issue where k records from across n Spark dataframes are written as k+y where y is non-deterministic, using concurrent write using same SF jdbc connection (drivers: spark-snowflake_2.11-2.5.2-spark_2.4.jar, snowflake-jdbc-3.9.1.jar).

write config:

SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
    
    # Set options below
    sfOptions = {
        "sfURL" : "****",
        "sfAccount" : "****", # Also needed create stage privilege.
        "sfUser" : username,
        "sfPassword" : password,
        "sfDatabase" : database,
        "sfSchema" : schema,
        "sfWarehouse" : warehouse,
        "sfRole" : role
    }
    
    sparkDF.write \
    .format(SNOWFLAKE_SOURCE_NAME) \
    .options(**sfOptions) \
    .option("dbtable", f"MY_WORKSPACE.{table}") \
    .mode('append') \
    .save()

How can I synchronize this to achieve data integrity?
"

ref: #3 (comment)

[vaibhavsingh007]

Issue resolved using latest drivers, spark-snowflake_2.11-2.8.3-spark_2.4.jar, snowflake-jdbc-3.12.17.jar
The data is now consistent.