Unable to Load Trino Spark Connector in AWS Glue Job

[BD20171998]

Description
I'm experiencing issues loading the Trino Spark Connector in an AWS Glue job. Despite following the documentation and troubleshooting, I've been unable to resolve the issue.

Environment

• AWS Glue version: 4.0

• Python version: 3.10

• Spark version: 3.3.0

• Trino 469

Glue Job Submission

 spark_configurations = [

    f"spark.sql.extensions", "io.trino.plugin.spark.TrinoSparkSessionExtensions",
    f"spark.sql.catalog.mytrinocat", "io.trino.plugin.spark.catalog.TrinoCatalog",
    f"spark.sql.catalog.mytrinocat.catalog", "mytrinocat",
    f"spark.sql.catalog.mytrinocat.schema", trino_schema,
    f"spark.sql.catalog.mytrinocat.host", trino_host,  
    f"spark.sql.catalog.mytrinocat.port", trino_port,
    f"spark.sql.catalog.mytrinocat.user", trino_user,
    f"spark.sql.catalog.mytrinocat.password", trino_password,

    f"spark.logConf=true",
    f"spark.driver.log.level=DEBUG",
    f"spark.executor.log.level=DEBUG",
    f"spark.driver.logClassPath=true",
    f"spark.executor.logClassPath=true"
]
        

    spark_string = ' --conf '.join(spark_configurations)
    #spark_string = ';'.join(spark_configurations)

    job_args = {
        'Description': description,
        'Role': 'AWSGlueServiceRole',
        'ExecutionProperty': {
            "MaxConcurrentRuns": 3
        },
        'Command': {
            "Name": "glueetl",
            "ScriptLocation": script_path,
            "PythonVersion": "3"
        },
        'GlueVersion': '4.0',
        'WorkerType': 'Standard',
        'NumberOfWorkers': 1,
        'DefaultArguments': {
        "--extra-py-files": ",".join(extra_py_files),
        "--additional-python-modules":python_modules,
        "--extra-jars": "s3://mybucket/bd_cap/trino-drivers/trino-glue-bundle.jar",
        "--conf": spark_string,
        "--user-jars-first": "true"
     
    }
    }

Steps Taken

• I've referenced the Trino Spark Connector documentation and ensured I'm using the correct version.

• I've added the Trino Spark Connector dependency to my AWS Glue job.

• I've verified that the Trino Spark Connector JAR is included in the job's classpath.

For further context, I've also posted this issue on Stack Overflow: SO issue

Additional Troubleshooting
I've attempted to create a custom fat JAR using Maven, but this did not resolve the issue.
I've set the --user-jars-first option to true in my AWS Glue job configuration, but this did not resolve the issue.

Logging and Debugging
To further diagnose the issue, I've logged the driver and executor classpaths using spark.sparkContext.getConf().get('spark.driver.class.path') and spark.sparkContext.getConf().get('spark.executor.class.path'). The output reveals that both classpaths are None:

INFO:__main__:Driver classpath: None
INFO:__main__:Executor classpath: None

To investigate the loaded JARs, I've used the following code:

jars = spark.sparkContext._jvm.ClassLoader.getSystemResources("")
while jars.hasMoreElements():
    jar = jars.nextElement()
    logging.info(jar.toString())

The output shows the loaded JARs:

INFO:root:Loaded jars:
INFO:root:file:/tmp/
INFO:root:file:/opt/amazon/conf/

Expected Behavior
The Trino Spark Connector should load successfully in the AWS Glue job.

Actual Behavior
The Trino Spark Connector fails to load, resulting in an error.

Error Message

org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'mytrinocat': org.apache.spark.sql.trino.TrinoCatalog at org.apache.spark.sql.errors.QueryExecutionErrors$.catalogPluginClassNotFoundForCatalogError(QueryExecutionErrors.scala:1620)

Any help would be greatly appreciated!

[BD20171998]

I was able to resolve the issue using a different set of jars:

• iceberg-spark-runtime-3.3_2.12-1.5.2.jar

• iceberg-aws-bundle-1.5.2.jar

• hadoop-aws-3.2.0.jar

For further details pleasee see my StackOverflow post