Create a key named -conf for your AWS Glue job, and set it to the following value. For more information, see AWS Glue job parameters. config("_catalog.type", "hive")Īn error occurred while calling o87.sql. To enable Iceberg for AWS Glue, complete the following tasks: Specify iceberg as a value for the -datalake-formats job parameter. config("_catalog", ".SparkSessionCatalog") Updated the config to this (based on iceberg table configuration): spark = ("", "25g") I've tried validating my table using aws glue get-table -database-name dev_db -name temp_tag_thrshld_iceberg For more information, see AWS Regions and Endpoints in the Amazon Web Services General Reference.
I'm also not able to read the data directly from S3 as its an ORC format with Snappy compression so I don't get any results (I'm probably missing the correct framework to read S3 ORC directly but that's another issue for another day) Data streams AWS Glue for Spark can stream data from the following systems: Amazon Kinesis Data Streams Apache Kafka AWS Glue is available in several AWS Regions.
#AWS GLUE ICEBERG FULL#
You can refer to the Glue Developer Guide for a full explanation of the Glue Data Catalog functionality.
#AWS GLUE ICEBERG CODE#
If I replace my spark.sql("Select * from a_normal_athena_table) the code runs fine. hashicorp aws Version 5.4.0 Latest Version aws Overview Documentation Use Provider Resource: awsgluecatalogtable Provides a Glue Catalog Table Resource. Temp_tag_thrshld_data = spark.sql("SELECT * FROM dev_db.temp_tag_thrshld_iceberg") This is my code : spark = ("", "25g").appName(app_name).getOrCreate() Temp_tag_thrshld_iceberg (Service: null Status Code: 0 Error Code: StorageDescriptor#InputFormat cannot be null for table: Unable to fetch table temp_tag_thrshld_iceberg. I'm getting this error -ĪnalysisException: .ql.metadata.HiveException: I'm trying to read data from an iceberg table, the data is in ORC format and partitioned by column.
0 kommentar(er)
