example AWS Glue pyspark job script that fetches all the catalog databases and tables metadata.
see main.py
- first method uses spark sql
- second method uses python boto3 Glue client to interact with Glue API directly
ensure --enable-glue-datacatalog is enabled for glue job to allow spark sql to access metadata catalog
Glue Console Script View
