Adding AWS Glue connector code

managed-connectivity/community-contributed-connectors/aws-glue-connector/Dockerfile

@@ -0,0 +1,35 @@
+# Step 1: Use an official OpenJDK base image, as Spark requires Java
+FROM openjdk:11-jre-slim
+
+# Step 2: Set environment variables for Spark and Python
+ENV SPARK_VERSION=3.5.0
+ENV HADOOP_VERSION=3
+ENV SPARK_HOME=/opt/spark
+ENV PATH=$SPARK_HOME/bin:$PATH
+ENV PYTHONUNBUFFERED=1
+
+# Step 3: Install Python, pip, and other necessary tools
+RUN apt-get update && \
+    apt-get install -y python3 python3-pip curl && \
+    rm -rf /var/lib/apt/lists/*
+
+# Step 4: Download and install Spark
+RUN curl -fSL "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" -o /tmp/spark.tgz && \
+    tar -xvf /tmp/spark.tgz -C /opt/ && \
+    mv /opt/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} ${SPARK_HOME} && \
+    rm /tmp/spark.tgz
+
+# Step 5: Set up the application directory
+WORKDIR /app
+
+# Step 6: Copy and install Python dependencies
+COPY requirements.txt .
+RUN pip3 install --no-cache-dir -r requirements.txt
+
+# Step 7: Copy your application source code
+COPY src ./src
+COPY config.json .
+COPY pyspark_job.py .
+
+# Step 8: Define the entry point for running the PySpark job
+ENTRYPOINT ["spark-submit", "pyspark_job.py"]

managed-connectivity/community-contributed-connectors/aws-glue-connector/README.md

@@ -0,0 +1,230 @@
+# AWS Glue to Google Cloud Dataplex Connector
+
+This connector extracts metadata from AWS Glue and transforms it into a format that can be imported into Google Cloud Dataplex. It captures database, table, and lineage information from AWS Glue and prepares it for ingestion into Dataplex, allowing you to catalog your AWS data assets within Google Cloud.
+
+This connector is designed to be run from a Python virtual environment.
+
+***
+
+## Prerequisites
+
+Before using this connector, you need to have the following set up:
+
+1.  **AWS Credentials**: You will need an AWS access key ID and a secret access key with permissions to access AWS Glue.
+2.  **Google Cloud Project**: A Google Cloud project is required to run the script and store the output.
+3.  **GCP Secret Manager**: The AWS credentials must be stored in a secret in Google Cloud Secret Manager.
+4.  **Python 3** and **pip** installed.
+
+***
+
+## AWS Credentials Setup
+
+This connector requires an IAM User with `GlueConsoleFullAccess` (or read-only equivalent) and `S3ReadOnly` (to download job scripts for lineage).
+
+1.  Create an IAM User in AWS Console.
+2.  Attach policies: `AWSGlueConsoleFullAccess`, `AmazonS3ReadOnlyAccess`.
+3.  Generate an **Access Key ID** and **Secret Access Key**.
+4.  Store these in GCP Secret Manager as a **JSON object**:
+    ```json
+    {
+      "access_key_id": "YOUR_AWS_ACCESS_KEY_ID",
+      "secret_access_key": "YOUR_AWS_SECRET_ACCESS_KEY"
+    }
+    ```
+
+***
+
+## Setup Resources
+
+To run this connector, you must first create the required Dataplex resources.
+
+### Required Catalog Objects
+
+Note: Before importing metadata, the Entry Group and all Entry Types and Aspect Types found in the metadata import file must exist in the target project and location. This connector requires the following Entry Group, Entry Types and Aspect Types:
+
+| Catalog Object | IDs required by connector |
+| :--- | :--- |
+| **Entry Group** | Defined in `config.json` as `entry_group_id` |
+| **Entry Types** | `aws-glue-database`  `aws-glue-table`  `aws-glue-view` |
+| **Aspect Types** | `aws-glue-database`  `aws-glue-table`  `aws-glue-view`  `aws-lineage-aspect` |
+
+See [manage entries and create custom sources](https://cloud.google.com/dataplex/docs/ingest-custom-sources) for instructions on creating Entry Groups, Entry Types, and Aspect Types.
+
+### Option 1: Automated Setup (Recommended)
+Run the provided script to create all resources automatically:
+
+```bash
+# Set your project and location
+export PROJECT_ID=your-project-id
+export LOCATION=us-central1
+export ENTRY_GROUP_ID=aws-glue-entries
+
+# Run the setup script
+chmod +x scripts/setup_dataplex_resources.sh
+./scripts/setup_dataplex_resources.sh
+```
+
+### Option 2: Manual Setup
+If you prefer to create them manually, ensure you define the following:
+
+**Entry Types:**
+*   `aws-glue-database`
+*   `aws-glue-table`
+*   `aws-glue-view`
+
+**Aspect Types:**
+*   `aws-glue-database`, `aws-glue-table`, `aws-glue-view` (Marker Aspects)
+*   `aws-lineage-aspect` (Schema below)
+
+<details>
+<summary>Click to see Schema for aws-lineage-aspect</summary>
+
+```json
+{
+  "type": "record",
+  "recordFields": [
+    {
+      "name": "links",
+      "type": "array",
+      "index": 1,
+      "arrayItems": {
+        "type": "record",
+        "recordFields": [
+          {
+            "name": "source",
+            "type": "record",
+            "index": 1,
+            "recordFields": [
+              { "name": "fully_qualified_name", "type": "string", "index": 1 }
+            ]
+          },
+          {
+            "name": "target",
+            "type": "record",
+            "index": 2,
+            "recordFields": [
+              { "name": "fully_qualified_name", "type": "string", "index": 1 }
+            ]
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+</details>
+
+For more details see [manage entries and create custom sources](https://cloud.google.com/dataplex/docs/ingest-custom-sources).
+
+***
+
+## Configuration
+
+The connector is configured using the `config.json` file. Ensure this file is present in the same directory as `main.py`.
+
+| Parameter | Description |
+| :--- | :--- |
+| **`aws_region`** | The AWS region where your Glue Data Catalog is located (e.g., "eu-north-1"). |
+| **`project_id`** | Your Google Cloud Project ID. |
+| **`location_id`** | The Google Cloud region where you want to run the script (e.g., "us-central1"). |
+| **`entry_group_id`** | The Dataplex entry group ID where the metadata will be imported. |
+| **`gcs_bucket`** | The Google Cloud Storage bucket where the output metadata file will be stored. |
+| **`aws_account_id`** | Your AWS account ID. |
+| **`output_folder`** | The folder within the GCS bucket where the output file will be stored. |
+| **`gcp_secret_id`** | The ID of the secret in GCP Secret Manager that contains your AWS credentials. |
+
+***
+
+## Running the Connector
+
+You can run the connector from your local machine using a Python virtual environment.
+
+### Setup and Execution
+
+1.  **Create a virtual environment:**
+    ```bash
+    python3 -m venv venv
+    source venv/bin/activate
+    ```
+2.  **Install the required dependencies:**
+    ```bash
+    pip install -r requirements.txt
+    ```
+3.  **Run the connector:**
+    Execute the `main.py` script. It will read settings from `config.json` in the current directory.
+    ```bash
+    python3 main.py
+    ```
+
+***
+
+## Output
+
+The connector generates a JSONL file in the specified GCS bucket and folder. This file contains the extracted metadata in a format that can be imported into Dataplex.
+
+***
+
+## Importing Metadata into Dataplex
+
+Once the metadata file has been generated, you can import it into Dataplex using a metadata import job.
+
+1.  **Prepare the Request File:**
+    Open the `request.json` file and replace the following placeholders with your actual values:
+    *   `<YOUR_GCS_BUCKET>`: The bucket where the output file was saved.
+    *   `<YOUR_OUTPUT_FOLDER>`: The folder where the output file was saved.
+    *   `<YOUR_PROJECT_ID>`: Your Google Cloud Project ID.
+    *   `<YOUR_LOCATION>`: Your Google Cloud Location (e.g., `us-central1`).
+    *   `<YOUR_ENTRY_GROUP_ID>`: The Dataplex Entry Group ID.
+
+2.  **Run the Import Command:**
+    Use `curl` to initiate the import. Replace `{project-id}`, `{location}`, and `{job-id}` in the URL.
+
+    ```bash
+    curl -X POST \
+         -H "Authorization: Bearer $(gcloud auth print-access-token)" \
+         -H "Content-Type: application/json; charset=utf-8" \
+         -d @request.json \
+         "https://dataplex.googleapis.com/v1/projects/{project-id}/locations/{location}/metadataJobs?metadataJobId={job-id}"
+    ```
+
+***
+
+## Metadata Extracted
+
+The connector maps AWS Glue objects to Dataplex entries as follows:
+
+| AWS Glue Object | Dataplex Entry Type |
+| :--- | :--- |
+| **Database** | `aws-glue-database` |
+| **Table** | `aws-glue-table` |
+| **View** | `aws-glue-view` |
+
+### Lineage
+The connector parses AWS Glue Job scripts (Python/Scala) to extract lineage:
+-   **Source**: `DataSource` nodes in Glue Job graph.
+-   **Target**: `DataSink` nodes in Glue Job graph.
+-   **Result**: Lineage is visualized in Dataplex from Source Table -> Target Table.
+
+***
+
+## Docker Setup
+
+You can containerize this connector to run on Cloud Run, Dataproc, or Kubernetes.
+
+1.  **Build the Image**:
+    ```bash
+    docker build -t aws-glue-connector:latest .
+    ```
+
+2.  **Run Locally** (passing config):
+    Ensure `config.json` is in the current directory or mounted.
+    ```bash
+    docker run -v $(pwd)/config.json:/app/config.json -v $(pwd)/src:/app/src aws-glue-connector:latest
+    ```
+
+3.  **Push to GCR/Artifact Registry**:
+    ```bash
+    gcloud auth configure-docker
+    docker tag aws-glue-connector:latest gcr.io/YOUR_PROJECT/aws-glue-connector:latest
+    docker push gcr.io/YOUR_PROJECT/aws-glue-connector:latest
+    ```

managed-connectivity/community-contributed-connectors/aws-glue-connector/build_and_push_docker.sh

@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# Terminate script on error
+set -e
+
+# --- Read script arguments ---
+POSITIONAL=()
+while [[ $# -gt 0 ]]
+do
+key="$1"
+
+case $key in
+    -p|--project_id)
+    PROJECT_ID="$2"
+    shift # past argument
+    shift # past value
+    ;;
+    -r|--repo)
+    REPO="$2"
+    shift # past argument
+    shift # past value
+    ;;
+    -i|--image_name)
+    IMAGE_NAME="$2"
+    shift # past argument
+    shift # past value
+    ;;
+    *)    # unknown option
+    POSITIONAL+=("$1") # save it in an array for later
+    shift # past argument
+    ;;
+esac
+done
+set -- "${POSITIONAL[@]}" # restore positional parameters
+
+# --- Validate arguments ---
+if [ -z "$PROJECT_ID" ]; then
+    echo "Project ID not provided. Please provide project ID with the -p flag."
+    exit 1
+fi
+
+if [ -z "$REPO" ]; then
+    # Default to gcr.io/[PROJECT_ID] if no repo is provided
+    REPO="gcr.io/${PROJECT_ID}"
+    echo "Repository not provided, defaulting to: ${REPO}"
+fi
+
+if [ -z "$IMAGE_NAME" ]; then
+    IMAGE_NAME="aws-glue-to-dataplex-pyspark"
+    echo "Image name not provided, defaulting to: ${IMAGE_NAME}"
+fi
+
+IMAGE_TAG="latest"
+IMAGE_URI="${REPO}/${IMAGE_NAME}:${IMAGE_TAG}"
+
+# --- Build the Docker Image ---
+echo "Building Docker image: ${IMAGE_URI}..."
+# Use the Dockerfile for PySpark
+docker build -t "${IMAGE_URI}" -f Dockerfile .
+
+if [ $? -ne 0 ]; then
+    echo "Docker build failed."
+    exit 1
+fi
+echo "Docker build successful."
+
+# --- Run the Docker Container ---
+echo "Running the PySpark job in a Docker container..."
+echo "Using local gcloud credentials for authentication."
+
+# We mount the local gcloud config directory into the container.
+# This allows the container to use your Application Default Credentials.
+# Make sure you have run 'gcloud auth application-default login' on your machine.
+docker run --rm \
+    -v ~/.config/gcloud:/root/.config/gcloud \
+    "${IMAGE_URI}"
+
+if [ $? -ne 0 ]; then
+    echo "Docker run failed."
+    exit 1
+fi
+
+echo "PySpark job completed successfully."
+
+# --- Optional: Push to Google Container Registry ---
+read -p "Do you want to push the image to ${REPO}? (y/n) " -n 1 -r
+echo
+if [[ $REPLY =~ ^[Yy]$ ]]
+then
+    echo "Pushing image to ${REPO}..."
+    gcloud auth configure-docker
+    docker push "${IMAGE_URI}"
+    echo "Image pushed successfully."
+fi
+

managed-connectivity/community-contributed-connectors/aws-glue-connector/config.json

@@ -0,0 +1,10 @@
+{
+  "aws_region": "<YOUR_AWS_REGION>",
+  "project_id": "<GCP_PROJECT>",
+  "location_id": "<GCP_REGION>",
+  "entry_group_id": "<DATAPLEX_ENTRY_GROUP>", 
+  "gcs_bucket": "<GCS_BUCKET>",
+  "aws_account_id": "<AWS_ACCOUNT_ID>",
+  "output_folder": "<GCS_FOLDER_NAME>",
+  "gcp_secret_id": "<GCP_SECRET_ID>"
+}

managed-connectivity/community-contributed-connectors/aws-glue-connector/main.py

@@ -0,0 +1,8 @@
+import sys
+from src import bootstrap
+
+# Allow shared files to be found when running from command line
+sys.path.insert(1, '../src/shared')
+
+if __name__ == '__main__':
+    bootstrap.run()