diff --git a/README.md b/README.md index 6547e3c..69c6d30 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # GitHub Copilot Usage Lambda -This repository contains the AWS Lambda Function for updating the GitHub Copilot dashboard's historic information, stored within an S3 bucket. +This repository contains the AWS Lambda Function for updating the GitHub Copilot dashboard's organisation-wide historic data, Copilot teams, and teams history. The Copilot dashboard can be found on the Copilot tab within the Digital Landscape. @@ -16,7 +16,7 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca - [Makefile](#makefile) - [AWS Lambda Scripts](#aws-lambda-scripts) - [Setup - Running in a container](#setup---running-in-a-container) - - [Setup - running outside of a Container (Development only)](#setup---running-outside-of-a-container-development-only) + - [Setup - running outside of a Container (Development only)](#running-outside-of-a-container-development-only) - [Storing the container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr) - [Deployment to AWS](#deployment-to-aws) - [Deployment Prerequisites](#deployment-prerequisites) @@ -138,7 +138,23 @@ Further information can be found in [this project's documentation](/docs/index.m docker stop 3f7d64676b1a ``` -### Setup - running outside of a Container (Development only) +### Setup + +Export the required environment variables: + + ```bash + export AWS_ACCESS_KEY_ID= + export AWS_SECRET_ACCESS_KEY= + export AWS_DEFAULT_REGION=eu-west-2 + export AWS_SECRET_NAME= + export GITHUB_ORG=ONSDigital + export GITHUB_APP_CLIENT_ID= + export AWS_ACCOUNT_NAME= + ``` + +The lambda can be run outside of a container for development purposes, or inside a container image to push to AWS ECR. + +#### Running outside of a Container (Development only) To run the Lambda function outside of a container, we need to execute the `handler()` function. @@ -153,25 +169,13 @@ To run the Lambda function outside of a container, we need to execute the `handl **Please Note:** If uncommenting the above in `main.py`, make sure you re-comment the code _before_ pushing back to GitHub. -2. Export the required environment variables: - - ```bash - export AWS_ACCESS_KEY_ID= - export AWS_SECRET_ACCESS_KEY= - export AWS_DEFAULT_REGION=eu-west-2 - export AWS_SECRET_NAME= - export GITHUB_ORG=ONSDigital - export GITHUB_APP_CLIENT_ID= - export AWS_ACCOUNT_NAME= - ``` - -3. Run the script. +2. Run the script. ```bash python3 src/main.py ``` -### Storing the container on AWS Elastic Container Registry (ECR) +#### Storing the container on AWS Elastic Container Registry (ECR) When you make changes to the Lambda Script, a new container image must be pushed to ECR. @@ -294,13 +298,7 @@ If the application has been modified, the following can be performed to update t The reconfigure options ensures that the backend state is reconfigured to point to the appropriate S3 bucket. - **_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place. - This can be done using: - - ```bash - export AWS_ACCESS_KEY_ID="" - export AWS_SECRET_ACCESS_KEY="" - ``` + **_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place. Please refer to [setup](#setup). - Refresh the local state to ensure it is in sync with the backend diff --git a/docs/diagrams/architecture.drawio b/docs/diagrams/architecture.drawio index 251aee0..c789a4f 100644 --- a/docs/diagrams/architecture.drawio +++ b/docs/diagrams/architecture.drawio @@ -1,53 +1,52 @@ - - - + + + - - - - - - - + + + + + + + + + + + + + - - - - - + + - - + + - - + + - + - - + + - - + + - - - - - - - + - - + + + + - - + + diff --git a/docs/diagrams/architecture.png b/docs/diagrams/architecture.png index cb3a54d..76899c9 100644 Binary files a/docs/diagrams/architecture.png and b/docs/diagrams/architecture.png differ diff --git a/docs/index.md b/docs/index.md index 553c753..8c7b58c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,42 +1,17 @@ -# GitHub Copilot Usage Dashboard +# GitHub Copilot Usage Lambda ## Overview -This project contains an AWS Lambda Function which updates the GitHub Copilot dashboard's historic information, stored within an S3 bucket. +This project contains a Python script to automatically update the GitHub Copilot dashboard's historic information for the Copilot dashboard within the Digital Landscape. This allows trend analysis into how Software Engineers are using Copilot within ONS over time. The Copilot dashboard can be found on the Copilot tab within the Digital Landscape. [View the Digital Landscape's repository](https://github.com/ONS-Innovation/keh-digital-landscape). -## Techstack Overview - -## Architecture Overview - -![Architecture Diagram](./diagrams/architecture.png) - -This project uses 2 major components: - -- The Lambda Function -- The GitHub API Toolkit (**stored in another repository** - [Repository Link](https://github.com/ONS-Innovation/github-api-package)) - -### The Lambda Function - -This component updates the dashboard's historic information, stored within an S3 bucket. The lambda imports the GitHub API Toolkit to get the API response containing the usage information. The script then adds any new data to the existing historic data within the S3 bucket. - -### The GitHub API Toolkit - -This component is an imported library which is shared across multiple GitHub tools. The toolkit allows applications to make authenticated requests to the GitHub API. It is imported and used by both the dashboard and lambda function. - -## High Level Data Overview - -### Endpoint - -[View docs for the Copilot usage data endpoint](https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28#get-a-summary-of-copilot-usage-for-organization-members). - -### Historic Data - -This section gathers data from AWS S3. The Copilot usage endpoints have a limitation where they only return the last 28 days worth of information. To get around this, the project has an AWS Lambda function which runs weekly and stores data within an S3 bucket. - ## Getting Started To setup and use the project, please refer to the [README](https://github.com/ONS-Innovation/github-copilot-usage-lambda/blob/main/README.md). + +## Technical Documentation + +For information about the technical aspects of the Copilot Usage Lambda, please refer to the [Technical Documentation](./technical_documentation/overview.md). diff --git a/docs/configuration.md b/docs/technical_documentation/configuration.md similarity index 93% rename from docs/configuration.md rename to docs/technical_documentation/configuration.md index f3ca326..03bd52b 100644 --- a/docs/configuration.md +++ b/docs/technical_documentation/configuration.md @@ -15,7 +15,7 @@ The `config.json` file contains the following: } ``` -### `features` Section +### `features` This section contains feature flags that control which of the tool's features are enabled or disabled. @@ -29,8 +29,6 @@ When deploying to AWS, this should be set to `false` to avoid files being writte If set to `true`, the tool will skip writing to the appropriate AWS S3 bucket and instead write data for copilot teams, historic usage, and teams history to `local_data`. -**When deploying to AWS, this must be set to `false` to ensure the tool writes to AWS.** - When debugging locally, you can set this to `true` to use the local configuration file. This is useful if you need to see the logs locally, without affecting the cloud deployment. ### Example During Local Testing diff --git a/docs/technical_documentation/overview.md b/docs/technical_documentation/overview.md new file mode 100644 index 0000000..880040d --- /dev/null +++ b/docs/technical_documentation/overview.md @@ -0,0 +1,44 @@ +# Overview + +AWS Lambda Function which updates the GitHub Copilot dashboard's: + +- Organisation-wide historic data +- Copilot teams +- Teams history + +## Tech Stack Overview + +This project uses: + +- Python +- AWS Lambda +- AWS S3 + +## Architecture Overview + +![Architecture Diagram](../diagrams/architecture.png) + +This project uses 2 major components: + +- The Lambda Function +- The GitHub API Toolkit (**stored in another repository** - [Repository Link](https://github.com/ONS-Innovation/github-api-package)) + +### The Lambda Function + +This component updates the Digital Landscape's Copilot dashboard data, stored within S3 buckets. The lambda imports the GitHub API Toolkit to get the API response containing the data, then adds any new data to the relevant S3 bucket. + +### The GitHub API Toolkit + +This component is an imported library which is shared across multiple GitHub tools. The toolkit allows applications to make authenticated requests to the GitHub API. It is imported and used by both the dashboard and lambda function. + +### Endpoint + +[View docs for the Copilot usage data endpoint](https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28#get-a-summary-of-copilot-usage-for-organization-members). + +### Historic Usage Data + +This section gathers data from AWS S3. The Copilot usage endpoints have a limitation where they only return the last 100 days worth of information. To get around this, the project has an AWS Lambda function which runs weekly and stores data within an S3 bucket. + +### Copilot Teams Data + +This section gathers a list of teams within the organisation with Copilot data and updates the S3 bucket accordingly. This allows all relevant teams to be displayed within the dashboard. diff --git a/docs/team_usage.md b/docs/technical_documentation/team_usage.md similarity index 79% rename from docs/team_usage.md rename to docs/technical_documentation/team_usage.md index e814c71..cc43d8f 100644 --- a/docs/team_usage.md +++ b/docs/technical_documentation/team_usage.md @@ -19,17 +19,15 @@ To retrieve data, a team must meet the following criteria: When on the team usage page, a `Login with GitHub` is displayed. Once clicked on, the user is taken to GitHub's authorisation page. The permissions **read-only** personal user data and **read-only** organizations and teams are required for this application. Clicking on the green `Authorize` button takes the user back to the application. -The session is saved to streamlit's `session_state`. For security, the session resets after refresh but not when navigating between pages. This means the user has to press the `Login with GitHub` button each page refresh. - ### Types of Access #### Regular User -A user within ONSDigital. Upon authentication, the app identifies the teams they belong to and populates the selection box accordingly. If the user is part of a qualifying team, they can view the data. Users not associated with any team cannot select teams. +A user within ONSDigital. Upon authentication, the app identifies the teams they belong to and populates the UI selection accordingly. If the user is part of a qualifying team, they can view the data. Users not associated with any team cannot select teams. #### Admin User -An enhanced regular user with the ability to search for any team. This user belongs to a specific whitelisted team, enabling them to view metrics for any team that meets the Copilot usage data requirements. +An enhanced regular user with the ability to view any team. This user belongs to a specific whitelisted team, enabling them to view metrics for any team that meets the Copilot usage data requirements. ## Metrics