Pondurance Home > Main Menu > Log Ingestion > Cloud API > GitHub

GitHub

Overview

Configuring a cloud source in Scope is a two-step process.

Generating the required AWS credentials (to ingest GitHub audit logs) in the AWS Management Console. Please refer to Section 1 – GitHub Setup
Setting up the GitHub cloud source in the Scope application. Please refer to Section 2 – Scope Setup

GitHub Setup (Generate the required AWS credentials)

To get started, you’ll need to obtain the following credentials from the AWS Management Console –

Access Key
Secret Access Key
AWS Region(s)
Bucket Name
Prefix (for individual data provider)

Step 1: Create a Custom IAM Policy

Sign in to the AWS Management Console.
Navigate to Identity and Access Management (type IAM in the search bar and select IAM from the dropdown).
Click on Policies in the left navigation pane.
Click on Create Policy.
In Policy Editor, select JSON.

In Specify Permissions, create a policy allowing only the following permissions:

ListBucket
GetObject

The following sample JSON can be used for creating the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListBucketWithPrefix",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-bucket",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "my-prefix/*"
                }
            }
        },
        {
            "Sid": "AllowObjectLevelActions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::my-bucket/my-prefix/*"
        }
    ]
}

Where:

my-bucket – Provide the name of the S3 bucket for which access should be granted.
my-prefix – Provide the prefix (folder path) within the bucket for which access should be granted.

For example, if the bucket name is log-stream-bucket and the prefix is audit-logs/, the policy JSON should be constructed as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListBucketWithPrefix",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::log-stream-bucket",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "audit-logs/*"
                }
            }
        },
        {
            "Sid": "AllowObjectLevelActions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::log-stream-bucket/audit-logs/*"
        }
    ]
}

Once the JSON is added in the Policy Editor, click Next.
In the Review and Create section, provide a name for the policy in the Policy Name field, and click Create Policy.
The custom policy is created.

Step 2: Create an IAM User

Sign in to the AWS Management Console and navigate to Identity and Access Management (for IAM navigation reference, see Step 1).
Click on Users in the left navigation pane.
Click on Create user.
In the Specify user details section, enter a name in the User name field and click Next.
In the Set Permissions section, select Attach policies directly. In Permissions policies, select the policy created in Step 1 (in this case S3_Log_Ingestion_Policy) and click Next.
In the Review and Create section, review the User details and Permissions summary, then click Create User.
The IAM user will be created.

Step 3: Generate Access Key & Secret Access Key

In the AWS Management Console, navigate to IAM -> Users (for IAM navigation reference, see Step 1).
Click on the created/required IAM user.
Select the Security Credentials tab.
Click on Create Access Key.
In the Access key best practices & alternatives section, select Third-party service, select the Confirmation checkbox, and click Next.
(Optional) In the Set description tag section, provide a description.
Click on Create access key.
In the Retrieve access keys section, the Access Key and Secret access key are generated. Copy them or click Download .csv file to download the keys. Store the keys in a safe location.

Note: The access keys can only be viewed or downloaded at this point. They cannot be recovered later. If the keys are lost or forgotten, generate new access keys by repeating this step and disable the old key.

The generated Access Key and Secret Access Key are to be configured in Scope Setup: Step 1 for initiating the GitHub log ingestion.

Step 4: Determine AWS Region

Sign in to the AWS Management Console.
Click on Profile Name and navigate to Account.
In the AWS Regions section, identify the required AWS Regions from which GitHub logs are to be ingested.

The identified AWS Region(s) are to be configured in Scope Setup: Step 1 for initiating the GitHub log ingestion.

Step 5: Determine the required AWS S3 bucket for GitHub logs

Sign in to the AWS Management Console.
In the Home Page search box, search for S3 and click on S3 from the search results.
In the S3 General Purpose Buckets section, identify the required bucket from which GitHub events are to be ingested.

The Bucket Name is to be configured in Scope Setup: Step 1 for initiating the GitHub log ingestion. This bucket must be the same destination bucket that you configure for GitHub audit log streaming in Step 6.

Step 6: GitHub Audit Log Streaming to AWS S3

Step 6.1: Create an Amazon S3 Bucket

Create an Amazon S3 bucket and ensure public access is blocked.
Refer to the AWS documentation for instructions on creating and configuring Amazon S3 buckets.

Step 6.2: Create an IAM Policy for GitHub

Create an IAM policy that allows GitHub to write audit events to the S3 bucket.

Replace EXAMPLE-BUCKET in the policy below with the name of your S3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::EXAMPLE-BUCKET/*"
    }
  ]
}

Refer to the AWS documentation for instructions on creating IAM policies.

Step 6.3: Configure GitHub Audit Log Streaming

Note: You will need a separate AWS Access Key ID and Secret Access Key with permissions to write to the target S3 bucket before configuring log streaming from GitHub.

Log in to GitHub Enterprise.
Navigate to Settings -> Audit Log -> Log Streaming.
Select Configure Stream -> Amazon S3.
Under Authentication, select Access Keys.

Configure the following settings:

Settings	Description
Region	Select Auto Detect or the AWS region where the S3 bucket is hosted.
Bucket	Enter the name of the S3 bucket.
Access Key ID	Enter the AWS Access Key ID.
Secret Key	Enter the AWS Secret Access Key.

Click Check Endpoint to verify that GitHub can connect to and write to the S3 bucket.
After successful validation, click Save.

Scope Setup

Step 1: GitHub Cloud Source Registration in the Scope Application

Once the credentials are generated, they must be configured in the Scope application to establish the connection and enable data ingestion from GitHub through AWS S3.

In the Scope application, to register a GitHub cloud source, navigate to the cloud source registration page –

Log into the Scope application
Select the required Organization from the Organization dropdown
Navigate to the side menu -> Administration
Navigate to the Cloud sources tab
Click on the +Add Source button
In the Add Source pop-up, provide the parameters below.
Source: Select the GitHub source from the Source dropdown.
Ingestion Method: Currently, only AWS S3 is supported, and it will be the default/available value.
Site: The user defined name for the GitHub cloud source.
Bucket Name: The bucket name identified in Step 5.
Access Key: The Access Key generated in Step 3.
Secret Access Key: The Secret Access Key generated in Step 3.
Polling Interval: The polling interval for making periodic API calls to the AWS S3 SDK for ingesting GitHub logs. Select the time interval from the dropdown.
Region: Select the region(s) from which GitHub events are ingested (identified in Step 4).
Contact Email: The email address of the person who registers the GitHub cloud source in Scope.

Once the required connection parameters are entered, the GitHub cloud source registration is complete in Scope and is ready for ingestion of GitHub logs.

Reference Links

Streaming GitHub logs to AWS S3