Google Cloud Storage

Overview

Configuring a cloud source in Scope is a two-step process.

  • Generating Google Cloud Storage service account credentials in the Google Cloud Console. Please refer to Section 1 – Google Cloud Storage Setup

  • Setting up the Google Cloud Storage cloud source in the Scope application. Please refer to Section 2 – Scope Setup

Google Cloud Storage Setup

To get started, you’ll need to generate the following credentials in the Google Cloud Console –

  1. Service Account Key (JSON key file)
  2. Bucket Name
  3. Directory Path (for individual data providers)

Step 1: Enable the Cloud Storage API

  • Open the Google Cloud Console and make sure you are working in the correct project.

  • From the dashboard, navigate to APIs & Services.

  • Click on ENABLE APIS AND SERVICES.

  • In the API library, use the search box and search for Cloud Storage.

  • From the search results, click on Cloud Storage from the Google Enterprise API.

  • On the Cloud Storage API details page, click ENABLE.

    • If the API is already enabled, proceed to the next step.

Step 2: Create the GCS Bucket

  • Open the Google Cloud Platform (GCP) menu and navigate to Cloud Storage -> Buckets.

  • Click on the CREATE button to start creating a new bucket.

  • Configure your bucket with the following settings:

    • Bucket Name: Enter a globally unique name (e.g., gcs-aef220-dee21).
    • Storage Location: Choose a region for your data (you can accept the default or select a specific region).
    • Storage Class: Choose the default storage class (e.g., Standard).
    • Access Control: Set Public Access Prevention to On and use Uniform access control.
    • Data Protection: Configure options such as soft delete, versioning, retention policies, etc.

  • Click CREATE to proceed. A prompt will appear notifying you that public access will be prevented.

  • Click CONFIRM to proceed unless you have a specific use case that requires public access.

  • After the bucket is created, copy and save the Bucket Name, as you will need it later for Scope configuration (e.g., gcs-aef220-dee21).

Step 3: Create a Custom IAM Role

  • Sign in to the Google Cloud Console as a project editor.

  • Navigate to IAM & Admin → Roles and select Create Role.

  • Enter a Title (e.g., Get buckets & objects) and an optional Description for the custom role.

  • Click Add Permissions.

  • Filter the list of permissions and add the following:

    • storage.buckets.get
    • storage.objects.list

  • Click Add, then click Create.

Step 4: Set Up the Service Account and IAM Permissions

  • Navigate to IAM & Admin → Service Accounts.

  • Click CREATE SERVICE ACCOUNT.

  • Provide a Service Account Name (this automatically populates the Service Account ID field) and click CREATE AND CONTINUE.

    Note: Make sure you are clicking the CREATE AND CONTINUE button and not the more prominent DONE button.

  • In the role selection section:

    • Grant the Project -> Viewer role for the service account.

    • Click ADD ANOTHER ROLE and select the custom role created in Step 3.

  • To limit the service account to only this bucket, add an IAM condition by clicking ADD IAM CONDITION:

    • Create IAM Condition - Name your IAM condition and paste the bucket name into the value field. Set the condition as shown:

      • Condition Type: Name
      • Operator: is
      • Value: Name of the bucket created in Step 2

  • Click SAVE to apply the condition, then click DONE.

Step 5: Create Service Account Credentials (JSON Key)

  • In the Google Cloud Console, navigate to IAM & Admin → Service Accounts.

  • Go to Service Accounts and Select your service account.

  • Click Keys -> Add key -> Create new key.

  • Select JSON, then click Create.

    Your new public/private key pair is generated and downloaded to your machine as a JSON file. Save this file as credentials.json in your working directory.

    Note: This file is the only copy of this key, so save it in a secure location.

  • Click Close.

The generated Service Account Key (JSON file) and Bucket Name are to be configured in Scope Setup: Step 1 for initiating the Google Cloud Storage log ingestion.

Scope Setup

Step 1: Google Cloud Storage Cloud Source Registration in the Scope Application

Once the credentials are generated, they must be configured in the Scope application to establish the connection and enable data ingestion from the Google Cloud Storage environment.

In the Scope application, to register a Google Cloud Storage cloud source, navigate to the cloud source registration page –

  • Log into the Scope application

  • Select the required Organization from the Organization dropdown

  • Navigate to the side menu -> Administration

  • Navigate to the Cloud sources tab

  • Click on the +Add Source button

  • In the Add Source pop-up, provide the parameters below.

    • Source: Select the Google Storage - GCP source from the Source dropdown.

    Input Method Config Tab:

    • Site: The user defined name for the Google Cloud Storage cloud source.

    • Bucket Name: The bucket name created in Step 2.

    • Service Acc Key: Upload the credentials.json file generated in Step 5.

    • Polling Interval: The polling interval for making periodic API calls to Google Cloud Storage. Select the time interval from the dropdown.

    • Contact Email: The email address of the person who registers the Google Cloud Storage cloud source in Scope.

    Data Provider Tab:

    Configure the required data providers by selecting the appropriate checkbox and providing the Directory Path (prefix) for each:

    • Firewall – Select the Firewall checkbox and provide the required directory path prefix.
    • DNS – Select the DNS checkbox and provide the required directory path prefix.
    • VPC – Select the VPC checkbox and provide the required directory path prefix.
    • Audit – Select the Audit checkbox and provide the required directory path prefix.
    • CustomLogs – Select the CustomLogs checkbox and provide the required directory path prefix.

Once the required connection parameters are entered, the Google Cloud Storage cloud source registration is complete in Scope and is ready for ingestion of logs from Google Cloud Storage.

Configure Google Cloud logs to ingest into the Cloud Storage bucket

This section helps users configure the following Google Cloud logs to ingest into the Google Cloud Storage bucket:

  • Audit logs
  • VPC flow logs

Configuring Audit Logs to Cloud Storage

Prerequisites

Before you create a sink, make sure you have the following:

  • You have a Google Cloud folder or organisation with logs that you can see in Logs Explorer.

  • You have one of the following IAM roles for the Google Cloud organisation or folder from which you’re routing logs:

  • Logging Admin (roles/logging.admin)

  • Logs Configuration Writer (roles/logging.configWriter)

  • The permissions in these roles allow you to create, delete, or modify sinks.

  • You have a destination resource in a supported destination or the ability to create one. The routing destination must be created before the sink, through either Google Cloud CLI, Google Cloud console, or the Google Cloud APIs.

  • You can create the destination in any Cloud project, but the sink service account must have permissions to write to the destination.

    Example: There are two projects, project 1 and project 2.

Step 1: Create a storage bucket in Project 1

  • Create a GCS bucket in the destination project.

  • Once your GCS bucket is up and running, the next step is to create a sink.

  • Note: You can create up to 200 sinks per Cloud project.

Step 2: Create a Sink in Project 1

  • If you’re creating an aggregated sink at the folder level whose destination is a GCS bucket, your command might look like the following:

    gcloud logging sinks create SINK_NAME \
storage.googleapis.com/projects/PROJECT_ID \
--include-children \
--folder=FOLDER_ID \
--log-filter="logName:activity"

  • SINK_NAME is the name of the sink you want to give (for example, audit-sink).

  • PROJECT_ID should be replaced by the project ID.

  • The --include-children flag is important so that logs from all projects under the selected parent are also included.

    Note: The sink is created as soon as you execute the command, but it takes some time to start. You may see a “starting up” message in the sink row.

Step 3: Edit the sink destination

  • Click the three dots at the far right of your sink and select Edit sink.

  • Scroll down to Sink destination and select Cloud Storage bucket.

  • Select the GCS bucket created in Project 1.

  • In the inclusion filter, select Include logs ingested by this folder and all child resources.

  • In the filter text box, type:logName:activity

  • Save the changes. Note: For any new sink, if you don’t specify filters, all logs match and are routed to the sink destination. You can configure the sink to select specific logs by setting an inclusion filter, and you can also set exclusion filters.

Step 4: Set permissions for the sink

  • Click the three dots and select View sink details.

  • Copy the service account shown in Writer identity.

  • In the destination project, go to IAM and assign the service account the Storage Object Creator role.

    Note: Make sure the destination Cloud project contains the storage bucket you’re using to aggregate the logs.

Step 5: Generate logs to verify the sink

  • Generate activity to verify that the sink is routing logs correctly.

  • If the sink is using audit logs, one validation method is to start a VM in a different project (for example, project 2) and confirm the event appears in the logs.

  • Alternatively, perform IAM role or permission changes and verify that those events are captured.

Step 6: Verify output in the bucket

  • Check the GCS bucket to confirm logs are being populated.

  • Only logs created after the sink was created will be collected.

  • Logs are aggregated hourly, so allow some time after new activity for logs to appear.

  • Click a file in the bucket to view its contents.

  • Click the Authenticated URL link to open the log file.

  • The log output is displayed in JSON format.

Note: This configuration allows you to collect logs from your project and other projects into a single GCS bucket.

Configure VPC Logs to Cloud Storage

  • Sign in to Google Cloud with a privileged account.

  • On the Welcome page, click VPC Networks.

  • Click Default to open the subnet page.

  • Select All logs.

  • Click Flow Logs -> Configure.

  • Set Aggregation Interval (for example, 30 seconds).

  • Set Sample Rate (for example, 50%).

  • Click Save.

  • Search for Logging in the search bar and press Enter.

  • In Log Explorer, filter the logs by selecting VPC_flows in the Log Name and click Apply.

  • Click More Actions.

  • Click Create Sink.

  • Configure the sink:

    • Sink Details -> enter a name and description.

    • Click Next.

    • Sink Destination -> select Cloud Storage Bucket.

    • Cloud Storage Bucket -> select the bucket created in Step 2 or create a new bucket.

    • Click Next.

    • Choose Logs to include in Sink -> a default log is populated when you select the Cloud Storage bucket.

    • Click Next.

    • Optional: choose logs to filter out of the sink.

  • Click Create Sink.