Skip to main content

Hybrid setup betaenterprise

Set up Hybrid projects to upload dbt Core artifacts into dbt Cloud for better collaboration and visibility.

Available in private beta

Hybrid project is available in private beta to dbt Cloud Enterprise accounts. To register your interest in the beta, reach out to your account representative.

Set up Hybrid projects​

In a hybrid project, you use dbt Core locally and can upload artifacts of that dbt Core project to dbt Cloud for central visibility, cross-project referencing, and easier collaboration.

This setup requires connecting your dbt Core project to a dbt Cloud project and configuring a few environment variables and access settings.

Follow these steps to set up a dbt Cloud Hybrid project and upload dbt Core artifacts into dbt Cloud:

Make sure to enable the hybrid projects toggle in dbt Cloud’s Account settings page.

Make dbt Core models public (optional)​

This step is optional and and only needed if you want to share your dbt Core models with other dbt Cloud projects using the cross-project referencing feature.

Before connecting your dbt Core project to a dbt Cloud project, make sure models that you want to share have access: public in their model configuration. This setting makes those models visible to other dbt Cloud projects for better collaboration, such as cross-project referencing.

  1. The easiest way to set this would be in your dbt_project.yml file, however you can also set this in the following places:

    • dbt_project.yml (project-level)
    • properties.yml (for individual models)
    • A model's .sql file using a config block

    Here's an example using a dbt_project.yml file where the marts directory is set as public so they can be consumed by downstream tools:

    dbt_project.yml
    models:
    define_public_models: # This is my project name, remember it must be specified
    marts:
       +access: public

  2. After defining access: public, rerun a dbt execution in the dbt Core command line interface (CLI) (like dbt run) to apply the change.

  3. For more details on how to set this up, see access modifier and access config.

Create hybrid project​

Create a hybrid project in dbt Cloud to allow you to upload your dbt Core artifacts to dbt Cloud.

A dbt Cloud account admin should perform the following steps and share the artifacts information with a dbt Core user:

  1. To create a new project in dbt Cloud, navigate to Account home.
  2. Click on +New project.
  3. Fill out the Project name. Name the project something that allows you to recognize it's a dbt Core project.
    • You don't need to set up a data warehouse or Git connection, however to upgrade the hybrid project to a full dbt Cloud project, you'd need to set up data warehouse and Git connection.
  4. Select the Advanced settings toggle and then select the Hybrid development checkbox. Click Continue.
    • The hybrid project will have a visible Hybrid indicator in the project list to help you identify it.
Hybrid project new projectHybrid project new project
  1. After creating a project, create a corresponding production environment and click Save. Note that you can leave the Connection field blank.
  2. (Optional) To update an existing dbt project to a hybrid project, navigate to Account settings and then select the Project. Click Edit and then check the Hybrid development checkbox.
Hybrid project for an existing projectHybrid project for an existing project

Generate service token and artifact upload values​

A dbt Cloud admin should perform these steps to generate a service token and copy the values needed to configure a dbt Core project so it's ready to upload generated artifacts to dbt Cloud. The dbt Cloud admin should share the values with a dbt Core user.

  1. Go to the Hybrid project environment you created in the previous step by navigating to Deploy > Environments and selecting the environment.
  2. Select the Artifact upload button and copy the following values, which the dbt Core user will need to reference in their dbt Core's dbt_project.yml configuration:
    • Tenant URL
    • Account ID
    • Environment ID
    • Create a service token
      • dbt Cloud creates a service token with Job Runner or Job Viewer permissions.
      • Note if you don't see the Create service token button, it's likely you don't have the necessary permissions to create a service token. Contact your dbt Cloud admin to either get the necessary permissions or create the service token for you.
Generate hybrid project service tokenGenerate hybrid project service token
  1. Make sure to copy and save the values as they're needed to configure your dbt Core project in the next step. Once the service token is created, you can't access it again.

Configure dbt Core project and upload artifacts​

Once you have the values from the previous step, you can prepare your dbt Core project for artifact upload by following these steps:

  1. Check your dbt version by running dbt --version and you should see the following:

       Core:
       - installed: 1.10.0-b1
       - latest:    1.9.3     - Ahead of latest version!

  2. If you don't have the latest version, upgrade your dbt Core project by running python -m pip install --upgrade dbt-core.

  3. Set the following environment variables in your dbt Core project by running the following commands in the CLI. Replace the your_account_id, your_environment_id, and your_token with the actual values in the previous step.

    export DBT_CLOUD_ACCOUNT_ID=your_account_id
    export DBT_CLOUD_ENVIRONMENT_ID=your_environment_id
    export DBT_CLOUD_TOKEN=your_token
    export DBT_UPLOAD_TO_ARTIFACTS_INGEST_API=True
    • Set the environment variables in whatever way you use them in your project.
    • To unset an environment variable, run unset environment_variable_name, replacing environment_variable_name with the actual name of the environment variable.

  4. In your local dbt Core project, add the following items you copied in the previous section to the dbt Core's dbt_project.yml file:

    • tenant_hostname
    name: "jaffle_shop"
    version: "3.0.0"
    require-<Constant name="dbt" />-version: ">=1.5.0"
    ....rest of dbt_project.yml configuration...

    <Constant name="dbt" />-cloud:
      tenant_hostname: cloud.getdbt.com # Replace with your Tenant URL

  5. Once you set the environment variables using the export command in the same dbt Core CLI session, you can execute a dbt run in the CLI.

     dbt run

    To override the environment variables set, execute a dbt run with the environment variable prefix. For example, to use a different account ID and environment ID:

     DBT_CLOUD_ACCOUNT_ID=1 DBT_CLOUD_ENVIRONMENT_ID=123 dbt run

  6. After the run completes, you should see a Artifacts uploaded successfully to artifact ingestion API: command run completed successfully message and a run in dbt Cloud under your production environment.

Review artifacts in dbt Cloud​

Now that you've uploaded dbt Core artifacts into dbt Cloud and executed a dbt run, you can view the artifacts job run:

  1. Navigate to Deploy
  2. Click on Jobs and then the Runs tab.
  3. You should see a job run with the status Success with a </> Artifact ingestion indicator.
  4. Click on the job run to review the logs to confirm a successfully artifacts upload message. If there are any errors, resolve them by checking out the debug logs.
Hybrid project job run with artifact ingestionHybrid project job run with artifact ingestion

Benefits of using Hybrid projects​

Now that you've integrated dbt Core artifacts with your dbt Cloud project, you can now:

  • Collaborate with dbt Cloud users by enabling them to visualize and perform cross-project references to dbt models that live in Core projects.
  • (Coming soon) New users interested in the Visual Editor can build off of dbt models already created by a central data team in dbt Core rather than having to start from scratch.
  • dbt Core users can navigate to Explorer and view their models and assets. To view Explorer, you must have a read-only seat.
0
Edit this page