Set Up Zero-Copy Data In with Databricks
Prerequisites
- Configure Databricks on AWS.
- Take note of your DatabricksWarehouse URLandAccount IDfrom your account details.
Task 1: Create a Zero-Copy Connection to Databricks
You can access Databricks table metadata without copying data by creating a zero-copy connection from Workday to your Databricks Iceberg catalog.
- In Workday, access the Prism Data Catalog report.
- Select+ Create>Connection.
- In theCreate Connectionwizard, underZero Copy Connectors, select theDatabrickstile.
- Enter the connection details:FieldDescriptionNameThe connection name must be unique in the Data Catalog. Example:Databricks_ZeroCopy_<first name><last initial>.Description(Optional) A brief description of the connection's purpose.Catalog URLThe URL of the Databricks catalog. Format:Databricks_Warehouse_URL/api/2.1/unity-catalog/iceberg-rest.Catalog NameDatabricks Catalog name.Authentication TypeSelectOAuth 2.0 with Token Exchange.IssuerIssuer identifier of the authorization server. Format:https://databricks.com/account/<account_id>/oidc/v1/token.PrincipalService Principal from your Databricks system.Token EndpointOAuth2 token endpoint URL from your external system. Format:Databricks_Warehouse_URL/oidc/v1/token.ScopesOAuth2 scopes from your Databricks system that define the level of access for the Principal.
- Save the connection but donottest it yet. Before you test the connection, you must complete the certificate setup in Databricks using the JWK certificate generated by Workday.
- On theView Connection Detailspage that displays, selectCertificate>Download>JWK.
- Update Databricks with the JWK Certificate.
- Log in to your Databricks account console.
- Locate your service principal used for the Zero-Copy Data In connection.
- Update the federation policy of your service principal, adding the JWK certificate content from Workday to the "keys" list in the existing JWKS JSON.
- Save your changes.
- In Workday, on theView Connection Detailspage, selectTest Connection.If the test fails, verify:
- Catalog URL format.
- Token endpoint URL.
- Service principal identity and account values.
- JWKS JSON federation policy configuration in Databricks.
Task 2: Create an External Catalog
An external catalog acts as a virtual window into your external data, allowing you to browse tables in real-time.
- Access theData Catalogreport.
- Select+ Create>External Catalog.
- Enter the external catalog details:FieldDescriptionExternal Catalog NameEnter a descriptive name for your external catalog. Example:EA_Procurement_CatalogorEA_HR_External_Data.ConnectionSelect the connection you just created.NamespaceThe exact namespace where the Iceberg connector tables reside. This information must match the Databricks system value exactly. Example:enterprise.finance.procurement.Description(Optional) The description of the external catalog.
- ClickSave.
- Wait for theSuccessnotification that the External Catalog was created.
- SelectBack to Data Catalog.
- SelectExternal Catalog, and validate that schema sync is successful on your External Catalog.
After you save the External Catalog, Workday automatically runs a
Schema Sync
to:- Read the metadata of Iceberg tables in the specified Databricks namespace.
- Register those tables asExternal Tablesin Workday.
- Make table structures visible in the Data Catalogwithout moving or copying data.
The duration of this sync depends on the number and complexity of tables in the namespace.
Task 3: Explore and Validate External Tables
Explore the external tables registered from the external partner and perform basic schema validation inside Workday.
- Locate the Databricks external tables.
- Access theData Catalogreport.
- In the left-hand menu, selectExternal Tables.
- Click on theRefreshbutton.
- In the External Tables view, confirm that all Iceberg tables from the external partner are listed under the schema/namespace you selected during external catalog creation.
- Inspect an external table.
- Right-click or double-click the table name to openView External Table Details.
- On the details page, review:
- Column names
- Data types
- Any available column statistics
- Sample data preview (if present)
- Verify that:
- Data types (Example: Numeric vs. string) align with the definitions in the external system.
- Values appear as expected.