Concept: Dataset Schema Changes
A schema is a set of rules that define how data is organized, structured, and constrained. Various Workday artifacts and components have a schema, including datasets and dataset components.
A dataset might consist of multiple schemas depending on how its configured. Each of these dataset components has its own schema:
- The files containing external data that you load into a base dataset.
- The custom report you create to bring Workday data into a base dataset.
- The input to a stage. For the first stage in a base dataset, this schema is determined by the dataset source.
- The output of a stage. For the last stage in the Primary Pipeline, this schema determines the output schema of the entire dataset.
If you're new to Workday, you don't have access to create or edit base datasets.
Schema changes can happen anywhere in a dataset. Schemas change when you add new fields or remove existing fields, such as when you:
- Add a Prism calculated field.
- Hide or expose a field.
- Add a stage that adds or deletes fields in that stage.
Schemas can also change in base datasets when you:
- Upload an external file.
- Import a file from an SFTP server.
- Import data from a custom report.
Where changes occur in a dataset can affect other dataset components. Changes to dataset schemas can break components that were based on the original schema. Where changes occur can also affect how you manage changes.
When you change the schema and Workday imports the new source file into the dataset, you must open the dataset and save the changes. If you don’t save the dataset, Workday continues to use the old schema definition. The next time you publish the dataset, you might get inconsistent data in the Prism data source.
When you import data that changes the schema of a base dataset from an SFTP server or custom report, you must manually edit the base dataset to incorporate the changes. When new fields get added to the dataset due to the source schema changing, Workday hides them automatically. Expose the new fields and save the base dataset to include the fields in the dataset schema.
Workday recommends that you ensure that the dataset is up to date and includes the fields you want. In most cases, you use the Manage Fields stage to manage dataset schema changes.
Manage Fields
The Manage Fields stage uses the output of the previous stage as a baseline from which to monitor changes. When this baseline changes, Workday warns you in the pipeline details panel. If the baseline changes, in the Manage Fields stage, Workday:
- Displays all new and removed fields.
- Doesn't yet include all changes in the dataset schema.
- Displays no data in the example table.
Workday recommends that you:
- Add a Manage Fields stage at the beginning of the Primary Pipeline of a derived dataset when you want to monitor the schema of the table or dataset from which it’s derived.
- Add a Manage Fields stage at the end of the Primary Pipeline of a dataset that you intend to publish. This enables you to detect any schema changes that might break reports that use the Prism data source of this published dataset.
- Add a Manage Fields stage at the end of a pipeline when you need to hide fields or expose new fields.
- Add a Manage Fields stage in a base dataset to ensure that no future integration can unintentionally remove an existing field.
- Include no more than 2 Manage Fields stages in a single pipeline.
In some cases, Workday handles schema changes without using a Manage Fields stage. Example: When uploading a new version of a source file into a base dataset, Workday handles schema changes based on the source file header row:
Source File Contains Header Row | Result |
|---|---|
Yes | Workday uses the field names in the header row of the base dataset to determine which fields are in the new source file.
You can add or delete fields anywhere in the source file. If the field names in the source file don’t change, Workday updates the schema of the base dataset into which you import the new source file. |
No | Workday handles changes that occur only at the end of the source file.
You can only add new fields at the end of the source file. If you do so, Workday updates the schema of the base dataset into which you're importing the new source file. |