FAQ: Model Adaptability and Maintenance

How often do we re-evaluate and potentially retrain or switch forecasting algorithms for existing customers? What triggers this re-evaluation (e.g., performance degradation, significant data shifts, new data availability)?

We regularly re-evaluate our forecasting algorithms for existing customers to ensure their continued accuracy and effectiveness. Our approach is divided into two primary activities: retraining and retuning, each with its own cadence and triggers.

Our machine learning (ML) models are retrained at every forecasting run to generate the most current forecasts using the latest data. Retraining every time we forecast ensures that our models incorporate the very latest information and capture any new patterns or trends as soon as they emerge, maintaining the accuracy of our predictions.

The hyperparameters (e.g., learning rate, batch size, etc.) of our models are retuned quarterly. This process is a deeper adjustment aimed at optimizing the model’s performance on a specific metric. By adjusting hyperparameters on a regular schedule, we can ensure the model is consistently aligned with our key performance indicators, such as minimizing error or maximizing accuracy.

If there is a significant, sustained change in underlying trends (e.g., a new product launch, a major market shift, or a global event like a pandemic), how long does it typically take for the ML model to react and adjust its forecasts to reflect these new trends?

The time it takes for a machine learning model to react to a significant, sustained change in trends is not a fixed number. It’s a dynamic process that depends on several factors, but we can outline the key considerations.

In a practical sense, it typically takes weeks to a few months for a forecasting model to fully react and adjust to a major market shift, but it can be faster or slower depending on the metric and its historical data. This is because the model’s ability to learn and adapt is tied to the availability of new data that reflects the change.

Some key factors influencing reaction time are data velocity and retraining frequency. Our retraining schedule is the most important factor in a model’s ability to react. The model begins to see the new data—and the emerging trend—the very next time the forecasting process is run. However, it takes time for this new data to constitute a significant portion of the training set. A major shift that is clearly visible in one week of data might only represent a tiny fraction of a model trained on several years of historical data.

Sudden, Drastic Shifts: Events like a pandemic-related lockdown are shock events
. While the model’s performance will degrade almost immediately (as our monitoring systems would detect), it will take time for the new data to be collected and for the model to learn the new, post-event patterns.

Gradual, Sustained Shifts: A new product launch or a slow market shift allows the model to adapt more incrementally. Our retraining process will help the model gradually learn the new trend as more and more recent data becomes available.

The Memory of the Model: The length of the historical data used in training is crucial. A model trained on five years of data will take longer to forget the old trends than a model trained on one year of data. In a scenario with a major shift, it may be necessary to consciously forget some of the oldest data or apply more aggressive weighting to the most recent data to accelerate the model’s adaptation.

The Role of Retuning: While retraining happens every time the forecasting process is rerun, it’s the quarterly retuning of our hyperparameters that allows for more fundamental adjustments. A significant, sustained change in trends may trigger an earlier-than-scheduled retuning. For example, if we see that a model’s performance has consistently degraded for several weeks, we would manually trigger a retuning to see if a different set of parameters could help the model better capture the new reality. This is our primary mechanism for “tuning” the model’s ability to adapt.

What mechanisms are in place to detect and address model "drift" (when the model's predictions become less accurate over time due to changes in the data distribution)?

We recognize that the real-world data our models encounter is constantly evolving. This can lead to model drift, where a model’s predictive accuracy degrades over time due to changes in the underlying data patterns. To effectively manage this, we employ a strategy based on internal visualization, data pattern analysis, and expert investigation.

Our Approach to Drift Management

Internal Visualization and Data Pattern Tools - Our technical teams use dedicated internal tools and visualizations to actively look for shifts in the data patterns that could indicate drift.

Pattern Visualization: We manually review charts and dashboards that show the statistical properties of the incoming data. This allows us to visually inspect for unexpected changes or anomalies in the features the model uses to make predictions.

Error Analysis: We specifically analyze where and why the model is making new types of errors. This hands-on review helps us diagnose if the model is encountering patterns it was never trained on.

Remediation (Model Retraining) - Once drift is identified, we immediately take steps to retrain the model.

We collect the latest, most representative data that caused the shift.

The model is then retrained on this new, updated dataset to learn the current real-world patterns.

This new, corrected version is then deployed to restore accuracy and performance.

We rely on expert manual analysis using powerful internal visualization tools to efficiently identify, diagnose, and correct model drift, ensuring we maintain high standards for accuracy.

How far out into the future (either in absolute time or relative intervals) can we forecast and to what accuracy?

Our forecasting capabilities are optimized to provide predictive insights within a specific, effective window, and the expected accuracy is highly dependent on the factors unique to your data.

We can reliably forecast up to a maximum of eight weeks (two months) into the future. This horizon ensures that the predictions are grounded in recent data and maintain a practical level of relevance and accuracy for your planning needs.

Our accuracy varies based on a number of key factors, but we generally achieve robust results across most projections:

General Accuracy: We have historically achieved an accuracy of 70% and higher for the majority of the metrics we forecast.

Controlling Factors: Accuracy is highly variable and depends on several critical elements:

Data Quality: The reliability and completeness of the historical data available for a specific metric.

Time Period: Shorter-term forecasts generally exhibit higher accuracy than those projected toward the 8-week limit.

Metric Complexity: Factors like seasonality, data sparsity, and the inherent stability of the metric being measured (e.g., Sales Orders, inventory, etc.) all impact the achievable accuracy.

We continuously monitor and refine our models to provide the most accurate forecasts possible within this defined 8-week window.

What is the industry standard for good accuracy? What is Workday’s standard?

The concept of a single "good accuracy" standard is not universal across the industry, as acceptable accuracy levels vary widely by the type of metric, industry, and demand volatility. Our standard is defined by our proven performance and is structured to deliver reliable projections for your business.

There is no single, fixed benchmark for forecast accuracy that applies to all industries. What is considered "good" depends on:

Metric and Volatility: A highly stable metric (like a long-term supply contract) can often achieve 90%+ accuracy, while highly volatile metrics (like new product sales influenced by seasonality or promotions) might consider 70% accuracy to be excellent.

Time Horizon: Accuracy naturally decreases the further out you forecast.

Time Range: Accuracy can fluctuate depending on the timeframe used for analysis, particularly in situations where extreme outliers are present.

Accuracy Metric Used: The formula used (e.g., MAPE, WMAPE) impacts the resulting percentage. Internally we calculate accuracy as:

Percentage Accuracy = 100 − ( ∑∣Predicted−Actual∣ / ∑Actual ×100)

In general business contexts, accuracy rates of 70% or higher are often considered a solid and reliable foundation for planning, especially when dealing with the inherent uncertainty of real-world demand.

Workday's internal standard is based on the robust performance our models have demonstrated across a diverse range of customer data:

Workday's Standard: We have been able to achieve an accuracy of 70% and higher for the majority of the metrics we forecast.

Variable Performance: We achieve this target even while managing the complexity of factors in your data, such as:

The data quality and consistency of the available historical information.

The degree of seasonality, sparsity, or inherent volatility in the specific metric being forecasted.

The specific time period and level of organizational detail (e.g., Sales Order vs. Product Family) being analyzed.

By consistently meeting or exceeding the 70% threshold across the majority of metrics, we ensure that the vast majority of your forecasts are sufficiently reliable for effective operational and financial decision-making.