[Research Briefing] Adaptive MLaaS Composition Framework for IoT Environments

Introduction

Modern enterprise IoT ecosystems collect real-time data from numerous sensors and edge devices, leveraging cloud-based MLaaS for processing. However, operational environments are subject to ‘dynamic variability,’ such as changes in user behavior or network fluctuations, which rapidly degrade the performance of initially designed ML model combinations.

From an enterprise perspective, reconfiguring all service combinations from scratch every time results in immense computational waste and service interruptions. Consequently, the capability for ‘adaptive composition’—precisely targeting and replacing only specific underperforming modules while maintaining overall system functionality—becomes a critical factor in determining the operational efficiency of corporate AI.

This research presents a mechanism for finding optimal replacement services by combining multi-dimensional QoS attributes, such as latency, reliability, and data distribution alignment, rather than relying on a single accuracy metric. This provides significant implications for enterprises employing multi-cloud strategies to prevent vendor lock-in and maintain peak performance.

Research Summary

This paper addresses the degradation of initially configured MLaaS (Machine Learning as a Service) combinations over time due to data distribution shifts (concept drift, data heterogeneity) and changing system requirements in IoT environments. Since traditional total recomposition is computationally expensive and time-consuming, the authors propose an adaptive composition framework based on incremental updates.

The research team generated MLaaS data using MNIST, FMNIST, and HAR (Human Activity Recognition) datasets and constructed a simulation environment by mining data from 10 major providers, including Amazon SageMaker and Google AI Platform. The proposed framework consists of five core components: a Service Assessment Model (SAM), a candidate service selection model, an adaptive composition model, and a Contextual Multi-Armed Bandit (CMAB) algorithm for optimization.

Experimental results show that the proposed adaptive model achieved over 95% accuracy across MNIST, FMNIST, and HAR datasets, significantly outperforming QoS-based (avg. 55%) and rule-based (78%) approaches. Furthermore, the CMAB algorithm demonstrated scalability by drastically reducing execution time compared to brute-force methods as the number of services increased.

The time complexity of the Service Assessment Model is O(n), making it more efficient than SHAP (O(n²)), with an average execution time of 0.9ns. However, the paper acknowledges limitations, specifically low efficiency in identifying trade-off services and a reliance on a 1:1 replacement strategy.

Deep Dive into Core Failure Factors

First is the risk of performance degradation caused by IoT data heterogeneity and concept drift. The paper explains that shifts in data distribution degrade the effectiveness of MLaaS services over the long term. Specifically, rule-based and QoS-based compositions using small models recorded a very low accuracy of 26.21% on FMNIST16, illustrating the vulnerability of static selection methods in dynamic environments.

Second is the exponential increase in computational cost during total recomposition. As the number of services grows, the number of possible combinations increases exponentially; the brute-force approach revealed scalability limits, with execution times spiking to 1.2ns. In IoT environments where real-time response is critical, this factor threatens overall system availability.

Third is the risk of inappropriate service selection due to the limitations of single-metric evaluations. Existing LOO or NCS methods focus exclusively on accuracy, overlooking latency and reliability. While this study proposes a model combining Functional Contribution (SCS) and QoS scores to resolve this, the paper admits that the current 1:1 replacement strategy may limit efficiency by lengthening the optimization cycle.

Source: Photo on Unsplash / Data Quality Archive

Service Assessment Model (SAM) and SCS Calculation Mechanism

The Service Assessment Model (SAM) is the core mechanism for identifying underperforming services by multi-dimensionally measuring the contribution of individual services within an MLaaS composition. Departing from single-metric evaluations, this model calculates a Service Contribution Score (SCS) that combines functional characteristics and non-functional quality (QoS). According to Equation (1), SCS is defined as the weighted sum of the QoS-based contribution score ($Qo{S}_{Mc}$) and the functional contribution score ($f(c)$), applying weights $\alpha$ and $\beta$ respectively. This design reflects both quantitative performance and structural alignment between models.

To calculate QoS-based contribution, Equation (3) computes the basic QoS score by averaging four factors: validity ($E_f$), quality ($Q$), response time ($t$), and reliability ($R$). Equation (4) then measures the change in overall QoS when a specific service $i$ is excluded, normalizing this against the total change across all services to quantify the relative impact of that service on maintaining system performance.

Functional contribution ($f(c)$) evaluates alignment between services using ML specifications such as model weights and gradients. The process in Equation (6) utilizes the principles of the FedTruth framework, starting by measuring the distance ($d(\cdot )$) between the weight update of an individual service and the update of the overall composition. A smaller distance value ($S(c{)}_t$), measured via Euclidean or angular distance, indicates stronger alignment with the overall learning direction. This is then normalized using a reciprocal function $g(S(c{)}_t)$ to derive the final functional contribution score.

The final SCS is determined by the weighted sum of the aforementioned QoS and functional contributions. A high SCS indicates the service is pivotal to performance maintenance and model alignment, while a low score suggests the service hinders efficiency or is misaligned with current data distributions and model directions.

Algorithm 1 (Service Assessment Model) automates this SCS calculation, where the filtering threshold $\theta$ plays a decisive role. After calculating SCS for all services, those scoring below $\theta$ ($\theta >10\%$) are classified as underperforming services ($M_u$) and extracted. This threshold defines the minimum acceptable contribution level, enabling the precise identification of modules requiring replacement without necessitating total recomposition.

The SAM mechanism overcomes the limitations of Leave-One-Out (LOO) and SHAP. While SHAP has high complexity at $O(n^2)$, the proposed SAM maintains $O(n)$ time complexity. Unlike LOO or NCS, which focus on simple accuracy, SAM provides a foundation for responding flexibly to the dynamic variability of IoT environments by integrating functional and non-functional attribute evaluations.

Analysis of 5 Core Validation Rules for Adaptive Composition

The core of adaptive MLaaS composition is the multi-dimensional validation of how well a candidate service aligns with the existing composition. To achieve this, five precise validation rules are applied. First is Data Utility Measurement (DUM), which uses Equation (7) to measure the difference in data distribution between the candidate and the existing composition via Euclidean Distance (ED). It returns 1 only when the distribution of samples per class label is below a set threshold ${\theta }_d$, selecting only services with high consistency and low distribution skew.

Second is Model Utility Measurement (MUM). Equation (8) measures the divergence between the candidate service’s weights $w$ and the average weights of the existing composition ${\bar{w}}_{cp}$. Analyzing parameter variance from a Collaborative Learning (CL) perspective, a divergence value that is too low identifies the service as insignificant and adds no value to the system, leading to its exclusion.

Third is Scalability Measurement (SM), a rule to ensure real-time system responsiveness. Equation (9) calculates the ratio between the existing response time $T$ and the candidate’s response time $t^{\prime }$, applying a penalty coefficient $\alpha$. If the combined response time exceeds threshold ${\theta }_T$, the service is classified as unsuitable due to its potential to increase overall system latency.

Fourth and fifth are Historical Quality Score (HQS) and Service Reliability Score (SRS), which verify stability and reliability. HQS measures similarity between historical quality vectors $q$ via Equation (10), assigning 1 if similarity exceeds ${\theta }_q$. SRS calculates the ratio of completed tasks with reliability scores above ${\theta }_r$ via Equation (11). This ensures the service is sustainable and stable, rather than providing a temporary performance spike.

The results of these five rules (DUM, MUM, SRU, SM, HQU) are integrated into a final Confidence Score (CS) via Algorithm 3 (Adaptive MLaaS Composition Model). By multiplying the binary results (0 or 1) by a weight coefficient ${\lambda }_s$ set by the IoT provider, a customized confidence score is generated based on the provider’s priorities.

The final CS value ranges from $[0,1]$, where a value closer to 1 signifies perfect alignment with the existing composition. This mechanism maximizes the efficiency of adaptive composition by predicting the suitability of a candidate service in real-time, avoiding the high cost of full recomposition.

CMAB-based Combination Optimization and Scalability Proof

Due to the dynamic nature of IoT environments, the functional and non-functional attributes of MLaaS services constantly change, turning the search for an optimal service combination into a complex combinatorial optimization problem. To resolve this, the study introduces a Contextual Multi-Armed Bandit (CMAB) framework. Rather than exploring all possible combinations, it selects the optimal service (Arm) that maximizes reward based on contextual features. Contextual features $C_{f,i}$ consist of data specifications ($D_i$) and QoS metrics ($Qo{s}_i$)—including efficiency, latency, reliability, and quality—providing the basis for dynamic response to environmental changes.

The core of CMAB is learning by utilizing the Confidence Score (CS) derived from the adaptive composition model as a reward. The Reward Function (Eq. 13) is calculated via the sum of CS for selected candidate services, and the expected reward is modeled as the dot product of the contextual feature vector and a coefficient vector ${\theta }^{\ast }$. The algorithm iteratively updates the decision matrix $A_i$ and vector $b_i$ to refine the correlation between contextual features and actual rewards, enabling rapid optimal selection based on past learning experience even when new services are introduced.

To determine the optimal combination, the algorithm calculates the UCB (Upper Confidence Bound) Score (Eq. 14) to balance exploration and exploitation. The UCB score is derived by adding a confidence interval representing uncertainty to the current estimated reward. Through the exploration-exploitation parameter $\alpha$, the system provides exploration opportunities for insufficiently verified candidates, ultimately selecting the top $K$ service combinations with the highest UCB scores to maximize cumulative reward.

The efficiency of the proposed CMAB approach is clearly demonstrated by the execution time data in Figure 3. In scenarios where the number of possible combinations increases exponentially with the number of services, the brute-force approach showed severe scalability limits, with computation time exploding to 1.2ns. Conversely, the CMAB algorithm maintained low execution times even as the service scale expanded by efficiently exploring the search space.

CMAB also proved superior to Epsilon-Greedy and Genetic Algorithms. The reduction in execution time became more pronounced in large-scale environments exceeding 2,000 services, suggesting this structure is highly suitable for IoT operational environments where optimal replacement services must be identified and applied in real-time from vast MLaaS marketplaces.

Consequently, this framework implements a scalable optimization mechanism that drastically reduces the computational cost of total recomposition while maintaining system QoS above a certain level through feedback-based learning. It is technically distinguished by mathematically modeling and solving the real-time variability of data and system constraints, moving beyond simple static rule-based selection.

Dataset Configuration and Performance Benchmark Results

To verify performance, a simulation environment was constructed using three representative datasets: MNIST, FMNIST, and HAR. To reflect actual MLaaS market characteristics, data from 10 major providers—including Amazon SageMaker, Google AI Platform, and Microsoft Azure Machine Learning—were mined to create a dataset containing functional attributes and non-functional QoS metrics. Experiments were designed with IID and non-IID settings, as well as concept drift and data drift scenarios, to simulate data distribution variability.

Analysis of Table I, the core of the performance benchmark, shows that the proposed adaptive MLaaS composition model held an overwhelming advantage over existing methods across all dataset and model size combinations. In terms of Accuracy, the proposed model maintained high performance ranging from a minimum of 91.22% to a maximum of 96.75%, regardless of the dataset. This is a significant gap compared to traditional QoS-based (avg. 55%) or rule-based (78%) compositions, proving its robustness in dynamic IoT environments.

Examining performance changes by model size (16, 32, 64), overall performance tended to improve as model capacity increased. However, the performance gap between methodologies was most stark in small-scale models (size 16). Rule-based composition recorded a low accuracy of 44.47% on MNIST16, and QoS-based composition showed a highly vulnerable figure of 26.21% on FMNIST16. In contrast, the proposed model minimized model size constraints, recording high performance of 96.75% on MNIST16 and 94.20% on FMNIST16.

The proposed model also demonstrated consistent superiority in precision and recall metrics. According to the figures in Table I, the proposed model secured over 90% precision and recall in all test cases. This indicates that it achieved high detection power and precise classification, rather than simply increasing the hit rate of predictions. For the HAR dataset, it achieved 95.41% accuracy with the size 64 model, showing it can find optimal combinations even for complex human activity recognition tasks.

Ultimately, this benchmark proves how easily existing methods relying on static rules or single QoS metrics collapse in small-model or high-data-variability environments. It is numerically verified that the proposed adaptive composition model effectively prevents performance degradation and maintains consistent high performance by integrally considering functional contribution and QoS attributes, even when model size is limited.

Adaptive MLaaS Composition and Optimization Checklist

To preemptively block such system collapses in practical engineering pipelines, the following core defense mechanisms must be enforced in the operational phase:

• Definition of Functional/Non-functional Attributes: Are both model specifications (weights, gradients) and QoS (accuracy, latency, reliability) metrics defined?
• Service Contribution Measurement: Is the SCS (Service Contribution Score), combining functional contribution and QoS, being calculated instead of using LOO or SHAP?
• Candidate Filtering Logic: Is there a stage that first filters only services with matching data modality and characteristics, rather than performing a simple search?
• Confidence Score Calculation: Is replacement reliability measured through multi-dimensional rules such as Data Utility (DUM), Model Utility (MUM), and Scalability (SM)?
• Optimization Algorithm Application: Is computational cost optimized using exploration-exploitation balance algorithms like CMAB (Contextual Multi-Armed Bandit) instead of brute force?
• Incremental Update System: Is there a mechanism to identify and partially replace only underperforming services rather than performing a total reconfiguration?

Author’s Perspective and Practical Recommendations

This research addresses the very practical perspectives of MLaaS ‘composition’ and ‘dynamic replacement.’ In particular, the focus on ‘Functional Alignment’—evaluating contribution by measuring distances between model weights—is a successful application of Federated Learning insights to MLaaS composition, a point often overlooked by engineers.

However, the use of simulation datasets (such as MNIST) leaves questions about whether identical performance would be achieved with complex domain data in actual industrial fields. Furthermore, moving beyond the simple ‘1:1 replacement’ strategy to the logic of ‘integrating and replacing multiple services with one high-performance service,’ as suggested for future research, would make this a far more powerful tool.

From a C-level perspective, this is highly valuable as it provides an architectural direction to automate performance optimization while reducing the risk of dependency on a specific cloud vendor’s ML API.

Conclusion and Enterprise Governance Guide

To ensure internal AI projects move beyond initial PoC demos and establish real business value, the following governance principles must be established from the planning stage:

1. Multi-dimensional Performance Evaluation Principle: Determine service retention based on a comprehensive score (SCS) including latency, reliability, and data alignment, rather than accuracy alone.
2. Incremental Optimization Principle: Avoid the cost of total system reconfiguration and prioritize adaptive strategies that identify and partially update only underperforming modules.
3. Data Alignment Verification Principle: When introducing replacement services, prioritize those with close model weight distances and existing data distribution alignment over those with simply higher QoS figures.
4. Exploration-Exploitation Balance Principle: Maintain an algorithmic system that continuously explores new candidates while exploiting current best practices to find the optimal service combination.

References and Original Analysis Data