Abstract
 

With the rapid digitalization of education, large amounts of heterogeneous multimodal data (i.e., written responses, handwritten documents, and audio recordings) are being generated by students within education systems. The vast amount of multimodal student-generated data that is available for advanced educational analytics and intelligent learning systems creates critical privacy, security, and regulatory risks to students from centralised machine learning architectures, especially related to sensitive student information. Although Federated Learning provides a decentralised approach for collaborative training of models without requiring the sharing of raw data, existing federated frameworks are primarily unimodal and computationally intensive, neither of which supports multilingual and resource-constrained educational environments.

The purpose of this paper is to conduct a comprehensive literature review of the various federated optimisation techniques, privacy-preserving mechanisms, and multimodal federated learning methods with respect to their applicability to educational data ecosystems. Through this review, a unified reference architecture is proposed that includes modality-specific encoders and hybrid privacy mechanisms and communication-efficient optimisation strategies to support heterogeneous multimodal data, configurable privacy protection, and large-scale implementation in low-bandwidth environments. A formal problem formulation of privacy-preserving, multimodal federated learning is related to communication and regulatory constraints. Additionally, open research opportunities and future research directions on creating trustworthy, inclusive, policy-compliant AI systems for education were identified.

 

Keywords

Federated Learning, Multimodal Learning, Privacy-Preserving ML, Educational Analytics, Indic Languages

 

I. Introduction

Digital learning platforms are now ubiquitous within educational ecosystems, transforming them into data-rich environments. Student-generated multimodal data now includes textual data submitted by students, digitally scanned handwritten responses, and audio recordings of student responses to oral assessments. Multimodal data generates opportunities for automated grading, speech-based assessment, and individualised educational analytics [31], [32]. Centralised machine learning techniques require student-centred data to be aggregated into a central server, creating significant privacy, legal, and regulatory challenges for student data.

Federated Learning (FL) [1] is a decentralised approach for training models by aggregating model updates in lieu of raw data. FL has been developed to include algorithms to address heterogeneity through FedProx [2] and adaptive optimisation methods [35]; as well as large-scale deployment challenges [7] and communication-efficient methods [8]. Although utilising FL eliminates many instances of raw data exposure, no formal privacy guarantees currently exist. Differential privacy (DP) was introduced to deep learning paradigms in [10] and adapted for federated learning in [11] and [12], and measures for secure aggregation [12] and homomorphic encryption [13], [34] noted to enhance confidentiality measures exist but do so at the expense of significant computational and communication overhead.

Recent proposals for multimodal federated learning include using contrastive representation learning and modality-aware aggregation [16] to [19]. However, to date, multimodal federated learning frameworks are domain-specific and not sufficiently integrated to provide privacy, definition of multimodality, and to support multilingual educational constraints. Similarly, models for Indic languages (i.e., IndicBERT [24], MuRIL [25], IndicBART [26], IndicTrans2 [27]) continue to be utilised in primarily centralised frameworks.

Through this paper, authors present the following contributions:

• Review of literature for synthesis of techniques used in federated optimisation, privacy preservation, and multimodal learning.

• Formal problem formulation for privacy-preserving multimodal federated learning.

• Creation of a unified reference architecture specific to the Indic educational environment.

• Identification of open research opportunities for scalable and regulatory compliant implementations.

II. Background and Related Work 

A. Federated Learning Optimisation 

The federated optimised function has been defined as follows: 

 

  is the overall local loss for client 𝑘,   is the size of local dataset, and  

FedAvg [1] performs a number of local SGD steps, and then performs a weighted average amongst the local models. However, local model updates diverge from one another when working with non-IID data [4], which reduces overall convergence quality of the federation. 

FedProx [2] introduces the concept of proximal regularisers.

The use of modified ADAM and Yogi optimisers [35] can help provide further convergence stability by allowing the use of previous learned weights within the federated learning framework. 

​

B. Privacy-Preserving Federated Learning 

1) Differential Privacy 

DP-SGD [10] provides privacy protection to each client by utilizing gradient clipping and by adding Gaussian noise. Each client’s contribution to the FL is limited by the bounds set for each client within the DP framework [11]. 

2) Secure Aggregation 

Secure aggregation protocols [12] provide the capability for federated learning algorithms to limit the amount of information provided to the sever after the addition of all local models; i.e. 

The securely aggregated contribution to the global model is:

 

 

While the server does not see any contribution from individual clients. 

​

3) Homomorphic Encryption 

​

HE-based federated learning [13],[34] enables the server to aggregate the contributions of each client using cryptographic protocols; however, this increases the computational overhead of aggregation and may eventually lead to increasing times to convergence. 

C. Multimodal Federated Learning 

Multimodal machine learning uses multiple modalities to create their respective representations [20]. Methods of fusion include: (1) early fusion; (2) late fusion; or (3) alignment. 

Contrastive multimodal federated learning approaches [16] optimise their embedding based upon:
 
Thus enabling the use of multiple embedding layers. Multimodal federated machine learning frameworks such as FedMM [17] allow for different models to be aligned across different modalities, and there are some benchmark studies available [18] to further standardise evaluation of these frameworks. However, privacy-preserving multimodal federated machine learning frameworks are not widely available. 

D. Key Characteristics of Educational Multimodal Data 

The characteristics of educational multimodal data within digital education platforms present unique attributes as compared to traditional benchmark datasets used within machine learning research. Understanding these key characteristics is critical to formulating the design of effective privacy-preserving multimodal federated learning systems within digital education environments. This sub-section defines and explores the important structural, statistical, and operational characteristics of educational multimodal data.

​

1) Modality Diversity

​

Data are generated across a variety of means by educational ecosystems and consist of heterogeneous multimodal entities. Textual-based data may consist of short textual responses, written essays, and communications via a discussion board; extracted from an educational institution. Handwritten data are created using an electronically scanned answer sheet, pen-based device, or other similar devices electronically, whereas spoken data are obtained through oral assessments (e.g., oral examinations, assessments of proficiency in a spoken language).

The characteristics of each modality are statistically different from one another, with textual data being sequence data based on tokens; handwritten data being image data based on spatial orientation, handwriting; and spoken data being time-based waveform data (e.g., digital recordings of verbal speech). Consequently, there are distinct feature extraction and learning strategies based upon the various modalities, which increases the complexity of the model and requires the development of multimodal representation learning frameworks [20].

​

2) Modality Heterogeneity and Partial Availability

​

In contrast to curated multimodal datasets, in which different modalities are present uniformly, educational institutes do not necessarily have complete coverage of all modality types. For example; at one educational institution only, textual submissions are being processed while at another educational institution both handwritten and also hearsay documents have been processed.

Therefore; between institutions, the issuance of each modality is different than at the other institutions and therefore the global model must provide support for missing modalities during training was undertaken. To this end; training Federated Learning in such environments must enable the global model to provide a singular (i.e., global) model that provides for the training of the global model such that there is no degradation in regard to the ability for the model to converge [16],[17].

​

3) Statistical Non-IID Distributions

​

Educational data are routinely and inherently statistically non-Identically Distributed Across all educational institutions. There are statistical variances between Institutions based on; Curriculum, medium of instruction, geographic regions, and diversified demographics within students.

Due to this nature of the heterogeneous distributions of educational datasets, there will be client-specific data distributions that violate the IID assumption, that has been commonly used when training the centralized models. Earlier research indicated that there are substantial disruptive effects on the stability of federated optimization methods when using non-independent and identically distributed (non-IID) data. In a multilingual education environment, given that the languages vary from one another, this additional variation causes the negative impact to be greater still.

​

4) Diversity of Scripts and Languages:

​

Multilingualism adds to the complexity of education systems that use an "Indic" based language. Text data will have multiple scripts and languages including Hindi, Marathi, English and code-mixed data, among others. The tokenization process, vocabulary represented in the data and context-based embeddings of each of these languages varies considerably. Pre-trained models such as IndicBERT [24] and MuRIL [25] have been built specifically to address these issues. However, typically these models were built in a centralized manner.

In federated clients that utilise multiple scripts, this creates inconsistencies regarding the cross-lingual representation of the data. Therefore, careful design of the encoders and the methods for alignment of the representations must occur to be able to resolve these issues.

​

5) Variability of Privacy Sensitivity by Modality:

​

The privacy risk of each modality is different when collecting the data. When gathering speech recordings, the recording will contain biometric features that can be used to identify the individual, such as vocal characteristics. Samples of handwriting can also reveal traits that are specific to the identity of the individual. When collecting text responses, there is potential for the data to contain personally identifying information about the individual or contain contextual details that should remain confidential.

Thus, the privacy risk is modality-dependent and cannot be adequately or uniformly balanced for optimum privacy and utility by implementing the same privacy mechanisms for all modalities. Therefore, this motivated the need for employing configurable privacy mechanisms in federated learning systems [10], [11], [12].

​

6) Constraints of Infrastructure and Resources:

​

Education institutions located in rural or semi-urban areas typically operate under bandwidth limitations and weak computational resources. Devices employed for data collection may be either low-powered devices or computed under shared use. Communication-intensive federated protocols may introduce deployment difficulties in implementing those models. In this context, such restrictions will demand communication-efficient optimization and adaptive strategies for client participation.[7], [8]

​

7) Temporal and evolutionary dynamics:

​

New and existing educational data will experience temporal fluctuations due to; academic progress, changes to curriculum, as well as changes in the time of the year they are examined, and changes in student cohorts. These temporal changes can lead to concept drift and the emergence of dynamically changing distributions of data. As a result, educational federation systems must continuously adapt in order to account for the fact that data environments are not static.

The preceding characteristics described collectively create significant design and implementation challenges for centralized and federated machine learning systems, they highlight the need for; modality-specific encoding strategies, optimization approaches that are capable of accounting for non-IID data distributions, configurable and layered privacy protection mechanisms, communication-aware system design, and effective alignment of multilingual representations.

These challenges both support and motivate the unified reference architecture presented in Section IV that is specifically developed to deal with the multimodal heterogeneity, privacy sensitivity, and resource constraints of educational ecosystems.

​

E. Design of Synthetic Datasets for Controlled Analysis of Multimodal Educational Data Characteristics and Federated Learning Implications

In order to conduct a controlled analysis of the various multimodal characteristics of education, and the implications these multimodal characteristics have for federated learning, a synthetic dataset was created that allows for the simulation of distributed institutional settings. Each client represents an educational institution and contains various types of multimodal data with respect to textual, handwriting image, as well as speech data. The synthetic dataset allows for the systematic exploration of multimodal diversity in a controlled environment at the client-level.

Non-IID behaviour was introduced by assigning clients with probability distributions specific to the client's language, labels, and subject domains. As examples of how to model multilingual differences, the synthetic dataset includes English, Hindi, Marathi, and code-mixed representations of text for each of the above-mentioned modalities in order to correspond to realistic educational environments in the Indian context. In addition, non-IID distribution of labels were created in order to correspond to the differences in student performance between the various institutions.

In order to model the heterogeneous nature of the different modalities, missing modality scenarios were included in the synthetic dataset by imposing predetermined probabilities of omission for text, images and audio, respectively. Missing data from an institution occurs when the institution does not have the infrastructure to support the collection of that modality. Furthermore, several forms of noise were introduced into the synthetic dataset in order to simulate real-world imperfections of the data; for example, text data contained character-level perturbations, handwriting images contained visual distortions, and audio samples contained signal-level noise.

 

​

​​

​

​

​

​

​

​

​

Fig. 1 Sample distribution across federated clients

 

​

 

 

 

 

 

 

 

 

Fig. 2 Non IID label distribution

 

 

 

​

​

​

​

​

​

​

​

​

 

Fig. 3 Multilingual distribution

​

 

 

 

 

 

 

​

​

​

 

Fig. 4 Modality availability in the dataset

​

​

The generated data set has a structured client-based format, using associated metadata to provide information on modality presence, language used, topics being studied and other label information. The selected statistical properties of this data set from all clients are shown in Figures 1, 2, 3 and 4 in order to illustrate how these controlled variations in size of data sets are included. Figure 1 shows the distribution of the samples from each of the contributing clients, demonstrating controlled variations (real-world) in size of data sets from each of the participating clients. Figure 2 demonstrates that the distribution of labels of the data set is not identically distributed (non-IID) and that the proportions of classes of the labels are represented differently for each client in the data set due to some of the inherent heterogeneity of the learning environment. Figure 3 presents the multilingual aspect of the distributions of clients in the data set, showing that there are several different varieties of languages from the Indic language family, and also code-mixed languages in the data set. Figure 4 demonstrates that there are various data modalities in the data set, and that there is a modality imbalance with a partial representation of the other modalities in the institutions participating in the study. Taken together, these observations indicate the presence of multimodal diversity, label imbalance and linguistic diversity, which exemplify the main characteristics of educational data collected in the real world. This well-controlled data set will provide analytical support for the evaluation of the proposed work outlined in Section II.D, and provide a basis for conducting analyses of the multimodal, multilingual and non-IID characteristics described within this work, and also provide a baseline for the analysis of the proposed federated learning approach to this data set, under educational/real-world conditions. By validating these characteristics, the proposed work of identifying and analyzing the properties of multimodal educational data and the accompanying challenges associated with distributed learning environments can be supported. Based on the results of this analysis, the problem statement for multimodal federated learning with privacy and communication limitations will be documented in the next section.

III. Formal Problem Formulation

The proposed Privacy-Preserving Multimodal Federated Learning Framework is summarized mathematically in Fig. 5.

Under our model, each participating client (i.e., educational institution) maintains a local multimodal dataset made up of non-homogeneous data types (e.g., textual input response; scanned handwriting; audio recording). These individualized datasets are not shared within the client base; rather, the clients will collaborate and train by contributing individual client data for the purpose of improving an established globally identified performance metric or objective. In this manner, the clients will contribute to a single/model/object(s) through the individual clients customarily contributing to their client home site application. Therefore, the global objective is the weighted summation of the client-specific loss functions; hence, each individual client is contributing based on the relative size of their individual local dataset compared to the sum total of all clients’ datasets; thus, those clients with a greater number of data sources will contribute more to the global model than those with fewer data sources while also maintaining their respective data security as they collectively and collaboratively contribute to the global model.

The objective of this learning is to provide a single global model that coordinates through generalization across various and differently non-identically distributed data collections. However, the federated learning models used in this study, as compared to traditional centralized learning models, are subject to additional requirements that must be considered explicitly. First, each individual client will contribute to training all clients shared global model while maintaining their individual privacy assurances through the enforced use of differential privacy techniques to ensure that client data cannot be used by a third party due to the contribution method. Second, to provide all clients with safe use of each participating clients individual dataset only, the individual client contributions will be provided to the central server as an aggregated dataset of all participating clients’ aggregated datasets using secure aggregation protocols. Third, communication restrictions refer to the limitations imposed on the client base, such as the use of telephone and/or the network equipment system, to preclude resource usage, thus necessitating efficient methods for transmission and aggregation of the updates.

Collectively, these restrictions result in an overall system optimization view of the problem at hand. In this view; there are multiple constraints that must be optimized at once; for example, the requirements of achieving a high predictive performance by the model (i.e., training the model in such a manner that each client develops high model accuracy); the requirements for strong individual client privacy through differential private requirements; and the minimization of the communication costs associated with transmitting updates. Because this multi-objective description, with respect to education and the fact that many of the different K-12 education systems utilize multiple languages of non-homogeneous and non-identically distributed modalities to deliver K-12 education, are all present in an educational setting, the importance of multi-modal federated learning within an educational environment is to recognize that the problem of educational multi-modal federated learning is not a traditional optimization problem but is a constrained and system aware learning experience.

​

​

​

​

​

​

​

​

​​

​

​

 

Fig. 5 Problem Formulation

​

This formulation captures the trade-offs between accuracy, privacy, and communication efficiency. 

IV. Unified Reference Architecture

The formal problem statement found in Section III defines the overarching goals of privacy-preserving multimodal federated learning, which serve as the basis for the establishment of a federated platform with multiple levels of privacy-preserving measures, as well as regulatory constraints governing the transmission and storage of multimodal data, during the model training process. Figure 6 below illustrates the integration of multiple functional capabilities to support education for all four two-dimensional dimensions of multimodal representation learning capability, privacy-preserving methods, federated optimization, and communication-aware deployment methods. Unlike traditional federated platforms, which typically emphasize the optimization of the model training process, the emphasis of this new architecture will be the holistic integration of all systems components with the privacy component and multimodal component of the model will be included as part of the training process.

 

​

​

​

​

​

​

​

​

​

​

​

​

Fig. 6 Unified Reference Architecture

​

A. Architectural Overview

The proposed architecture is composed of four components that are capable of communicating with one another: 

• Local Client-Side Multimodal Processing 

• Secure Aggregation of Privacy-Preserved Data 

• Federated Optimization Core

• Communication/Deployment Layer

 The basis of operation for the system will involve multiple rounds of communication iterations among clients. During each round, selected clients will conduct locally based multimodal training using non-IID, heterogeneous datasets collected from their respective students' multimodal data, thereby producing and transmitting privacy-preserved updated model data back to the central server for aggregation. The central server will aggregate the individual updated models from all clients, create one aggregated global model, and then transmit the aggregated global model back to each of the clients whose students were involved in the local training process during the previous round. This iterative interaction between local clients and the central server will continue until the aggregated global model reaches convergence or fails to meet the pre-determined minimum performance criteria.

B. Local Client-Side Multimodal Processing

As part of this system, the educational institution will be regarded as the federated client and will possess access to either a heterogeneous or a non-IID dataset(s) based on the individual institution's curricular differences as well as the individual institutions' language of instruction difference as well as their respective methods of assessing student achievement. The modality used in gathering data may differ across institutions and therefore some clients' students will not be able to participate in the creation of speech or handwriting datasets.

1) Local Pre-processing 

All pre-processing will take place within the respective educational institutions in order to maintain the confidentiality of the students' data. The various operations associated with this include: 

• Text normalization and tokenizing of Indic Languages 

• Denoise and Segment images from handwritten documents 

• Extraction of features from speech signals 

• Localized preprocessing will maintain that all raw form data does not leave the institution's boundaries

2) Modality-Specific Encoders

To handle the heterogeneous nature of the given data, the architecture utilizes specific encoders for each modality. A transformer-based Language Model (LM) could be used for text input, while Convolutional and Vision-based architectures could be used for handwritten input images. Self-supervised speech encoders can be used as the encoder for audio-based data. Each encoder generates a dense representation which captures modality-specific semantic content about the information. Hence, if any given modality is not present for a specific task, the flexible usage of Encoded Dense Representations allows the system to be utilized by all modalities.

3) Multimodal Representation Fusion

Once each representation has been generated, there will be several data fusion techniques that can occur for the encoded representations such as data concatenation, attentional or contrastive alignment, which encourage consistency among different modalities. Local data fusion will occur, leading to the learning of all cross-modal correlations even though sensitive raw data will never leave the institution.

C. Privacy-Preserving Secure Aggregation

Federated learning provides a mechanism for indirect data-sharing among entities, yet the act of sharing model updates may expose private and/or sensitive client information. Therefore, a privacy-preserving communication mechanism will be established between the Client and Server within the Application Service Provider (ASP) environment.

1) Differential Privacy Integration

Before clients transmit local model updates (e.g., gradients), they will apply gradient clipping and controlled noise injection on each local update to bound the influence of each individual training sample, providing bounded leakage of privacy at the client and enabling configurable privacy budgets dependent upon institutional policies.

2) Secure Aggregation Protocol

The Secure Aggregation Protocol ensures that client model updates are sent to the central server as either encrypted or obscured. This allows the server to compute the aggregated (combined) model update; the server does not have access to the individual client updates. This approach ensures that raw data remains on the client-side, that individual model updates remain confidential, and that the only model parameters visible to the server are the aggregated parameters.

 

3) Optional Enhanced Encryption

In cases of educational deployments where the data requires a high level of protection, a homomorphic encryption mechanism may be used to perform the aggregation in an encrypted environment. While this option is much more computationally intensive, it provides a greater level of protection in highly regulated environments.

D. Federated Optimization Workflow

The Federated Optimization Process is communicated in a structured workflow:

1. The server initializes and broadcasts the global model.

2. A randomly selected subset of available clients is identified.

3. Each selected client performs local multimodal training (Number) times over several epochs.

4. Privacy mechanisms are applied to the local updates.

5. The server uses Secure Aggregation to combine the local updates.

6. The server redistributes the aggregated global model.

This workflow is designed to allow for partial participation; therefore, an institution with intermittent connectivity may contribute to the optimization process as it becomes available. Proximal regularization or adaptive optimization methods may be employed at the server to stabilize convergence in a non-IID context.

The fundamental logic that allows for optimization is completely independent of the privacy mechanisms in place; this is important, as it allows for modular adaptation, without requiring the redesign of the entire architecture.

E. Communication and Deployment Layer

Many educational institutions are limited in terms of bandwidth and computational resources. For this reason, the architecture includes communication-aware approaches (i.e., compressing/quantizing model updates; selecting clients based on connectivity; allowing asynchronous participation; using bandwidth-aware aggregation strategies). These approaches will help to reduce communication overhead while providing acceptable performance. Furthermore, the modular design will allow institutions with limited infrastructure to be able to contribute effectively without negatively impacting the overall stability of the global training process.

F. Evaluation and Monitoring

Beyond measuring the accuracy of the task completed by a client, the architecture provides for evaluating multiple dimensions of performance. Therefore, other predictive metrics can be used to measure performance, as well as monitoring privacy budgets, analyzing the costs of communication, and evaluating fairness across linguistic and institutional types. The combination of the aforementioned evaluation methods ensures that accuracy improvements do not come at an expense to privacy or inclusiveness.

The Proposed Unified Architecture does not introduce a new federated optimization algorithm; instead, it integrates existing techniques into a coherent, modular system designed to support multilingual educational ecosystems. Additionally, this architecture integrates multimodal representation learning, hybrid privacy mechanisms, and communication-efficient deployment strategies in a unified workflow, thus addressing the key fragmentation issues identified in the literature that currently exists.

This design serves as a reference architecture for validating and deploying new educational technologies in real-world settings.

V. Comparative Analysis and Research Gaps

The previous sections examined (1) federated optimization techniques; (2) privacy preserving methods; and (3) multimodal federated learning; however, these research areas have mostly developed separately from one another. This section presents a structured review and comparison of each three areas using a framework with three dimensions: (1) robustness of optimization techniques, (2) the level of privacy protections, and (3) the capability of multimodal federated learning; and identifies gaps that lead us to propose a unified reference architecture.

A. Federated Optimization Under Heterogeneity

The original federated learning studies focused primarily on how to perform distributed optimization in such a way as to maximize communication efficiency. The original research that led to FedAvg [1] provided a model for performing decentralized training through repeated aggregation of model data from clients. However, FedAvg was also dependent upon the absence of heterogeneity in the data being used in training, and created an environment that made it unlikely that convergence would occur when the data was not identically distributed (i.e., the data was NON-IID). [4] provided information supporting this claim.

Recent work on algorithms that improve the ability of algorithms to converge has attempted to alleviate the limitations introduced by FedAvg's dependence on having homogeneous data, by (1) including a proximal regularization term that decreases the amount of drift that will occur by limiting the degree of change between each individual client's weight, and (2) by using momentum in conjunction with an adaptive learning rate (e.g., ADAM) during the optimization phase. ([2]) described the former, while ([35]) described the latter.

Additionally, there has been work describing how systems would perform (i.e., system characteristics) under conditions relevant [7] for large scale deployment, as well as techniques for filtering out unnecessary communication when sending and receiving updates from clients [8].

Overall, with the exception of studies that evaluated the ability of their respective systems to be used in education settings where clients are heterogeneous relative to one another due to: (1) variations from statistical perspective (i.e., non-IID) and/or (2) differences in modality (i.e., images vs. text) and the different languages associated with the respective modality types, most of the work done to date in federated optimization has focussed primarily on evaluating the reliability of their methods to achieve convergence in the fastest time possible and have not considered any feature alignment related to mutlti-modal features. Table I provides an overview of federated learning methods related to their non-IID handling, adaptability, communication efficiency, and system capabilities.

​

Table I Comparison of Federated Learning Methods Under Heterogeneity

​

​

​

​

​

​

​

​

​

​

 

According to Table I, many recent approaches do improve some aspect of robustness under non-IID conditions, yet they do not have integrated multi-modal data processing and privacy-preserving capabilities. This indicates a requirement for a more comprehensive design that takes into account optimization, heterogeneity of modality, and privacy constraints simultaneously.

B. Privacy–Utility Trade-offs in Federated Learning

While federated learning can lessen the need for centrally stored data, model updates themselves may expose sensitive information. Differential privacy (DP) was introduced for deep learning [10] and then defined for federated environments [11] to formally guarantee bounds on the influence of each individual’s contribution to the dataset. Secure aggregation protocols [12] can be used to ensure that only the aggregated updates will be sent to the server; homomorphic encryption-based methods [13] and [34] allow for computation on encrypted data as well.

Each of these techniques has its own inherent trade-off with respect to the privacy of data. As a noise injection method of accomplishing differential privacy by adding randomness will potentially lower the accuracy of a model, especially when data is minimal or highly heterogeneous, secure aggregation adds to the communication costs of using aggregated data due to additional protocol steps, and homomorphic encryption imposes considerable computational and latency costs to perform computations on otherwise encrypted data. Hybrid approaches [15] have been introduced to attempt to balance these trade-offs; however, they generally focus on generic machine learning tasks as opposed to multi-modal educational analytics.

Finally, typical privacy-preserving techniques use uniform levels of protection for all data modalities. This is not effective in an educational setting since different modalities may produce various levels of sensitivity. For example, a speech data could produce a biometric identification, and a handwritten sample could produce an identification that is unique to that individual.

Table II provides a comparison of the main privacy-preserving techniques applied to federated learning with respect to their trade-offs regarding computational overhead, communication cost, and compatibility with multi-modal data.

​

​

Table II Comparison of Privacy-Preserving Techniques in Federated Learning

​

​

​

​

​

​

​

​

​

​

​

 

As evident from Table II, no existing approach simultaneously achieves strong privacy guarantees, multimodal compatibility, and communication-efficient deployment. This limitation underscores the importance of configurable and modality-aware privacy mechanisms.

​

C. Multimodal Federated Learning

​

Multimodal federated learning is a new area of research that addresses how to combine several different types of data (i.e., heterogeneous data) in a distributed manner when addressing a problem using machine learning techniques. New techniques based on contrastive representational techniques [16] have been developed that allow multiple modalities to align representation (embeddings) of one modality with that of another (multimodal). As examples, FedMM is a type of modally heterogenous multimodal federated learning model that explicitly addresses modality (the type of data) heterogeneity across multiple devices (e.g., mobile, tablet, smart speakers, etc.) in the design of the model [17]. In addition to these developments, several benchmarks [18] related to multimodal federated learning and/or multimodal federated learning benchmarking have developed disentangled training methods [19] that can enhance the robustness of multimodal federated systems.

While significant progress has been made in the area of multimodal federated learning, numerous challenges exist for multimodal federated learning systems as they exist today. First, as noted, most multimodal federated learning systems do not utilize formal privacy-preserving protocols (e.g., differential privacy, encryption) at any point in the multimodal training pipeline. Second, most studies that evaluate multimodal federated learning systems typically have a domain-oriented (e.g., medical imaging, sensor data) paradigm, and very few studies presently exist looking at education-oriented data. Third, while there are presently many multilingual models available (e.g., Indic language models such as IndicBERT [24], MuRIL[25] and larger-scale corpora such as Samanantar [28]), there are also no available multilingual federated learning systems at present.

In conclusion, while substantive advances have been made in multimodal federated learning, most of this work addresses the challenge of modality heterogeneity and does not include strategies for privacy preservation or multilingual environments, particularly in educational contexts.

​

As illustrated in the review, current federated learning methods more or less focus on developing techniques to allow multimodal learning to occur separately from one another, without providing privacy guarantees that accompany multimodal federated learning systems, and without addressing the issues of multilingual data representation. These are important, particularly with multimodal and multilingual data in educational contexts.

D. Deployment Constraints in Educational Environments

The implementation of real-world federated learning systems often occurs under added constraints that have received no attention in the current literature. In this analysis, intermittent network connectivity, limited computational resources, institutional policy restrictions, and different linguistic ecosystems are all discussed as barriers to effective federated systems. Several platform-oriented solutions [23] and educational analytics studies [31], [32] acknowledge these barriers, but they do not give a coherent integrated multimodal federated framework. Similarly, although applications of federated learning are being developed for medical applications [33] providing an example of cross-institutional collaboration without sharing data, none of those applications address the unique challenges posed by multimodal and multilingual educational data.

These challenges associated with deploying systems prove the need for communication-aware, resource-efficient system designs, especially in low-bandwidth and heterogeneous institutional environments.

​

E. Identified Research Gaps

Upon review of the foregoing comparative analysis, a number of critical research gaps were identified:

Fragmentation Across Research Dimensions:

Federated optimization, privacy preservation and multimodal learning continue to be researched in isolation from one another; no unified integration exists at this time.

Absence of Modality-Aware Privacy Designs:

Current privacy mechanisms provide no distinction between different sensitivity levels by modality.

Limited Attention to Multilingual Educational Data:

Guides to Indic language resources [24]–[30] continue to be available principally through centralized and poorly utilized distribution within federated settings.

Inadequate Communication-Aware Multimodal Integration:

As considering communication associated with multimodal models increases, the concurrent optimization of accuracy, privacy and communication cost has infrequently been addressed.

Absence of Unified Deployment Framework:

Currently, no documented framework brought together an integrated approach to multimodal encoders, hybrid privacy mechanisms, adaptive optimization, and communication-efficient strategies into one unified architecture.

F. Positioning of the Proposed Architecture

The unified reference architecture that was covered in Section IV attempts to address these identified gaps by synthesizing insights from federated optimization [1], [2], [35], privacy-preserving strategies [10]–[15], [34], and multimodal learning strategies [16]–[19] into an integrated and modular framework designed specifically for multilingual educational settings.

Rather than introducing yet another optimization algorithm, the proposed architecture will include established techniques that have been previously identified in the literature within a structured system design framework. This framework is specifically designed to allow for multimodal heterogeneity, configurable privacy guarantees, and communication-efficient deployment. This integrated approach is what sets this proposed framework apart from the majority of existing work; therefore, the previous works have primarily attempted to address one of those components independently.

VI. Open Challenges and Future Research Directions

There remain numerous open challenges in the areas of federated optimization and multimodal representations, as well as for privacy preserved learning, all of which need to be addressed with regard to the current state of educational data ecosystems.

First, very little is yet known about privacy design being aware of the modalities involved in educational data usage. The different types of educational data include modality types that have varying levels of privacy sensitivity, such as speech data potentially being used as a biometric identifier; handwriting can provide evidence of identifying characteristics; and, text data might have contextualised personal identifying information.  The vast majority of the differential privacy methods currently in use have used a uniform distribution for calibrating noise, which generally does not provide the correct balance of privacy vs. utility across different modalities.  Future efforts should focus on adaptive privacy budgets which take into account the different levels of privacy risk associated with each educational data modality.

Second, educational institutions have vast differences with regard to their curriculum, language of instruction and demographics, which results in educational data for each institution being extremely non-IID, multilingual and heterogeneous.  Therefore, developing optimisation processes that are able to provide robust performance when confronted by educational data systems containing numerous diverse language modalities is still an open and unresolved research problem, despite having seen some improvement in convergence stability from the use of various methods, such as FedProx [2] and adaptive federated optimisation [35]. Also, it is important to develop fairness-aware aggregation techniques to ensure that a client that is part of a majority language group does not dominate the learning process at the expense of other minority language groups.

Thirdly, missing modality robustness is a critical requirement regarding the deployment of educational data systems in the real world.  Educational institutions may not have consistently collected data on text, handwriting, or speech, thus requiring multimodal federated learning systems to be able to appropriately deal with situations where one or more modalities are absent from their respective datasets and not degrade the performance of the global federated model.  The development of scalable solutions which integrate both modality alignment with privacy and security safeguards still requires further research and development.

Fourth, because multimodal learning systems are expected to become large-scale systems, improving their communication efficiency becomes increasingly important.  There are several existing compression methods [8] and scalable system designs [7]; however, there has not yet been a thorough investigation into how to simultaneously optimise for all three factors, to the extent possible, i.e., accuracy, privacy, and communication cost, particularly in resource-constrained educational institutions.

Fifth, the conceptual and practical nature of the issues surrounding explainability, fairness, and compliance with data protection laws and regulations represent broader socio-technical issues to be resolved.  Educational AI systems need to ensure that they are transparent to users, fair across users who speak different languages or who are in different educational institutions, and in compliance with data protection laws, through the integration of privacy and security auditing and compliance mechanisms incorporated into federated, multimodal systems is an emerging area of research for future development.

Addressing the above challenges is necessary for transforming privacy-preserving multimodal federated learning from theoretical frameworks to implementable, scalable, inclusive and compliant systems in educational settings.

VII. Conclusion

In this article a comprehensive review of the literature and unified reference framework is presented for privacy preserving multimodal federated learning in multilingual educational environment, particularly for the Indic languages ecosystem. The review includes a systematic literature review of peer-reviewed research articles on federated optimization, privacy preserving mechanism, multimodal learning, and Indic language resources. A review of the above much peer-reviewed research revealed significant fragmentation among these research categories, with limited integration of optimization efficacy, hybrid privacy preservation and multimodal representation learning within an integrated system suitable for use in education.

To address the above gap a unified framework for privacy aware multimodal federated learning is proposed through the application of modular client-side analytics, secure aggregation secure adaptive optimization, and efficient deployment communication methods to support scalable collaborative processes between dissimilar organizations while maintaining data localities. This framework does not create any new optimization algorithms, but is built on existing techniques and provides a blueprint for future work.

In addition, the article discusses open areas of research that will need to be addressed in order to develop trustworthy Artificial Intelligence in Education with respect to privacy; include fairness across linguistic groups, modality-aware privacy budgeting, explainable federated multimodal models, and scalable deployment of Federated Learning systems that meet resource constraints.

In summary this article provides a conceptual and architectural framework to begin the experimentation and implementation of privacy preserving multimodal federated learning systems, to assist in the continued development of inclusive, compliant, and scalable Artificial Intelligence powered Educational Analytics across diverse and multilingual environments.

 

Funding Information: No external funding was received for this research.

Data Availability Statement: The dataset used in this study is synthetically generated to simulate multimodal and multilingual educational data under federated learning settings. The dataset and generation scripts can be made available upon reasonable request.

Conflict of Interest: The authors declare that there are no conflicts of interest regarding the publication of this paper.

Other Ethics Statements: This study does not involve real human participant data. The dataset used is entirely synthetic and generated for research purposes; therefore, ethical approval and participant consent were not required.

Ethical approval: The research was conducted using simulated data to model real-world educational scenarios. No personally identifiable or sensitive human data were used.

Author’s Contribution Statement:

Bhakti Patil: Conceptualization, dataset simulation, methodology design, investigation, analysis and interpretation of results, writing original draft, and manuscript editing.

Avinash Golande: Supervision, conceptual guidance, validation of research direction, and investigation of challenges.

Acknowledgment

The authors would like to acknowledge the academic environment and institutional support that facilitated this research work.

​

References

​

[1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2017

[2] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proc. MLSys Conf., 2020.

[3] P. Kairouz et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021.

[4] Tzu-Ming Harry Hsu et.al., “Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification,” arXiv:1909.06335 [cs.LG], Sep. 2019.

[5] H. Wang et al., “Federated learning with matched averaging,” in Proc. Int. Conf. Learning Representations (ICLR), 2020.

[6] P. Liang, T. Liu et.al. “Think locally, act globally: Federated learning with local and global representations,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020.

[7] K. Bonawitz et al., “Towards federated learning at scale: System design,” in Proc. MLSys Conf., 2019.

[8] Daniel Becking,Heiner Kirchhoffer, “Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

[9] Tianfei Zhou & Ender Konukoglu, “FEDFA: Federated Feature Augmentation,” in Proc. ICLR 2023.

[10] M. Abadi et al., “Deep learning with differential privacy,” in Proc. ACM Conf. Computer and Communications Security (CCS), 2016.

[11] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client-level perspective,” in Proc. NeurIPS, 2017.

[12] K. Bonawitz et al., “Practical secure aggregation for privacy-preserving machine learning,” in Proc. ACM CCS, 2017.

[13] Yuxuan Cai, Wenxiu Ding.et.al., “SecFed: A Secure and Efficient Federated Learning Based on Multi-Key Homomorphic Encryption,” IEEE Transactions on Dependable and Secure Computing, VOL.21, NO.4, July/August 2024 

[14] Timothy Stevens, et.al., “Efficient Differentially Private Secure Aggregation for Federated Learning via Hardness of Learning with Errors,” in Proceedings of the 31st USENIX Security Symposium, 2022.

[15] S. Truex et al., “A hybrid approach to privacy-preserving federated learning,” in Proc. ACM AISec, 2019.

[16] Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, Jingjing Liu, “Multimodal Federated Learning Via Contrastive Representation Ensemble,” in Proc. ICLR 2023

[17] Yuanzhe Peng, Jieming Bian, Jie Xu, “Fedmm:Federatedmulti-Modal Learning with Modality Heterogeneity in Computational Pathology,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IEEE ICASSP, 2024

[18] Tiantian Feng, et.al., “FedMultimodal: A Benchmark for Multimodal Federated Learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (August 2023) Proc.

[19] Xiaomin Ouyang, et.al., “Harmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training,” in Proc. ACM MobiSys, 2023.

[20] T. Baltrusaitis, C. Ahuja, and L. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019.

[21] Tuan Nguyen et al., “Federated Learning for ASR based on Wav2Vec2.0,” in Proc. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

[22] A. Baevski, M. Auli, et.al., “Unsupervised speech recognition,” in Proc. Neural Information Processing Systems (NeurIPS 2021)

[23] Farkhod Yusubov and Kang Yoon Lee, “A Platform of Federated Learning Management for Enhanced Mobile Collaboration,” Electronics 2024, 13, 4104. https://doi.org/10.3390/electronics13204104

[24] A. Kakwani et al., “IndicNLPSuite: Monolingual corpora, IndicBERT, and benchmarks for Indic languages,” in Proc. EMNLP, 2020.

[25] A. Khanuja et al., “MuRIL: Multilingual representations for Indian languages,” in Proc. EMNLP, 2021.

[26] R. Dabre, A. Kunchukuttan, et.al., “IndicBART: A Pre-trained Model for Indic Natural Language Generation,” in Proc. ACL, 2022.

[27] Jay Gala et al., “IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages,” in Transactions on Machine Learning Research, 2023.

[28] Gowtham Ramesh et al., “Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages,” in Transactions of the Association for Computational Linguistics, vol. 10, pp. 145–162, 2022.

[29] Mozilla Foundation, “Common Voice corpus: Indic languages,” 2021. [Online]. Available: https://commonvoice.mozilla.org

[30] Parth Patil, et.al., “L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT models,” Proceedings of the WILDRE-6 Workshop, 2022.

[31] Christian Fachola, et.al., “Federated Learning for Data Analytics in Education,” in Proc. Data 2023, 8, 43. https://doi.org/10.3390/data8020043.

[32] M. van Haastrecht et al., “Federated Learning Analytics: Investigating the Privacy-Performance Trade-Off in Machine Learning for Educational Analytics,” in Proc. Artificial Intelligence in Education (pp.62-74) , 2024

[33] M. Sheller et al., “federated learning in medicine: facilitating multi‑institutional collaborations without sharing patient data,” Scientific Reports, vol. 8, no. 1, pp. 1–12, 2020.

[34] Weizhao Jin, et.al., “FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System,” in Proc. Neural Information Processing Systems (NeurIPS 2023). 

[35] Sashank J. Reddi ,et.al., “Adaptive Federated Optimization,” in Proc. ICLR 2021

​

​

​

Web of Science Impact Factor = 0.3

​

Scopus Quartile = 3

​

​

​

​

​

​

​

​

​

​

​

​