Federated Learning (FL) has been termed a revolutionary manner of machine learning because it provides the capability of collaborative model training across devices in a decentralized manner while preserving data privacy. Instead of transferring data to a centralized server for training, devices train locally, and only their model updates are shared. This way, it finds applicability in sensitive areas like healthcare, finance, and mobile applications. As Federated Learning continues to evolve, an increasingly diverse array of algorithms has emerged each designed to enhance communication efficiency, boost model accuracy, and strengthen resilience against data heterogeneity and adversarial challenges. This article will delve into the types, examples, and top 10 Federated Learning Algorithms.
Types of federated learning algorithms:
Federated Learning algorithms get classified by how data is laid out, by the system structure, and by the privacy requirements. Horizontal FL covers clients with the same features but distinct data points. Vertical FL captures the case where features are different but clients overlap. When users and features are both different, we use Federated Transfer Learning. Decentralized FL, as opposed to Centralized FL, doesn’t use a central server and instead allows for peer-to-peer communication. In terms of FL deployment methods, Cross-Silo FL consists of powerful participants like hospitals and banks, while Cross-Device FL focuses on lightweight devices, such as smartphones. In addition, Privacy-Preserving FL protects user data with encryption, differential privacy, and other techniques, and Robust FL attempts to protect the system from malicious, adversarial, or broken clients.
Examples of federated learning algorithms:
Examples of Federated Learning Algorithms: A number of algorithms have been created to overcome challenges specific to Federated Learning problems. The basic approach of Federated Learning is FedAvg, which, in contrast, models client averaging. FedProx, which is designed to work well with data heterogeneity, is a more advanced approach. For personalization, FedPer customizes top layers for each client, and pFedMe applies meta-learning techniques. Communication-efficient algorithms like SCAFFOLD and FedPAQ reduce bandwidth usage and client drift. Robust algorithms such as Krum, Bulyan, and RFA filter out malicious or noisy updates to maintain model integrity. Privacy-focused methods like DP-FedAvg and Secure Aggregation ensure data confidentiality during training. These algorithms are often tailored or combined to suit specific domains like healthcare, finance, and IoT.
Top 10 Federated Learning Algorithms:
- Federated Averaging (FedAvg):
FedAvg stands as the founding algorithm for Federated Learning. The weight averaging is performed after models are trained locally on each client for updating the global model. Due to its simple design and the ease with which one can scale, it has been widely implemented in practice.
- FedProx
FedProx improves upon FedAvg by adding a proximal term to the loss function. FedProx builds upon FedAvg by introducing a proximal term in the loss function. By penalizing local updates that diverge too much from the global model, this term helps stabilize training in settings with widely differing client data distributions. It is especially helpful in fields like healthcare and finance, where heterogeneous data is prevalent.
- FedNova (Federated Normalized Averaging)
To address the drift of the client, FedNova normalizes updates with respect to the number of local steps and learning rates. This ensures each client has an equal contribution to the global model regardless of its computational capabilities or data volume. This further favors convergence and fairness in heterogeneous setups.
- SCAFFOLD
SCAFFOLD, an abbreviation for Stochastic Controlled Averaging for Federated Learning, employs control variates to make corrections to the client’s updates. This limits the variance that exists owing to non-IID data and speeds the convergence. It is particularly effective in an edge computing environment, where data come from various sources.
- MOON (Model-Contrastive Federated Learning)
MOON brings contrastive learning into FL by aligning local and global model representations. It enforces consistency of models that are particularly necessary when client data are highly divergent. MOON should often be used for image and text classification tasks for very heterogeneous user bases.
- FedDyn (Federated Dynamic Regularization)
FedDyn incorporates a dynamic regularization term in the loss function to enable the global model to accommodate client-specific updates better. Because of this, it can withstand situations involving extremely diverse data, such user-specific recommendation systems or personalized healthcare.
- FedOpt
FedOpt substitutes in place of the vanilla averaging mechanisms with advanced server-side optimizers like Adam, Yogi, and Adagrad. Using these optimizers leads to faster and more stable convergence, which is paramount in deep learning tasks with large neural networks.
- Per-FedAvg (Personalized Federated Averaging)
Personalized Federated Averaging hopes to balance global generalization with local adaption by allowing clients to fine-tune the global model locally. Because of this, Per-FedAvg is suitable for personalized recommendations, mobile apps, and wearable health monitors.
- FedMA (Federated Matched Averaging)
The distinguishing feature of this method is the matching of neurons across client models before averaging. This retains the architecture of a deep neural network and hence allows for much more meaningful aggregation, especially for convolutional and recurrent architectures.
- FedSGD (Federated Stochastic Gradient Descent)
A simpler alternative to FedAvg, FedSGD sends gradients instead of model weights. It’s more communication-intensive but can be useful when frequent updates are needed or when model sizes are small.
Conclusion:
These algorithms represent the cutting edge of federated learning, each tailored to address specific challenges like data heterogeneity, personalization, and communication efficiency. As FL continues to grow in importance especially in privacy-sensitive domains these innovations will be crucial in building robust, scalable, and ethical AI systems.
