AlphaDropout

https://pytorch.org/docs/stable/generated/torch.nn.AlphaDropout.html

alpha_dropout_demo.py
import svetoviz_webgpu as sv

# 1. Define Layer Parameters
# Typically used with SELU activations for self-normalizing networks (SNNs)
alpha_drop = nn.AlphaDropout(p=0.1)

def terminal_callback(buffer, message, images, files):
    alpha_drop.train()
    output = alpha_drop(input_tensor)

    # Verify SNN properties (Mean 0, Var 1)
    mean_out = output.mean().item()
    var_out = output.var().item()

    buffer.send_system_message(f"Post-Dropout Mean: {mean_out:.4f}")
    buffer.send_system_message(f"Post-Dropout Variance: {var_out:.4f}")

    # Instead of zero, it drops to the saturation value of SELU
    min_val = output.min().item()
    buffer.send_system_message(f"Saturation Value: {min_val:.4f}")

# 2. Start the interactive session
sv.pytorch_web(module=alpha_drop, terminal_callback=terminal_callback)
Self-Normalizing Dropout
AlphaDropout activations
AlphaDropout is a specialized dropout for Self-Normalizing Neural Networks. Instead of zeroing elements, it sets them to the negative saturation value of the SELU activation. It then applies an affine transformation to ensure the mean and variance of the activations remain 0 and 1.