來源：DeepHub IMBA
本文多圖，建議閱讀5分鐘
在本文中，我通過自己寫的一個(gè)程序來挖掘截至 2022 年 4 月 22 日時(shí)維基百科頁面歷史中的每個(gè)唯一激活函數(shù)。

Binary step

Logistic, sigmoid, or soft step

ElliotSig 或 Softsign

雙曲正切 (tanh)

Arctangent / Arctan / atan

Softplus

Rectified linear unit (ReLU) (ReLU6)

Exponential linear unit (ELU)

Gaussian Error Linear Unit (GELU)

Scaled exponential linear unit (SELU)

Mish

Leaky rectified linear unit (Leaky ReLU)

Parametric rectified linear unit (PReLU)

Parametric Exponential Linear Unit (PELU)

S-shaped rectified linear activation unit (SReLU)

Bipolar rectified linear unit (BReLU)

Randomized leaky rectified linear unit (RReLU)

Sigmoid linear unit (SiLU) or Swish

Gaussian

Growing Cosine Unit (GCU)

Shifted Quadratic Unit (SQU)

Non-Monotonic Cubic Unit (NCU)

Shifted Sinc Unit (SSU)

https://arxiv.org/pdf/2111.04020.pdf。

Decaying Sine Unit (DSU)

https://arxiv.org/pdf/2111.04020.pdf

Phish

https://www.techrxiv.org/ndownloader/files/33227273/2

SQ-RBF

Inverse square root unit (ISRU)

Square nonlinearity (SQNL)

Sigmoid shrinkage

“Squashing functions”

Maxout

Bent Identity

Sinusoid

Sinc (taming the waves)

ArSinH

Soft Clipping (goldilocks)

Piecewise Linear Unit (PLU)

Adaptive piecewise linear (APL)

Inverse Cubic

Soft Exponential

LeCun hyperbolic tangent (42?)

https://en.wikipedia.org/wiki/Activation_function