Parameterized Nonlinearities in Feedforward Networks: An Adaptive Function Evaluation for Efficient Regression

• Jitendra Gupta

  Compunnel Inc.

  Author

Adaptive activation functions have emerged as a promising approach for improving neural network expressiveness without increasing architectural complexity. This paper investigates whether trainable nonlinearities can enhance regression performance in compact feedforward networks operating under limited computational budgets. We propose a modified Extreme Learning Machine (ELM) framework in which activation functions are parameterized and learned through two adaptation strategies: layer-wide parameter sharing and neuron-specific parameterization. Four activation functions, quadratic, PReLU, sigmoid, and swish, are evaluated in both their standard and adaptive forms across ten diverse regression benchmark datasets. Experimental results demonstrate that adaptive parameterization consistently improves predictive performance over fixed activation functions while introducing only a modest increase in computational cost. Neuron-specific adaptation achieves the lowest root mean squared error (RMSE) in the majority of datasets, whereas layer-wide adaptation offers a favorable trade-off between accuracy and parameter efficiency. Among the investigated nonlinearities, adaptive quadratic functions yield the largest improvements, with average error reductions exceeding 18% relative to their standard counterparts. Statistical analyses using one-way ANOVA and Tukey's post-hoc tests confirm that the observed performance gains are significant across most datasets. These findings indicate that functional plasticity through trainable activation parameters provides an effective alternative to increasing network depth or width, enabling more accurate and computationally efficient regression models for resource-constrained applications.

• PDF