Is there a connection between the modulation of Laplace’s Tidal Equation (LTE) solutions and the highly nonlinear fits of neural networks?
“Neural tensor networks have been widely used in a large number of natural language processing tasks such as conversational sentiment analysis, named entity recognition and knowledge base completion. However, the mathematical explanation of neural tensor networks remains a challenging problem, due to the bilinear term. According to Taylor’s theorem, a kth order differentiable function can be approximated by a kth order Taylor polynomial around a given point. Therefore, we provide a mathematical explanation of neural tensor networks and also reveal the inner link between them and feedforward neural networks from the perspective of Taylor’s theorem. In addition, we unify two forms of neural tensor networks into a single framework and present factorization methods to make the neural tensor networks parameter-efficient. Experimental results bring some valuable insights into neural tensor networks.”
https://w.sentic.net/new-perspective-for-neural-tensor-networks.pdf
The connection is via Taylor’s series expansion whereby neural nets try to resolve tight inflection points that occur naturally in the numerical flow of potentially turbulent fluid dynamics.
- Li, Wei, Luyao Zhu, and Erik Cambria. “Taylor’s theorem: A new perspective for neural tensor networks.” Knowledge-Based Systems 228 (2021): 107258.
- Zhao, H., Chen, Y., Sun, D., Hu, Y., Liang, K., Mao, Y., … & Shao, H. “TaylorNet: A Taylor-Driven Generic Neural Architecture”. submitted to ICLR 2023
Also wavelets can maneuver tight inflections.
“In this study, the applicability of physics informed neural networks using wavelets as an activation function is discussed to solve non-linear differential equations. One of the prominent equations arising in fluid dynamics namely Blasius viscous flow problem is solved. A linear coupled differential equation, a non-linear coupled differential equation, and partial differential equations are also solved in order to demonstrate the method’s versatility. As the neural network’s optimum design is important and is problem-specific, the influence of some of the key factors on the model’s accuracy is also investigated. To confirm the approach’s efficacy, the outcomes of the suggested method were compared with those of the existing approaches. The suggested method was observed to be both efficient and accurate.”
https://www.nature.com/articles/s41598-023-29806-3
- Uddin, Z., Ganga, S., Asthana, R. et al. Wavelets based physics informed neural networks to solve non-linear differential equations. Sci Rep 13, 2882 (2023). https://doi.org/10.1038/s41598-023-29806-3
Given the fact that NN results are so difficult to reverse engineer to match a physical understanding could this be a missing link?
This is an example using the Flux NN library using the programming language Julia:
using Flux, Plots, Distributions;
## Window of sinusoid
xTrain = collect(-3:0.01:3)
fxn(x) = cos(12.0*x)
hidden = 20
model = Chain(Dense(size(x, 2), hidden, tanh), #sigmoid
Dense(hidden, hidden, tanh),
Dense(hidden, hidden, tanh),
Dense(hidden, hidden, tanh),
Dense(hidden, hidden, tanh),
Dense(hidden, 1))
loss(x, y) = Flux.Losses.mse(model(x), y)
optimiser = Descent(0.1);
## Irregular spaced training window
Num = 500
x = rand(Normal(), Num)
y = fxn.(x)
## Trained multiple times, converting to rows
train_data = Iterators.repeated((Array(x'), Array(y')), 50000);
for Epochs in 1:1
Flux.train!(loss, Flux.params(model), train_data, optimiser)
end
yOut = zeros(length(xTrain))
for (i, x) in enumerate(xTrain)
yOut[i] = model([xTrain[i]])[1]
end
plot(xTrain, yOut, label="Predicted")
plot!(xTrain, fxn.(xTrain), label="True")
plot!(x, y, seriestype=:scatter, label="Data")
To be able to map to multiple periods of a sinusoid, the NN needs to be deep with several hidden levers, somewhat equivalent to the order of an extended Taylor’s series.
fxn(x) = cos(12.0*x) + cos(32.0*x) + cos(20.0*x)
Add higher frequency factors, will need more training or make it deeper — lots of tuning available with Flux:
GeoEnergy Math 
Typo above: “To be able to map to multiple periods of a sinusoid, the NN needs to be deep with several hidden levers” ==> “several hidden LAYERS”
LikeLike
Pingback: Sub(Surface)Stack | GeoEnergy Math
Pingback: Gist Evaluation | GeoEnergy Math
ChatGPT 4.0
How are the non-linear aspects of artificial neural networks related to Taylor’s series expansion of inputs, especially in regards to sinusoidally oscillating outputs? In particular, the optimal way of creating higher-order Taylor coefficients is by applying layered cross-terms of a neural net.
Neural networks are known for their ability to approximate complex, nonlinear functions. The connections between nodes, or “neurons,” in the network, have associated weights and biases that the network adjusts during the learning process to improve the approximation.
The relationship between artificial neural networks and Taylor series expansion can be drawn in terms of function approximation. Taylor series is a method to represent a function as an infinite sum of terms, each calculated from the function’s derivatives at a single point. It can be used to approximate nonlinear functions around a particular point.
In the context of neural networks, especially feedforward networks with activation functions, we can view these as approximating the Taylor series expansion of a function. Each layer of the network could be seen as contributing to the higher-order terms of the series.
Regarding sinusoidally oscillating outputs, sinusoidal functions can be represented using a Taylor series. Therefore, a sufficiently complex neural network should be able to learn to approximate a sinusoidal function. However, there are two key points to keep in mind:
1. Neural networks are typically better at interpolating within the range of data they’ve seen during training, rather than extrapolating beyond that range. So, if you’re trying to approximate a sinusoidal function that oscillates between specific points, you’ll want your training data to adequately cover this range.
2. When we say that each layer of a neural network can be seen as contributing to the higher-order terms of a Taylor series, this is a conceptual rather than a precise mathematical equivalence. In practice, the weights and biases that a neural network learns during training might not correspond directly to the coefficients in a Taylor series expansion. The network learns to approximate the function as a whole, rather than explicitly learning the coefficients of its Taylor series expansion.
Concerning the optimal creation of higher-order Taylor coefficients by applying layered cross-terms of a neural net, it might be more accurate to say that each layer in a deep neural network contributes to modeling the nonlinearities of the function it’s learning. The network doesn’t explicitly calculate or represent the Taylor series coefficients. Instead, it learns a set of weights and biases that allow it to approximate the function effectively. This learning process inherently captures the higher-order interactions between inputs, which is somewhat similar to the way higher-order terms in a Taylor series capture the function’s curvature.
Remember, the explanation here is a high-level conceptual interpretation. The actual mathematical operations that occur within a neural network during the learning process are quite complex and can’t be mapped directly to the computation of a Taylor series.
LikeLike