NN

20yrs of blogging in hindsight

/ Paul Pukite (@whut) / Leave a comment

Reminded by a 20-year anniversary post at RealClimate.org, that I’ve been blogging for 20 years + 6 months on topics of fossil fuel depletion + climate change. The starting point was at a BlogSpot blog I created in May 2004, where the first post set the stage:

Click on the above to go to the complete archives (almost daily posts) until I transitioned to WordPress and what became the present blog. After 2011, my blogging pace slowed down considerably as I started to write in more in more technical terms. Eventually the most interesting and novel posts were filtered down to a set that would eventually become the contents of Mathematical Geoenergy : Discovery, Depletion, and Renewal, published in late 2018/early 2019 by Wiley with an AGU imprint.

The arc that my BlogSpot/WordPress blogging activity followed occupies somewhat of a mirror universe to that of RealClimate. I initially started out with an oil depletion focus and by incrementally understanding the massive inertia that our FF-dependent society had developed, it placed the climate science aspect into a different perspective and context. After realizing that CO2 did not like to sequester, it became obvious that not much could be done to mitigate the impact of gradually increasing GHG levels, and that it would evolve into a slow-moving train wreck. That’s part of the reason why I focused more on research into natural climate variability. In contrast, RealClimate (and all the other climate blogs) continued to concentrate on man-made climate change. At this point, my climate fluid dynamics understanding is at some alternate reality level, see the last post, still very interesting but lacking any critical acceptance (no debunking either, which keeps it alive and potentially valid).

The oil depletion aspect more-or-less spun off into the PeakOilBarrel.com blog [*] maintained by my co-author Dennis Coyne. That’s like watching a slow-moving train wreck as well, but Dennis does an excellent job of keeping the suspense up with all the details in the technical modeling. Most of the predictions regarding peak oil that we published in 2018 are panning out.

As a parting thought, the RealClimate hindsight post touched on how AI will impact information flow going forward. Having worked on AI knowledgebases for environmental modeling during the LLM-precursor stage circa 2010-2013, I can attest that it will only get better. At the time, we were under the impression that knowledge used for modeling should be semantically correct and unambiguous (with potentially a formal representation and organization, see figure below), and so developed approaches for that here and here (long report form).

As it turned out, lack of correctness is just a stage, and AI users/customers are satisfied to get close-enough for many tasks. Eventually, the LLM robots will gradually clean up the sources of knowledge and converge more to semantic correctness. Same will happen with climate models as machine learning by the big guns at Google, NVIDIA, and Huawei will eventually discover what we have found in this blog over the course of 20+ years.

Note:
[*] In some ways the PeakOilBarrel.com blog is a continuation of the shuttered TheOilDrum.com blog, which closed shop in 2013 for mysterious reasons.

Sub(Surface)Stack

/ Paul Pukite (@whut) / 3 Comments

I signed up for a SubStack account awhile ago and recently published two articles on this account (SubSurface) in the last week.

The SubStack authoring interface has good math equation mark-up, convenient graphics embedding, and an excellent footnoting system. On first pass, it only lacks control over font color.

The articles are focused on applying neural network cross-validation to ENSO and AMO modeling, as suggested previously. I haven’t completely explored the configuration space but one aspect that may becoming clear is the value of wavelet neural networks (WNN) for time-series analysis. The WNN approach seems much more amenable to extracting sinusoidal modulation of the input-to-output mapping — trained on a rather short interval and then cross-validated out-of-band. The Mexican hat wavelet (2nd derivative of a Gaussian) as an activation function in particular locks in quickly to an LTE modulation that took longer to find with the custom search software I have developed at GitHub. I think the reason for the efficiency is that it’s optimizing to a Taylor’s series expansion of the input terms, a classic nonlinear expansion that NN’s excel at.

The following training run using the Mexican hat activation and ADAM optimizer is an eye-opener, as it achieved an admirable fit within a minute of computation.

The GREEN on BLUE is training on NINO4 data over two end-point intervals, with the RED cross-validation over the out-of-band region. The correlation coefficient is 0.34, which is impressive considering the nature of the waveform. Clearly there is similarity.

Moreover, if we compare the model fit to data via the WNN against the LTE harmonics approach, you can also see where the two fare equally poorly. Below in the outer frame is the NINO4 LTE fit with the YELLOW arrow pointing downward at a discrepancy (a peak in the data not resolved in the fit). In comparison the yellow-bordered inset shows the same discrepancy on the WNN training run. So the fingerprints essentially match with no coaching.

The neural net chain is somewhat deep with 6 layers, but I think this is needed to expand to the higher-order terms in the Taylor’s series. In the directed graph below, L01 is the input tidal forcing and L02 is the time axis (with an initial very low weighting).

It also appears temporally stationary across the entire time-span, so that the WNN temporal contribution appears minimal.

In a previous fit the horizontal striations (indicating modulation factor at a forcing level) matched with the LTE model, providing further evidence that the the WNN was mapping to an optimal modulation.

The other Sub(Surface)Stack article is on the AMO, which also reveals promising results. This is a video of the training in action