Introduction to PINNs

Main ideas and concepts covered

• Lecture goal (Lecture 3): Build and deploy Physics-Informed Neural Network (PINN)-based solvers.
• Why use ML/PINNs:
  • Use neural networks as surrogate models for PDE simulations.
  • Use neural networks for inverse problems where governing equations, parameters, or conditions are uncertain.
• Forward vs. inverse problems:
  • Forward problem: You know (or assume) the PDE and/or conditions and want to compute the solution.
  • Inverse problem: Some parts of the PDE/parameters/conditions are unknown or uncertain; you use observation data to learn the missing pieces (e.g., the PDE or conditions).

---

Neural network fundamentals (introduced to support PINNs)

Neural network structure

• Input layer
• Multiple hidden layers
• Output layer

Neuron/node behavior (conceptual)

Each neuron takes:

• Inputs
• Weights (W)
• Bias (b)

It computes a weighted sum (conceptually):

• (W_1X_1 + W_2X_2 + \cdots + b)

Then it applies an activation function (including a threshold-like effect):

• If the activated value is “high enough,” it propagates information forward.
• Otherwise it has little/no effect.

---

Loss functions, learning, and training

Loss function (core definition)

• Loss = a measure of error between:
  • the true value
  • and the predicted/estimated value
• Training aims to minimize loss, making predictions better.

Regression intuition (example)

• If true rent is 700 and the model guesses 1200:
  • the gap (error) becomes part of the loss
  • minimizing loss pushes predictions toward the truth

Common regression losses mentioned

• MAE-like idea: average of absolute differences between prediction and truth
• MSE-like idea: average of squared differences between prediction and truth

---

Backpropagation methodology (step-by-step, as described)

• Purpose: Decide how to update each neuron’s parameters (weights and biases) to reduce loss.
• Key mechanism: gradients via partial derivatives
  • Compute how changing a node output affects the loss using:
    • (\frac{\partial L}{\partial (\text{node output})})
  • Use the chain rule to relate changes in:
    • loss → node outputs → weights/biases
• Parameter update rule (gradient descent style):
  • New weights = old weights − (learning rate) × (gradient direction)
  • The learning rate is a hyperparameter controlling step size.
• Iterate training:
  • Repeat updates over many iterations until loss decreases sufficiently.
• Automatic differentiation (AD):
  • Instead of computing derivatives by hand, computers compute partial derivatives efficiently and accurately.
  • This is emphasized as important because PINNs rely on PDE derivatives.

---

PINNs: how physics constraints become part of the learning

Core idea of PINNs

Modify training so the predicted solution:

• satisfies the PDE
• satisfies initial/boundary/constraint conditions

This is done via a physics-informed loss: its minimum encourages compliance with the physics.

Inputs and outputs (example framing)

• Inputs to the PINN: typically coordinates in space and time, e.g.
  • ((x, t)) sampled over a domain
• Output from the PINN:
  • predicts the PDE solution (u(x,t)) (not only data labels)

Physics-informed loss (two terms)

• Total loss:

  • [ L = L_{PDE} + L_{BC/IC} ]

  • described as PDE part + conditional part

  • PDE residual term:
  • encourages:
    • (\frac{\partial u}{\partial t} - \Lambda\frac{\partial^2 u}{\partial x^2} \approx 0)
  • conceptually: (L_{PDE} = \text{(PDE residual)})
  • Constraint term:
  • enforces:
    • initial and/or boundary condition satisfaction
  • conceptually: loss is near zero when predicted values match the given condition equations

Training workflow and stopping/validation

• Use sampled training points over the domain.
• Use a validation set to check general behavior vs. memorizing training points.
• Stop training when validation loss stops decreasing.
• After training, the PINN acts as a surrogate model that can predict (u(x,t)) across the domain.

---

Example described (brief)

• Solves a PDE resembling: [ \frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2} ]

• Uses an initial condition like: [ u(x,0) = \sin(\pi x)\, e^{-t} ]

• Discusses:

  • PINN outputs over (x \in [-1,1]) and (t \in [0,1])
  • plotting/monitoring the PDE residual as a diagnostic

---

Error sources in PINNs (three “errors” / contributions)

• Optimization error
  • occurs if training doesn’t reach the global minimum (gets stuck in local minima)
  • mitigation:
    • train longer
    • restart with different initial weights/biases
    • use more powerful optimizers (with possible higher time/cost)
• Generalization error
  • due to finite training data / limited sampling
  • mitigation:
    • use more training points (may require more training time)
• Approximation error
  • due to limited network capacity to represent the true PDE solution
  • mitigation:
    • use a more expressive/more complex architecture
  • tradeoff:
    • can worsen optimization/generalization cost and complexity

---

Extensions and historical context (second speaker)

Broader PINN-related research overview

• PINNs are not the first physics + NN method:
  • Mentioned an earlier (1998) neural-network approach for solving ODE/PDE using:
    • collocation points
  • It required manual/unclear gradient/loss handling because AD wasn’t well defined then.
• Deep learning + renewed popularity:
  • PINNs became highly popular after increased deep-learning research activity.

Other related works mentioned

• 2019: Data-driven discovery of PDEs
  • infer PDE structure by fitting polynomial/differential terms from experimental data
  • described as influential for reviving physics-informed thinking
• Variational Physics-Informed Neural Networks (Variational PINNs)
  • uses a test function/network and variational formulation rather than only enforcing strong-form PDE
  • trains two networks:
    • one for the solution (u)
    • one for a test function (v)
• Variational Weak Adversarial Networks
  • introduces adversarial training between solution and test function networks
  • inspired by GAN-like alternating optimization
• Distributed/parallel domain decomposition
  • split the domain (\Omega) into subdomains (\Omega_i)
  • each GPU/MPI process trains its own PINN on its subdomain
  • enforce interface conditions at subdomain boundaries (and boundary conditions like normal/gradient constraints)
• Convergence/failure-mode research
  • multiple papers (2021) discussed:
    • when PINNs fail to train
    • failure modes
    • convergence analysis

Applications mentioned

• Surface reconstruction from point clouds
  • uses PDEs for signed distance fields (SDFs)
  • constraints:
    • boundary where SDF value is zero
    • normal equals gradient of SDF
  • loss terms from these constraints enable accurate reconstruction
• General claim: many other PDE-constrained deep learning applications exist.

Conclusion by second speaker

• Recapped:
  • physics-based surrogate models
  • PINNs and variants/extensions
• Next session planned:
  • hands-on training/coding PINNs for a specific application.

---

Speakers / sources featured (from subtitles)

People named

• Jian Yang (PhD student, Awa State University)
• Adu (co-worker, Awa State University)

Referenced research (paper sources; authors not explicitly named)

• A 1998 paper using artificial neural networks for ODE/PDE
• A 2019 Science Advances paper on data-driven discovery of PDEs
• Work on variational PINNs (2019, 2021 timeframe mentioned)
• Weak adversarial networks (2020 and 2021 mentioned)
• Distributed/parallel PINN extension (recent work; year not clearly stated beyond “more recently”)
• 2021 papers about PINN failure modes and convergence analysis