HermiteLBM

HermiteLBM is a high-performance computing project that implements the moment representation of the Lattice Boltzmann Method (LBM) on NVIDIA GPUs using CUDA. The code supports both single‑GPU and multi‑GPU execution (via domain decomposition and peer‑to‑peer memory transfers). This project is under active development and is primarily targeted at Linux‑based systems.

🚀 Features

• Lattice Boltzmann Method: Implements the moment‑based LBM for fluid dynamics simulations.
• D3Q19 and D3Q27 Velocity Sets: Both D3Q19 and D3Q27 lattices are fully supported for three‑dimensional simulations.
• Multi‑GPU Support: Scales across multiple GPUs using domain decomposition. The communication between GPUs is currently synchronous and not overlapped with computation.
• Example Cases: Two validation cases are provided:
  • Lid‑Driven Cavity: Standard benchmark for 3D flows.
  • Jet Flow: A submerged jet configuration demonstrating shear‑layer dynamics.

⚡ Performance

The code has been benchmarked with the lid driven cavity case using D3Q19 and FP32 for both storage and arithmetic. The performance on various NVIDIA GPUs is as follows:

• NVIDIA RTX A4000: ~3300 MLUPS
• NVIDIA RTX 4090: ~6600 MLUPS
• NVIDIA RTX 5090: ~12900 MLUPS

MLUPS stands for Million Lattice Updates Per Second. There is still room for improvement, as further optimizations are planned.

📅 Future Features

• Extended boundary conditions: Future versions will implement runtime-selectable boundary conditions via dynamic dispatch.

🔧 Getting Started

Prerequisites

• A C++ compiler (e.g., GCC)
• NVIDIA CUDA Toolkit

Installation

1. Clone the repository:

   git clone [https://github.com/Geoenergia-Lab/HermiteLBM.git](https://github.com/Geoenergia-Lab/HermiteLBM.git)

2. Navigate to the project directory and load the bashrc file:

   cd HermiteLBM
   source bashrc

3. Compile the project:

   make install

💨 Usage

Running a Simulation

To run a simulation, you must first navigate to the case directory (e.g., the lidDrivenCavity folder). You can then execute the compiled binary for the desired model:

# For the D3Q19 model
momentBasedD3Q19 -GPU 0

# For the D3Q27 model
momentBasedD3Q27 -GPU 0

Required Files

The following files must be present in the working (case) directory to run a simulation:

• programControl
• latticeMesh
• initialConditions

Additionally, the optional functionObjects file may be present to enable runtime post-processing.

Runtime Function Objects

The functionObjects file allows you to compute various quantities at runtime. Currently, it supports calculating the kinetic energy and the strain rate tensor of the flow, as well as their time averages.

For example, including the following in your functionObjects file:

functionObjectList
{
    S;
    SMean;
    k;
    kMean;
};

will enable the calculation of the strain rate and kinetic energy, and additionally their time-averaged values.

📦 Post-Processing

The project includes two key executables for post-processing: fieldConvert and fieldCalculate.

fieldConvert

fieldConvert is primarily used to convert the .LBMBin files (saved by the solver) into other file types for visualization (e.g., .vts for ParaView).

• Default Conversion: By default, it converts the 10 hydrodynamic moment variables.

  fieldConvert -fileType vts

• Convert Specific Fields: You can convert other fields, like those generated by functionObjects.

  fieldConvert -fileType vts -fieldName kMean

• Extract Cut Plane: It can also extract a cut plane in an arbitrary cardinal plane at a specific value.

  fieldConvert -fileType vts -fieldName kMean -cutPlane z=0.5

fieldCalculate

fieldCalculate is used to perform mathematical operations on the solution fields.

• Check for NaN: Inspect whether a field contains NaN values.

  fieldCalculate -calculationType containsNaN

• Calculate Derivatives:

  # Calculate vorticity
  fieldCalculate -calculationType vorticity

  # Calculate divergence of velocity
  fieldCalculate -calculationType div[U]

• Calculate Spatial Mean:

  # Calculate spatial mean of velocity (default)
  fieldCalculate -calculationType spatialMean

  # Calculate spatial mean of a specific field
  fieldCalculate -calculationType spatialMean -fieldName kMean

⏱️ Latest Time Flag

Both fieldConvert and fieldCalculate can be limited to run on only the latest saved time step by adding the -latestTime flag. This is useful for quick visualization or saving disk space.

fieldConvert -fileType vts -fieldName kMean -latestTime

fieldCalculate -fileType vts -calculationType div[U] -latestTime

📄 License

This project is licensed under the terms of the LICENSE file.

🙏 Acknowledgments

This codebase was heavily influenced by the MR-LBM project, although it has been completely rewritten. You can find the original repository here:

https://github.com/CERNN/MR-LBM