大网格-奇怪报错

random_ran

还是在算圆柱绕流。为了进入critical雷诺数，网格加密了很多，导致的问题也很多。192核那个例子的网格数在600W，运行没有什么问题。

可是在其他条件相同，但是用更多的网格的288核算例中出现奇怪的错误。读取初始和边界条件，湍流模型都没有什么问题，就是到 smoothsovler 那里出现了问题。

288核这个case，网格数在3千万。

个人感觉可能是内存的问题，但是我们这的计算单个节点内存是32GB，照理说12个节点的总内存量应该是能应付的？难道是计算到smoothsovler这一步，在某一个节点的内存量大大超过了32GB？

192 正常运算

Create time

Create mesh for time = 300


PISO: Operating solver in PISO mode

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type LES
Selecting LES turbulence model dynamicKEqn
Selecting LES delta type cubeRootVol
bounding k, min: 0 max: 0.0015093 average: 1.99878e-05
dynamicKEqnCoeffs
{
    filter          simple;
    Ce              1.048;
}

No MRF models present

No finite volume options present

Time = 300.02

Courant Number mean: 0.026086 max: 0.983623
smoothSolver:  Solving for Ux, Initial residual = 0.000816211, Final residual = 2.84247e-08,\
 No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.00267408, Final residual = 2.09581e-08, \
No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.00457375, Final residual = 3.57969e-08, \
No Iterations 4
GAMG:  Solving for p, Initial residual = 0.220597, Final residual = 0.00856105, No Iteration\
s 3
time step continuity errors : sum local = 6.24882e-08, global = 5.53688e-10, cumulative = 5.\
53688e-10
GAMG:  Solving for p, Initial residual = 0.00891699, Final residual = 0.000438876, No Iterat\
ions 12
time step continuity errors : sum local = 3.18791e-09, global = -1.1464e-10, cumulative = 4.\
39048e-10
GAMG:  Solving for p, Initial residual = 0.000445623, Final residual = 9.53731e-08, No Itera\
tions 25
time step continuity errors : sum local = 6.89127e-13, global = -7.61126e-14, cumulative = 4\
.38972e-10
smoothSolver:  Solving for k, Initial residual = 0.000553063, Final residual = 3.40938e-08, \
No Iterations 3
bounding k, min: -3.68878e-09 max: 0.00149781 average: 1.99956e-05
ExecutionTime = 6.64 s  ClockTime = 7 s

288 核异常错误

Create time

Create mesh for time = 0


PISO: Operating solver in PISO mode

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type LES
Selecting LES turbulence model dynamicKEqn
Selecting LES delta type cubeRootVol
bounding k, min: 0 max: 2e-05 average: 0
dynamicKEqnCoeffs
{
    filter          simple;
    Ce              1.048;
}

No MRF models present

No finite volume options present


Starting time loop

Time = 0.005


Courant Number mean: 0.000119441 max: 0.017568
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\
ions 2
GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\
9.25811e-08
GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\
tions 39
time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\
9.84343e-08
GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\
tions 74
time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\
.84333e-08
--------------------------------------------------------------------------
mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
 point exception).
--------------------------------------------------------------------------

另外的报错信息：

An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          cp1815 (PID 31613)
  MPI_COMM_WORLD rank: 89

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 105 with PID 0 on node cp1652 exited on signal 8 (Floating\
 point exception).

我查找了相关信息，在提交作业的脚本文件做了相应的改变：

mpiexec --mca mpi_warn_on_fork 0 -n 288 pisoFoam -parallel > simpleFoam_${PBS_JOBID}.out

还是不能计算

Time = 0.005

Courant Number mean: 0.000119441 max: 0.017568
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\
ions 2
GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\
9.25811e-08
GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\
tions 39
time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\
9.84343e-08
GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\
tions 74
time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\
.84333e-08
--------------------------------------------------------------------------
mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
 point exception).
--------------------------------------------------------------------------

还请有经验的朋友分析一下这个问题。

CFD中文网

@random_ran

Sorry, Chinese input missing.

mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
 point exception).

This is a quite general error, which does not mean it comes from setting up of mpi. Meanwhile, sometimes your simulation blows up in parallel running, see: link text.

Which version of OpenFOAM do you use? Could u please try PBiCG instead of smoothSolver? How do you decompose your grid?

random_ran

@cfd-china
补充信息：

OpenFOAM v4.1


/*--------------------------------*- C++ -*----------------------------------*\                                                                                            
| =========                 |                                                 |                                                                                            
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                                                                                            
|  \\    /   O peration     | Version:  4.x                                   |                                                                                            
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                                                                                            
|    \\/     M anipulation  |                                                 |                                                                                            
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                                                                                            

numberOfSubdomains 288;

method          hierarchical;

hierarchicalCoeffs
{
    n               (16 9 2);
    delta           0.001;
    order           xyz;


hierarchicalCoeffs
{
    n               (16 9 2);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}


// ************************************************************************* //

*--------------------------------*- C++ -*----------------------------------*\                                                                                            
| =========                 |                                                 |                                                                                            
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                                                                                            
|  \\    /   O peration     | Version:  4.x                                   |                                                                                            
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                                                                                            
|    \\/     M anipulation  |                                                 |                                                                                            
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      fvSolution;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                                                                                            

solvers
{
    p
    {
        solver          GAMG;
        tolerance       1e-07;
        relTol          0.05;
        smoother        GaussSeidel;
    }

    pFinal
    {
        $p;
        smoother        DICGaussSeidel;
        tolerance       1e-07;
	relTol          0;
    }

    "(U|k|B|nuTilda|s)"
    {
        solver          smoothSolver;
        smoother        GaussSeidel;
        tolerance       1e-07;
        relTol          0;
    }
}

PISO
{
    nCorrectors     3;
    nNonOrthogonalCorrectors 0;
}

relaxationFactors
{
  "U.*"               1;
  "nuTilda.*"         1;
}

random_ran

再补充：

192 case 的雷诺数在10^4
288 case 的在10^5

这两个case的分解方法，controDict, fvSchemes, fvSolution 基本一样。
不同的是288 case 的时间步长更小而已。

random_ran

k 文件 tubulent kinetic energy：

*--------------------------------*- C++ -*----------------------------------*\                  
| =========                 |                                                 |                  
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                  
|  \\    /   O peration     | Version:  4.x                                   |                  
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                  
|    \\/     M anipulation  |                                                 |                  
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       volScalarField;
    object      k;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                  

dimensions      [0 2 -2 0 0 0 0];

internalField   uniform 0;

boundaryField
{
    INLET
    {
        type            fixedValue;
        value           uniform 2e-05;
    }

    OUTLET
    {
        type            inletOutlet;
        inletValue      uniform 0;
        value           uniform 0;
    }


    BACK_CYC
    {
        type            cyclic;
    }
    FRONT_CYC
    {
        type            cyclic;
    }

    CYLINDER
    {
        type            fixedValue;
        value           uniform 0;
    }

}

random_ran

@cfd-china

非常感谢你的建议！非常管用，起码现在能跑起来了。


Time = 0.005

Courant Number mean: 0.000119441 max: 0.017568
DILUPBiCG:  Solving for Ux, Initial residual = 1, Final residual = 2.34151e-11, No Iterations 2
DILUPBiCG:  Solving for Uy, Initial residual = 1, Final residual = 1.61665e-10, No Iterations 2
DILUPBiCG:  Solving for Uz, Initial residual = 1, Final residual = 1.63882e-10, No Iterations 2
GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -9.25811e-08
GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Iterations 39
time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -9.84343e-08
GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Iterations 74
time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9.84333e-08
DILUPBiCG:  Solving for k, Initial residual = 0.297752, Final residual = 7.56559e-07, No Iterations 4
bounding k, min: -2.24189e-08 max: 2e-05 average: 2.54114e-09
ExecutionTime = 42.97 s  ClockTime = 43 s

我改动的fvSolution 部分是：

    "(U|k|B|nuTilda|s)"
    {
        solver          PBiCG;
        preconditioner  DILU
        tolerance       1e-07;
        relTol          0;
    }

目前运行了半个小时了。

我知道后面还会有的新的问题。

白礼耕

你好，我遇到了和你288核算例很相似的报错。请问“另外的报错信息”是怎么得到的？我只能看得到noticed that process rank 105 with PID 0 on node xxxxxx exited on signal 8 (Floating point exception) 要怎样才能得到另外的信息呢？

random_ran

@白礼耕 “”应该只是报警不是报错。

时间好久了，当时好像用到的命令：

mpiexec --mca mpi_warn_on_fork 0 -n 288 pisoFoam -parallel > 0-pisoFoam_LES_288_${PBS_JOBID}.out

CFD中文网

大网格-奇怪报错