200万网格并行算力测试（OpenFOAM版本）

[merlinizlf]

好的，我重新再算一遍

[玉玺]

沾一沾大佬算例的福气

[merlinizlf]

CPU型号：i7-8750H
系统：虚拟机
OpenFOAM版本：OpenFOAM-6
4 910.12
2 883.78
1 1060.35

重新跑的结果，这次除了虚拟机什么都没运行。。。
希望能对统计有点用。。。

[李东岳]

这回结果看起来还好

[李东岳]

CPU型号：霄龙7302
系统：linux系统
OpenFOAM版本：OpenFOAM-7
32 66.5
16 101.16
8 135.76
4 260.99
2 503.16
1 1074.03

[战气凌霄]

CPU型号：E5-2680V3
系统：linux系统
OpenFOAM版本：OpenFOAM-8
16 168.45
8 196.13
4 352.66
2 648.47
1 1274.33

CPU型号：E5-2680V2
系统：linux系统
OpenFOAM版本：OpenFOAM-8
16 198.1
8 204.31
4 350.34
2 615.33
1 1145.8

[李东岳]

@战气凌霄 多谢

CPU型号：金牌6130
系统：linux系统
OpenFOAM版本：OpenFOAM-7
32 91.22
16 121.43
8 157.92
4 310.89
2 622.44
1 1241.62

[huningdong]

Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz
MemTotal:       262566620 kB
CentOS Linux release 7.3.1611 (Core)
Using: OpenFOAM-6
40 86.58
32 86.77
16 160.35
8 183.99
4 336.65
2 624.18
1 1410.07

[李东岳]

多谢 @huningdong ，目前有2个6133的数据，你这个效果不错

[huningdong]

@东岳 在 200万网格管道流qDNS并行算力测试 中说：

多谢 @huningdong ，目前有2个6133的数据，你这个效果不错

不知道为什么 40和32 比较 完全就没什么差距呀
我再跑一下试试

[战气凌霄]

@东岳 是不是把内存配置也写上更容易比较？6133的结果看起来应该是内存条数量不一样

[huningdong]

@战气凌霄 在 200万网格管道流qDNS并行算力测试 中说：

@东岳 是不是把内存配置也写上更容易比较？6133的结果看起来应该是内存条数量不一样

会不会是内存通道的影响？
猜测 32和40 核心 比较结果 差别不大 是因为 32和用的通道数和40用的一样 所以 限制了计算性能的进一步提升？

[战气凌霄]

@huningdong 您的和另外一个6133的数据比起来，您的计算速度更快，可能是您把内存通道配满了吧？ 至于32核和40核的结果，我觉得确实有可能是因为内存带宽跑满了，不过也不排除进程多了后进程间通信拖累？
补充一下我的数据：

CPU型号：E5-2680V3，双路，8x8G DDR4-2133内存
系统：linux系统
OpenFOAM版本：OpenFOAM-8
24  139.36
16 168.45
8 196.13
4 352.66
2 648.47
1 1274.33

CPU型号：E5-2680V2，双路，8x8G DDR3-1600内存
系统：linux系统
OpenFOAM版本：OpenFOAM-8
20  180.27
16 198.1
8 204.31
4 350.34
2 615.33
1 1145.8

[kennyadapter]

CPU型号：E5-2699 v4
系统：linux
OpenFOAM版本：OpenFOAM-1906
44 173.99
32 170.82
16 193.78
8 194.74
4 313.11
2 585.66
1 1146.46

为什么我的44核心会更慢呢，东岳老师？我最近在算一个200w网格的overset算例，算了半个多月了，算的还是很慢，有什么更好的解决方法吗？@东岳

[李东岳]

@kennyadapter 我最近也发现有这个问题，主要在压力方程的求解器上，你看下你的44核与32核的每一步的压力方程迭代次数是否一致？

[曳尾鱼]

这个算例算了多少时间步?

[kennyadapter]

@东岳
这是我44核心的log文件的最后几个时间步：之前的迭代次数也差不多：

Time = 9.8

Courant Number mean: 0.733368 max: 1.27353
GAMG:  Solving for p, Initial residual = 0.0998056, Final residual = 0.00082233, No Iterations 6
time step continuity errors : sum local = 4.80691e-07, global = 1.99762e-20, cumulative = 1.59798e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123775
GAMG:  Solving for p, Initial residual = 0.0809999, Final residual = 0.000584259, No Iterations 6
time step continuity errors : sum local = 3.45394e-07, global = 5.63939e-20, cumulative = 1.60362e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120435
GAMG:  Solving for p, Initial residual = 0.0189678, Final residual = 9.4483e-07, No Iterations 39
time step continuity errors : sum local = 5.51633e-10, global = 3.45318e-20, cumulative = 1.60707e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
ExecutionTime = 170.51 s  ClockTime = 171 s

Time = 9.9

Courant Number mean: 0.73339 max: 1.27886
GAMG:  Solving for p, Initial residual = 0.100098, Final residual = 0.000824461, No Iterations 6
time step continuity errors : sum local = 4.82558e-07, global = 8.66441e-20, cumulative = 1.61574e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
GAMG:  Solving for p, Initial residual = 0.0809731, Final residual = 0.000577791, No Iterations 6
time step continuity errors : sum local = 3.42103e-07, global = 4.07868e-20, cumulative = 1.61981e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
GAMG:  Solving for p, Initial residual = 0.0189619, Final residual = 9.31189e-07, No Iterations 39
time step continuity errors : sum local = 5.44568e-10, global = 7.2382e-20, cumulative = 1.62705e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
ExecutionTime = 172.22 s  ClockTime = 172 s

Time = 10

Courant Number mean: 0.733413 max: 1.27933
GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
ExecutionTime = 173.99 s  ClockTime = 174 s

End

这是我32核心的：

Time = 9.8

Courant Number mean: 0.733369 max: 1.27363
GAMG:  Solving for p, Initial residual = 0.0998053, Final residual = 0.000728134, No Iterations 6
time step continuity errors : sum local = 4.26347e-07, global = 1.16708e-20, cumulative = 5.86486e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123776
GAMG:  Solving for p, Initial residual = 0.0810209, Final residual = 0.00054928, No Iterations 6
time step continuity errors : sum local = 3.24835e-07, global = 4.55333e-20, cumulative = 5.9104e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120434
GAMG:  Solving for p, Initial residual = 0.0189497, Final residual = 8.88034e-07, No Iterations 38
time step continuity errors : sum local = 5.18253e-10, global = 2.24608e-20, cumulative = 5.93286e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
ExecutionTime = 167.35 s  ClockTime = 167 s

Time = 9.9

Courant Number mean: 0.73339 max: 1.27894
GAMG:  Solving for p, Initial residual = 0.100093, Final residual = 0.00073128, No Iterations 6
time step continuity errors : sum local = 4.28756e-07, global = 1.40838e-20, cumulative = 5.94694e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
GAMG:  Solving for p, Initial residual = 0.0809929, Final residual = 0.000542798, No Iterations 6
time step continuity errors : sum local = 3.21522e-07, global = -2.66223e-20, cumulative = 5.92032e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
GAMG:  Solving for p, Initial residual = 0.0189464, Final residual = 8.9418e-07, No Iterations 38
time step continuity errors : sum local = 5.22767e-10, global = 2.26167e-20, cumulative = 5.94294e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
ExecutionTime = 169.06 s  ClockTime = 169 s

Time = 10

Courant Number mean: 0.733413 max: 1.27934
GAMG:  Solving for p, Initial residual = 0.0999826, Final residual = 0.000732193, No Iterations 6
time step continuity errors : sum local = 4.29802e-07, global = 1.57268e-20, cumulative = 5.95866e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
GAMG:  Solving for p, Initial residual = 0.0810305, Final residual = 0.00054485, No Iterations 6
time step continuity errors : sum local = 3.22957e-07, global = 2.11478e-20, cumulative = 5.97981e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
GAMG:  Solving for p, Initial residual = 0.0189583, Final residual = 8.97181e-07, No Iterations 38
time step continuity errors : sum local = 5.24979e-10, global = 2.66036e-20, cumulative = 6.00642e-18
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
ExecutionTime = 170.82 s  ClockTime = 171 s

End

[来瓶热阔落]

CPU型号：AMD-3900X
系统：Ubuntu18   32g 3000
OpenFOAM版本：OpenFOAM-7
8 406.12
4 382.44
2 512.83
1 915.2

为啥3900x 8核反而慢了

[李东岳]

@kennyadapter 你这个看起来就是带宽跟不上了
@来瓶热阔落 你看下你8核和4核的最开始时间步的log？类似这种，分别贴一下：

Time = 0.1

Courant Number mean: 0.733413 max: 1.27933
GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
ExecutionTime = 173.99 s  ClockTime = 174 s

[kennyadapter]

@东岳 在 200万网格管道流qDNS并行算力测试 中说：

东岳老师，那怎么调整带宽呢？有什么方法，或者需要更换什么配件吗？