200万网格并行算力测试（OpenFOAM版本）
-
@东岳 在 200万网格管道流qDNS并行算力测试 中说：
多谢 @huningdong ，目前有2个6133的数据，你这个效果不错
不知道为什么 40和32 比较 完全就没什么差距呀
我再跑一下试试
-
@战气凌霄 在 200万网格管道流qDNS并行算力测试 中说：
@东岳 是不是把内存配置也写上更容易比较？6133的结果看起来应该是内存条数量不一样
会不会是内存通道的影响？
猜测 32和40 核心 比较结果 差别不大 是因为 32和用的通道数和40用的一样 所以 限制了计算性能的进一步提升？ -
@huningdong 您的和另外一个6133的数据比起来，您的计算速度更快，可能是您把内存通道配满了吧？ 至于32核和40核的结果，我觉得确实有可能是因为内存带宽跑满了，不过也不排除进程多了后进程间通信拖累？
补充一下我的数据：CPU型号：E5-2680V3，双路，8x8G DDR4-2133内存 系统：linux系统 OpenFOAM版本：OpenFOAM-8 24 139.36 16 168.45 8 196.13 4 352.66 2 648.47 1 1274.33 CPU型号：E5-2680V2，双路，8x8G DDR3-1600内存 系统：linux系统 OpenFOAM版本：OpenFOAM-8 20 180.27 16 198.1 8 204.31 4 350.34 2 615.33 1 1145.8 -
CPU型号：E5-2699 v4 系统：linux OpenFOAM版本：OpenFOAM-1906 44 173.99 32 170.82 16 193.78 8 194.74 4 313.11 2 585.66 1 1146.46为什么我的44核心会更慢呢，东岳老师？我最近在算一个200w网格的overset算例，算了半个多月了，算的还是很慢，有什么更好的解决方法吗？@东岳
-
@kennyadapter 我最近也发现有这个问题，主要在压力方程的求解器上，你看下你的44核与32核的每一步的压力方程迭代次数是否一致？
-
@东岳
这是我44核心的log文件的最后几个时间步：之前的迭代次数也差不多：Time = 9.8 Courant Number mean: 0.733368 max: 1.27353 GAMG: Solving for p, Initial residual = 0.0998056, Final residual = 0.00082233, No Iterations 6 time step continuity errors : sum local = 4.80691e-07, global = 1.99762e-20, cumulative = 1.59798e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123775 GAMG: Solving for p, Initial residual = 0.0809999, Final residual = 0.000584259, No Iterations 6 time step continuity errors : sum local = 3.45394e-07, global = 5.63939e-20, cumulative = 1.60362e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120435 GAMG: Solving for p, Initial residual = 0.0189678, Final residual = 9.4483e-07, No Iterations 39 time step continuity errors : sum local = 5.51633e-10, global = 3.45318e-20, cumulative = 1.60707e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638 ExecutionTime = 170.51 s ClockTime = 171 s Time = 9.9 Courant Number mean: 0.73339 max: 1.27886 GAMG: Solving for p, Initial residual = 0.100098, Final residual = 0.000824461, No Iterations 6 time step continuity errors : sum local = 4.82558e-07, global = 8.66441e-20, cumulative = 1.61574e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823 GAMG: Solving for p, Initial residual = 0.0809731, Final residual = 0.000577791, No Iterations 6 time step continuity errors : sum local = 3.42103e-07, global = 4.07868e-20, cumulative = 1.61981e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471 GAMG: Solving for p, Initial residual = 0.0189619, Final residual = 9.31189e-07, No Iterations 39 time step continuity errors : sum local = 5.44568e-10, global = 7.2382e-20, cumulative = 1.62705e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674 ExecutionTime = 172.22 s ClockTime = 172 s Time = 10 Courant Number mean: 0.733413 max: 1.27933 GAMG: Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6 time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6 time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39 time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 173.99 s ClockTime = 174 s End这是我32核心的：
Time = 9.8 Courant Number mean: 0.733369 max: 1.27363 GAMG: Solving for p, Initial residual = 0.0998053, Final residual = 0.000728134, No Iterations 6 time step continuity errors : sum local = 4.26347e-07, global = 1.16708e-20, cumulative = 5.86486e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123776 GAMG: Solving for p, Initial residual = 0.0810209, Final residual = 0.00054928, No Iterations 6 time step continuity errors : sum local = 3.24835e-07, global = 4.55333e-20, cumulative = 5.9104e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120434 GAMG: Solving for p, Initial residual = 0.0189497, Final residual = 8.88034e-07, No Iterations 38 time step continuity errors : sum local = 5.18253e-10, global = 2.24608e-20, cumulative = 5.93286e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638 ExecutionTime = 167.35 s ClockTime = 167 s Time = 9.9 Courant Number mean: 0.73339 max: 1.27894 GAMG: Solving for p, Initial residual = 0.100093, Final residual = 0.00073128, No Iterations 6 time step continuity errors : sum local = 4.28756e-07, global = 1.40838e-20, cumulative = 5.94694e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823 GAMG: Solving for p, Initial residual = 0.0809929, Final residual = 0.000542798, No Iterations 6 time step continuity errors : sum local = 3.21522e-07, global = -2.66223e-20, cumulative = 5.92032e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471 GAMG: Solving for p, Initial residual = 0.0189464, Final residual = 8.9418e-07, No Iterations 38 time step continuity errors : sum local = 5.22767e-10, global = 2.26167e-20, cumulative = 5.94294e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674 ExecutionTime = 169.06 s ClockTime = 169 s Time = 10 Courant Number mean: 0.733413 max: 1.27934 GAMG: Solving for p, Initial residual = 0.0999826, Final residual = 0.000732193, No Iterations 6 time step continuity errors : sum local = 4.29802e-07, global = 1.57268e-20, cumulative = 5.95866e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.0810305, Final residual = 0.00054485, No Iterations 6 time step continuity errors : sum local = 3.22957e-07, global = 2.11478e-20, cumulative = 5.97981e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189583, Final residual = 8.97181e-07, No Iterations 38 time step continuity errors : sum local = 5.24979e-10, global = 2.66036e-20, cumulative = 6.00642e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 170.82 s ClockTime = 171 s End -
@kennyadapter 你这个看起来就是带宽跟不上了
@来瓶热阔落 你看下你8核和4核的最开始时间步的log？类似这种，分别贴一下：Time = 0.1 Courant Number mean: 0.733413 max: 1.27933 GAMG: Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6 time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6 time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39 time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 173.99 s ClockTime = 174 s -
@东岳 在 200万网格管道流qDNS并行算力测试 中说：
东岳老师，那怎么调整带宽呢？有什么方法，或者需要更换什么配件吗？
-
Time = 0.1 Courant Number mean: 0.731999 max: 1.22083 GAMG: Solving for p, Initial residual = 0.629705, Final residual = 0.00421935, No Iterations 3 time step continuity errors : sum local = 7.82336e-06, global = -1.9955e-20, cumulative = -1.9955e-20 Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114208 GAMG: Solving for p, Initial residual = 0.0563421, Final residual = 0.000354153, No Iterations 4 time step continuity errors : sum local = 9.75747e-07, global = 2.11991e-20, cumulative = 1.24408e-21 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117378 GAMG: Solving for p, Initial residual = 0.0152771, Final residual = 8.66768e-07, No Iterations 18 time step continuity errors : sum local = 2.28999e-09, global = -1.80834e-20, cumulative = -1.68393e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.00011696 ExecutionTime = 4.61 s ClockTime = 5 s Time = 0.2 Courant Number mean: 0.731698 max: 1.1641 GAMG: Solving for p, Initial residual = 0.75843, Final residual = 0.00508023, No Iterations 3 time step continuity errors : sum local = 6.71701e-06, global = 1.79247e-12, cumulative = 1.79247e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119141 GAMG: Solving for p, Initial residual = 0.0716405, Final residual = 0.000689194, No Iterations 5 time step continuity errors : sum local = 5.93336e-07, global = 1.79247e-12, cumulative = 3.58494e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117204 GAMG: Solving for p, Initial residual = 0.014621, Final residual = 9.75203e-07, No Iterations 25 time step continuity errors : sum local = 8.46786e-10, global = 1.79247e-12, cumulative = 5.37741e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116758 ExecutionTime = 9.1 s ClockTime = 9 s上面是4核，下面8核
Time = 0.1 Courant Number mean: 0.731999 max: 1.22083 GAMG: Solving for p, Initial residual = 0.629703, Final residual = 0.00603973, No Iterations 3 time step continuity errors : sum local = 1.11982e-05, global = 8.06587e-21, cumulative = 8.06587e-21 Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114193 GAMG: Solving for p, Initial residual = 0.0566187, Final residual = 0.00046644, No Iterations 4 time step continuity errors : sum local = 1.28491e-06, global = -2.9891e-20, cumulative = -2.18251e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117396 GAMG: Solving for p, Initial residual = 0.0153571, Final residual = 7.70794e-07, No Iterations 20 time step continuity errors : sum local = 2.0346e-09, global = -1.94705e-20, cumulative = -4.12956e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000116978 ExecutionTime = 5.64 s ClockTime = 6 s Time = 0.2 Courant Number mean: 0.731698 max: 1.16413 GAMG: Solving for p, Initial residual = 0.75841, Final residual = 0.00736995, No Iterations 3 time step continuity errors : sum local = 9.74458e-06, global = 8.78067e-13, cumulative = 8.78067e-13 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119176 GAMG: Solving for p, Initial residual = 0.073429, Final residual = 0.000652992, No Iterations 6 time step continuity errors : sum local = 5.62596e-07, global = 8.78067e-13, cumulative = 1.75613e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117203 GAMG: Solving for p, Initial residual = 0.0147366, Final residual = 9.38466e-07, No Iterations 23 time step continuity errors : sum local = 8.14733e-10, global = 8.78067e-13, cumulative = 2.6342e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116759 ExecutionTime = 11.02 s ClockTime = 11 s -
CPU: AMD Opteron(tm) Processor 6376 OS: debian 10 OpenFOAM: 8 # cores Wall time (s): ------------------------ 64 164.29 32 180.06 16 305.33 8 407.37 4 800.44 2 1981.19 1 3383.18CPU: Intel(R) Xeon(R) Platinum 8175M OS: ubuntu 18.04 OpenFOAM: 8 # cores Wall time (s): ------------------------ 48 77.08 24 99.2 12 123.14 6 234.82 3 366.02 1 1149.63CPU: Intel(R) Xeon(R) E5-2683 OS: ubuntu 14.04 OpenFOAM: 8 # cores Wall time (s): ------------------------ 28 224.8 14 214.99 8 246.61 4 383.36 2 721.88 1 1361滚来滚去……~(～o￣▽￣)～o 滚来滚去都不能让大家看出来我不是老师么 O_o
异步沟通方式(《posting style》from wiki)（下载后打开）：
https://www.jianguoyun.com/p/Dc52X2sQsLv2BRiqnKYD
提问的智慧(github在gitee的镜像)：
https://gitee.com/bestucan/How-To-Ask-Questions-The-Smart-Way -
滚来滚去……~(～o￣▽￣)～o 滚来滚去都不能让大家看出来我不是老师么 O_o
异步沟通方式(《posting style》from wiki)（下载后打开）：
https://www.jianguoyun.com/p/Dc52X2sQsLv2BRiqnKYD
提问的智慧(github在gitee的镜像)：
https://gitee.com/bestucan/How-To-Ask-Questions-The-Smart-Way -
