200万网格管道流qDNS并行算力测试



  • 这是一个管道流动的qDNS算例,基于OpenFOAM。使用pisoFoam就可以运行。

    如何测试CPU算力

    1. 下载 https://www.jianguoyun.com/p/DeabU1wQ9s3ZBhjc7LgD 解压,放到任意的文件夹下

    2. 打开Allrun文件,将其中的 for i in 4 2 1; do改成你的处理器数量,比如你的处理器是32核,就改成32 16 8 4 2 1,注意,上下有两处,都要改一下

    3. 直接在终端输入./Allrun即可运行,什么都不用管了。终端不要关闭。

    4. 看终端的结果,应该类似下图,其中的数字就是结果。

    5. 回帖,告诉我你的CPU型号和数据,我整理在下表中

    捕获.JPG

    系统 CPU 1 2 4 8 16 32 48 64 128
    真系统 7742 1370 717 337 205 128 103 62
    真系统 2699-v4 1146 585 313 194 193 170 173
    真系统 6376 3383 1981 800 407 305 180 164
    真系统 8175M 1124 573 312 167 132 87 76
    真系统 7402 - 568 283 140 84 68 65
    真系统 5218 1247 637 316 205 214 172
    真系统 6133(user1) 1258 860 437 329 388 358
    真系统 6133(user2) 1410 624 336 183 160 86
    真系统 7282 1125 498 268 140 121 82
    真系统 7302 1074 503 261 136 101 66
    真系统 6130 1241 622 310 157 121 91
    真系统 2680V3 1274 648 352 196 168
    真系统 2680V2 1145 615 350 204 198
    真系统 4110 1361 948 500 395 467
    虚拟机 8400 1114 864 861 900
    真系统 3900X 915 512 382 406
    虚拟机 5820k 1117 691 542
    虚拟机 8750H(user1) 1060 883 910
    虚拟机 8750H(user2) 1380 1109 963

    云图

    云图结果类似,结果曾经跟JCP一篇文章对比过,完全符合。更详细的关于算法的可参考 链接文本

    dns2.jpg

    替代文字



  • 回帖范例:

    CPU型号:5900k
    系统:linux系统
    OpenFOAM版本:OpenFOAM-7
    16 400
    8 442
    4 542
    2 691
    1 1119
    
    • 注意,一些处理器可能不是整数,那就这样也行:40 32 16 8 4 2 1,优先保障有32 16 8 4 2 1..


  • CPU型号:AMD 霄龙 7742
    系统:linux系统
    OpenFOAM版本:OpenFOAM-v1912
    64 62.23
    32 103.27
    16 128.03
    8 205.02
    4 337.93
    2 717.1 (不是最佳状态)
    1 1370.22(不是最佳状态)
    


  • @maoyanjun_dut 谢谢!:146: :146:



  • @东岳 持续关注中,排排队,看看哪个最能打:high:



  • CPU型号:i7-8750H
    系统:虚拟机
    OpenFOAM版本:OpenFOAM-6
    4 920.13
    2 1098.04
    1 572.75
    

    为什么单核的时候反而最快?:143:



  • CPU型号:Intel(R) Xeon (R) Gold 5218 (2)
    系统:Linux系统
    OpenFOAM版本: OpenFOAM-6
    32    172.83
    16    214
    8     205.85
    4     316.65
    2     637.1
    1     1247.31
    


  • 多谢各位,:xiexie:

    @merlinizlf 你重新跑一下试试?或者你看看结果都正确不?



  • 好的,我重新再算一遍:yes:



  • 沾一沾大佬算例的福气:xiezuoye:



  • CPU型号:i7-8750H
    系统:虚拟机
    OpenFOAM版本:OpenFOAM-6
    4 910.12
    2 883.78
    1 1060.35
    

    重新跑的结果,这次除了虚拟机什么都没运行。。。
    希望能对统计有点用。。。:xinlei:



  • 这回结果看起来还好 :xiexie:



  • CPU型号:霄龙7302
    系统:linux系统
    OpenFOAM版本:OpenFOAM-7
    32 66.5
    16 101.16
    8 135.76
    4 260.99
    2 503.16
    1 1074.03
    


  • CPU型号:E5-2680V3
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    16 168.45
    8 196.13
    4 352.66
    2 648.47
    1 1274.33
    
    CPU型号:E5-2680V2
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    16 198.1
    8 204.31
    4 350.34
    2 615.33
    1 1145.8
    


  • @战气凌霄 多谢:146:

    CPU型号:金牌6130
    系统:linux系统
    OpenFOAM版本:OpenFOAM-7
    32 91.22
    16 121.43
    8 157.92
    4 310.89
    2 622.44
    1 1241.62
    


  • Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz
    MemTotal:       262566620 kB
    CentOS Linux release 7.3.1611 (Core)
    Using: OpenFOAM-6
    40 86.58
    32 86.77
    16 160.35
    8 183.99
    4 336.65
    2 624.18
    1 1410.07
    

    :xiezuoye:



  • 多谢 @huningdong ,目前有2个6133的数据,你这个效果不错



  • @东岳200万网格管道流qDNS并行算力测试 中说:

    多谢 @huningdong ,目前有2个6133的数据,你这个效果不错

    不知道为什么 40和32 比较 完全就没什么差距呀:jingya:
    我再跑一下试试:xiezuoye:



  • @东岳 是不是把内存配置也写上更容易比较?6133的结果看起来应该是内存条数量不一样



  • @战气凌霄200万网格管道流qDNS并行算力测试 中说:

    @东岳 是不是把内存配置也写上更容易比较?6133的结果看起来应该是内存条数量不一样

    会不会是内存通道的影响?
    猜测 32和40 核心 比较结果 差别不大 是因为 32和用的通道数和40用的一样 所以 限制了计算性能的进一步提升?



  • @huningdong 您的和另外一个6133的数据比起来,您的计算速度更快,可能是您把内存通道配满了吧? 至于32核和40核的结果,我觉得确实有可能是因为内存带宽跑满了,不过也不排除进程多了后进程间通信拖累?
    补充一下我的数据:

    CPU型号:E5-2680V3,双路,8x8G DDR4-2133内存
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    24  139.36
    16 168.45
    8 196.13
    4 352.66
    2 648.47
    1 1274.33
    
    CPU型号:E5-2680V2,双路,8x8G DDR3-1600内存
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    20  180.27
    16 198.1
    8 204.31
    4 350.34
    2 615.33
    1 1145.8
    


  • CPU型号:E5-2699 v4
    系统:linux
    OpenFOAM版本:OpenFOAM-1906
    44 173.99
    32 170.82
    16 193.78
    8 194.74
    4 313.11
    2 585.66
    1 1146.46
    
    

    为什么我的44核心会更慢呢,东岳老师?我最近在算一个200w网格的overset算例,算了半个多月了,算的还是很慢,有什么更好的解决方法吗?@东岳



  • @kennyadapter 我最近也发现有这个问题,主要在压力方程的求解器上,你看下你的44核与32核的每一步的压力方程迭代次数是否一致?



  • 这个算例算了多少时间步?



  • @东岳
    这是我44核心的log文件的最后几个时间步:之前的迭代次数也差不多:

    
    Time = 9.8
    
    Courant Number mean: 0.733368 max: 1.27353
    GAMG:  Solving for p, Initial residual = 0.0998056, Final residual = 0.00082233, No Iterations 6
    time step continuity errors : sum local = 4.80691e-07, global = 1.99762e-20, cumulative = 1.59798e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123775
    GAMG:  Solving for p, Initial residual = 0.0809999, Final residual = 0.000584259, No Iterations 6
    time step continuity errors : sum local = 3.45394e-07, global = 5.63939e-20, cumulative = 1.60362e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120435
    GAMG:  Solving for p, Initial residual = 0.0189678, Final residual = 9.4483e-07, No Iterations 39
    time step continuity errors : sum local = 5.51633e-10, global = 3.45318e-20, cumulative = 1.60707e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
    ExecutionTime = 170.51 s  ClockTime = 171 s
    
    Time = 9.9
    
    Courant Number mean: 0.73339 max: 1.27886
    GAMG:  Solving for p, Initial residual = 0.100098, Final residual = 0.000824461, No Iterations 6
    time step continuity errors : sum local = 4.82558e-07, global = 8.66441e-20, cumulative = 1.61574e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
    GAMG:  Solving for p, Initial residual = 0.0809731, Final residual = 0.000577791, No Iterations 6
    time step continuity errors : sum local = 3.42103e-07, global = 4.07868e-20, cumulative = 1.61981e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
    GAMG:  Solving for p, Initial residual = 0.0189619, Final residual = 9.31189e-07, No Iterations 39
    time step continuity errors : sum local = 5.44568e-10, global = 7.2382e-20, cumulative = 1.62705e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
    ExecutionTime = 172.22 s  ClockTime = 172 s
    
    Time = 10
    
    Courant Number mean: 0.733413 max: 1.27933
    GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
    time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
    time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
    time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 173.99 s  ClockTime = 174 s
    
    End
    
    

    这是我32核心的:

    
    Time = 9.8
    
    Courant Number mean: 0.733369 max: 1.27363
    GAMG:  Solving for p, Initial residual = 0.0998053, Final residual = 0.000728134, No Iterations 6
    time step continuity errors : sum local = 4.26347e-07, global = 1.16708e-20, cumulative = 5.86486e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123776
    GAMG:  Solving for p, Initial residual = 0.0810209, Final residual = 0.00054928, No Iterations 6
    time step continuity errors : sum local = 3.24835e-07, global = 4.55333e-20, cumulative = 5.9104e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120434
    GAMG:  Solving for p, Initial residual = 0.0189497, Final residual = 8.88034e-07, No Iterations 38
    time step continuity errors : sum local = 5.18253e-10, global = 2.24608e-20, cumulative = 5.93286e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
    ExecutionTime = 167.35 s  ClockTime = 167 s
    
    Time = 9.9
    
    Courant Number mean: 0.73339 max: 1.27894
    GAMG:  Solving for p, Initial residual = 0.100093, Final residual = 0.00073128, No Iterations 6
    time step continuity errors : sum local = 4.28756e-07, global = 1.40838e-20, cumulative = 5.94694e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
    GAMG:  Solving for p, Initial residual = 0.0809929, Final residual = 0.000542798, No Iterations 6
    time step continuity errors : sum local = 3.21522e-07, global = -2.66223e-20, cumulative = 5.92032e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
    GAMG:  Solving for p, Initial residual = 0.0189464, Final residual = 8.9418e-07, No Iterations 38
    time step continuity errors : sum local = 5.22767e-10, global = 2.26167e-20, cumulative = 5.94294e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
    ExecutionTime = 169.06 s  ClockTime = 169 s
    
    Time = 10
    
    Courant Number mean: 0.733413 max: 1.27934
    GAMG:  Solving for p, Initial residual = 0.0999826, Final residual = 0.000732193, No Iterations 6
    time step continuity errors : sum local = 4.29802e-07, global = 1.57268e-20, cumulative = 5.95866e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.0810305, Final residual = 0.00054485, No Iterations 6
    time step continuity errors : sum local = 3.22957e-07, global = 2.11478e-20, cumulative = 5.97981e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189583, Final residual = 8.97181e-07, No Iterations 38
    time step continuity errors : sum local = 5.24979e-10, global = 2.66036e-20, cumulative = 6.00642e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 170.82 s  ClockTime = 171 s
    
    End
    


  • CPU型号:AMD-3900X
    系统:Ubuntu18   32g 3000
    OpenFOAM版本:OpenFOAM-7
    8 406.12
    4 382.44
    2 512.83
    1 915.2
    

    为啥3900x 8核反而慢了



  • @kennyadapter 你这个看起来就是带宽跟不上了
    @来瓶热阔落 你看下你8核和4核的最开始时间步的log?类似这种,分别贴一下:

    Time = 0.1
    
    Courant Number mean: 0.733413 max: 1.27933
    GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
    time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
    time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
    time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 173.99 s  ClockTime = 174 s
    


  • @东岳200万网格管道流qDNS并行算力测试 中说:

    东岳老师,那怎么调整带宽呢?有什么方法,或者需要更换什么配件吗?



  • Time = 0.1
    
    Courant Number mean: 0.731999 max: 1.22083
    GAMG:  Solving for p, Initial residual = 0.629705, Final residual = 0.00421935, No Iterations 3
    time step continuity errors : sum local = 7.82336e-06, global = -1.9955e-20, cumulative = -1.9955e-20
    Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114208
    GAMG:  Solving for p, Initial residual = 0.0563421, Final residual = 0.000354153, No Iterations 4
    time step continuity errors : sum local = 9.75747e-07, global = 2.11991e-20, cumulative = 1.24408e-21
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117378
    GAMG:  Solving for p, Initial residual = 0.0152771, Final residual = 8.66768e-07, No Iterations 18
    time step continuity errors : sum local = 2.28999e-09, global = -1.80834e-20, cumulative = -1.68393e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.00011696
    ExecutionTime = 4.61 s  ClockTime = 5 s
    
    Time = 0.2
    
    Courant Number mean: 0.731698 max: 1.1641
    GAMG:  Solving for p, Initial residual = 0.75843, Final residual = 0.00508023, No Iterations 3
    time step continuity errors : sum local = 6.71701e-06, global = 1.79247e-12, cumulative = 1.79247e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119141
    GAMG:  Solving for p, Initial residual = 0.0716405, Final residual = 0.000689194, No Iterations 5
    time step continuity errors : sum local = 5.93336e-07, global = 1.79247e-12, cumulative = 3.58494e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117204
    GAMG:  Solving for p, Initial residual = 0.014621, Final residual = 9.75203e-07, No Iterations 25
    time step continuity errors : sum local = 8.46786e-10, global = 1.79247e-12, cumulative = 5.37741e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116758
    ExecutionTime = 9.1 s  ClockTime = 9 s
    

    上面是4核,下面8核

    Time = 0.1
    
    Courant Number mean: 0.731999 max: 1.22083
    GAMG:  Solving for p, Initial residual = 0.629703, Final residual = 0.00603973, No Iterations 3
    time step continuity errors : sum local = 1.11982e-05, global = 8.06587e-21, cumulative = 8.06587e-21
    Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114193
    GAMG:  Solving for p, Initial residual = 0.0566187, Final residual = 0.00046644, No Iterations 4
    time step continuity errors : sum local = 1.28491e-06, global = -2.9891e-20, cumulative = -2.18251e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117396
    GAMG:  Solving for p, Initial residual = 0.0153571, Final residual = 7.70794e-07, No Iterations 20
    time step continuity errors : sum local = 2.0346e-09, global = -1.94705e-20, cumulative = -4.12956e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000116978
    ExecutionTime = 5.64 s  ClockTime = 6 s
    
    Time = 0.2
    
    Courant Number mean: 0.731698 max: 1.16413
    GAMG:  Solving for p, Initial residual = 0.75841, Final residual = 0.00736995, No Iterations 3
    time step continuity errors : sum local = 9.74458e-06, global = 8.78067e-13, cumulative = 8.78067e-13
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119176
    GAMG:  Solving for p, Initial residual = 0.073429, Final residual = 0.000652992, No Iterations 6
    time step continuity errors : sum local = 5.62596e-07, global = 8.78067e-13, cumulative = 1.75613e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117203
    GAMG:  Solving for p, Initial residual = 0.0147366, Final residual = 9.38466e-07, No Iterations 23
    time step continuity errors : sum local = 8.14733e-10, global = 8.78067e-13, cumulative = 2.6342e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116759
    ExecutionTime = 11.02 s  ClockTime = 11 s
    

  • Linux讲师 OpenFOAM讲师

    CPU:  AMD Opteron(tm) Processor 6376
    OS:   debian 10
    OpenFOAM: 8
    
    # cores   Wall time (s):
    ------------------------
    64 164.29
    32 180.06
    16 305.33
    8 407.37
    4 800.44
    2 1981.19
    1 3383.18
    
    CPU: Intel(R) Xeon(R) Platinum 8175M
    OS:  ubuntu 18.04
    OpenFOAM: 8
    # cores   Wall time (s):
    ------------------------
    48 77.08
    24 99.2
    12 123.14
    6 234.82
    3 366.02
    1 1149.63
    
    
    CPU: Intel(R) Xeon(R) E5-2683
    OS:  ubuntu 14.04
    OpenFOAM: 8
    # cores   Wall time (s):
    ------------------------
    28 224.8
    14 214.99
    8 246.61
    4 383.36
    2 721.88
    1 1361
    


  • @bestucan 8175这个可以按照12468163248这个跑一遍不 跟我那个表格里面对不上不好放进去:huahua:


  • Linux讲师 OpenFOAM讲师

    @东岳 好的,我回去后就试:huahua:


  • Linux讲师 OpenFOAM讲师

    @东岳

    CPU: Intel(R) Xeon(R) Platinum 8175M
    OS:  ubuntu 18.04
    OpenFOAM: 8
    
    # cores   Wall time (s):
    ------------------------
    48 76.67
    32 87
    16 132.63
    8 167.32
    6 234.28
    4 312.67
    2 573.06
    1 1124.36
    


  • @bestucan 已更新 数据挺好的



  • @来瓶热阔落 我最近在测试也发现了这个问题 你换PCG算压力方程试试?但这样跟别的就没有可比性了。但可以看看结果。

    一些CPU确实存在这个问题,目前我没有解决办法,只能加在CPU黑名单里 :135:



  • 
    
    CPU: EPYC 7402 x 2
    OS:  ubuntu 18.04
    OpenFOAM 7
    
    # cores   Wall time (s):
    ------------------------
    48 65.01
    32 68.3
    16 84.09
    8 140.9
    4 283.94
    2 568.89
    




  • @ustbwenwu 多谢,已更新 :140:



  • win10_bash ubuntu18.04
    CPU:  i7-9750H
    RAM: 16G 
    OpenFOAM-6
    # cores   Wall time (s):
    ------------------------
    8 842.92
    4 838.84
    2 846.9
    1 1414.95
    

    感觉有问题啊:143:



  • 心血来潮测一下服务器 为什么会有这样的error 提示? of41能做么

    Reading/calculating face flux field phi
    
    Selecting incompressible transport model Newtonian
    Selecting turbulence model type laminar
    No MRF models present
    
    Creating finite volume options from "constant/fvOptions"
    
    Selecting finite volume options model type meanVelocityForce
    [0] 
    [0] 
    [0] --> FOAM FATAL IO ERROR: 
    [0] keyword meanVelocityForceCoeffs is undefined in dictionary "/home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource"
    [0] 
    [0] file: /home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource from line 20 to line 25.
    [0] 
    [0]     From function const Foam::dictionary& Foam::dictionary::subDict(const Foam::word&) const
    [0]     in file db/dictionary/dictionary.C at line 642.
    [0] [2] 
    [2] 
    [2] --> FOAM FATAL IO ERROR: 
    [2] keyword meanVelocityForceCoeffs is undefined in dictionary "IOstream.momentumSource"
    [2] 
    


  • ubuntu18.04
    CPU: AMD EPYC 7282 16-Core Processor*2
    RAM: 252G 
    OpenFOAM-4.1
    # cores   Wall time (s):
    ------------------------
    32 82.4
    16 121.69
    8 140.26
    4 268.85
    2 498.06
    1 1125.41
    


  • 感谢大家 已更新主表



  • @TINGHAIK 虚拟机的都特别慢 bash的应该也会受影响



  • CPU型号:i7-7700
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    
    4 594.83
    2 666.07
    1 956.33
    

Log in to reply