200万网格管道流qDNS并行算力测试



  • CPU型号:E5-2699 v4
    系统:linux
    OpenFOAM版本:OpenFOAM-1906
    44 173.99
    32 170.82
    16 193.78
    8 194.74
    4 313.11
    2 585.66
    1 1146.46
    
    

    为什么我的44核心会更慢呢,东岳老师?我最近在算一个200w网格的overset算例,算了半个多月了,算的还是很慢,有什么更好的解决方法吗?@东岳


  • 管理员

    @kennyadapter 我最近也发现有这个问题,主要在压力方程的求解器上,你看下你的44核与32核的每一步的压力方程迭代次数是否一致?



  • 这个算例算了多少时间步?



  • @东岳
    这是我44核心的log文件的最后几个时间步:之前的迭代次数也差不多:

    
    Time = 9.8
    
    Courant Number mean: 0.733368 max: 1.27353
    GAMG:  Solving for p, Initial residual = 0.0998056, Final residual = 0.00082233, No Iterations 6
    time step continuity errors : sum local = 4.80691e-07, global = 1.99762e-20, cumulative = 1.59798e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123775
    GAMG:  Solving for p, Initial residual = 0.0809999, Final residual = 0.000584259, No Iterations 6
    time step continuity errors : sum local = 3.45394e-07, global = 5.63939e-20, cumulative = 1.60362e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120435
    GAMG:  Solving for p, Initial residual = 0.0189678, Final residual = 9.4483e-07, No Iterations 39
    time step continuity errors : sum local = 5.51633e-10, global = 3.45318e-20, cumulative = 1.60707e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
    ExecutionTime = 170.51 s  ClockTime = 171 s
    
    Time = 9.9
    
    Courant Number mean: 0.73339 max: 1.27886
    GAMG:  Solving for p, Initial residual = 0.100098, Final residual = 0.000824461, No Iterations 6
    time step continuity errors : sum local = 4.82558e-07, global = 8.66441e-20, cumulative = 1.61574e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
    GAMG:  Solving for p, Initial residual = 0.0809731, Final residual = 0.000577791, No Iterations 6
    time step continuity errors : sum local = 3.42103e-07, global = 4.07868e-20, cumulative = 1.61981e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
    GAMG:  Solving for p, Initial residual = 0.0189619, Final residual = 9.31189e-07, No Iterations 39
    time step continuity errors : sum local = 5.44568e-10, global = 7.2382e-20, cumulative = 1.62705e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
    ExecutionTime = 172.22 s  ClockTime = 172 s
    
    Time = 10
    
    Courant Number mean: 0.733413 max: 1.27933
    GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
    time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
    time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
    time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 173.99 s  ClockTime = 174 s
    
    End
    
    

    这是我32核心的:

    
    Time = 9.8
    
    Courant Number mean: 0.733369 max: 1.27363
    GAMG:  Solving for p, Initial residual = 0.0998053, Final residual = 0.000728134, No Iterations 6
    time step continuity errors : sum local = 4.26347e-07, global = 1.16708e-20, cumulative = 5.86486e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123776
    GAMG:  Solving for p, Initial residual = 0.0810209, Final residual = 0.00054928, No Iterations 6
    time step continuity errors : sum local = 3.24835e-07, global = 4.55333e-20, cumulative = 5.9104e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120434
    GAMG:  Solving for p, Initial residual = 0.0189497, Final residual = 8.88034e-07, No Iterations 38
    time step continuity errors : sum local = 5.18253e-10, global = 2.24608e-20, cumulative = 5.93286e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638
    ExecutionTime = 167.35 s  ClockTime = 167 s
    
    Time = 9.9
    
    Courant Number mean: 0.73339 max: 1.27894
    GAMG:  Solving for p, Initial residual = 0.100093, Final residual = 0.00073128, No Iterations 6
    time step continuity errors : sum local = 4.28756e-07, global = 1.40838e-20, cumulative = 5.94694e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823
    GAMG:  Solving for p, Initial residual = 0.0809929, Final residual = 0.000542798, No Iterations 6
    time step continuity errors : sum local = 3.21522e-07, global = -2.66223e-20, cumulative = 5.92032e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471
    GAMG:  Solving for p, Initial residual = 0.0189464, Final residual = 8.9418e-07, No Iterations 38
    time step continuity errors : sum local = 5.22767e-10, global = 2.26167e-20, cumulative = 5.94294e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674
    ExecutionTime = 169.06 s  ClockTime = 169 s
    
    Time = 10
    
    Courant Number mean: 0.733413 max: 1.27934
    GAMG:  Solving for p, Initial residual = 0.0999826, Final residual = 0.000732193, No Iterations 6
    time step continuity errors : sum local = 4.29802e-07, global = 1.57268e-20, cumulative = 5.95866e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.0810305, Final residual = 0.00054485, No Iterations 6
    time step continuity errors : sum local = 3.22957e-07, global = 2.11478e-20, cumulative = 5.97981e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189583, Final residual = 8.97181e-07, No Iterations 38
    time step continuity errors : sum local = 5.24979e-10, global = 2.66036e-20, cumulative = 6.00642e-18
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 170.82 s  ClockTime = 171 s
    
    End
    


  • CPU型号:AMD-3900X
    系统:Ubuntu18   32g 3000
    OpenFOAM版本:OpenFOAM-7
    8 406.12
    4 382.44
    2 512.83
    1 915.2
    

    为啥3900x 8核反而慢了


  • 管理员

    @kennyadapter 你这个看起来就是带宽跟不上了
    @来瓶热阔落 你看下你8核和4核的最开始时间步的log?类似这种,分别贴一下:

    Time = 0.1
    
    Courant Number mean: 0.733413 max: 1.27933
    GAMG:  Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6
    time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387
    GAMG:  Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6
    time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508
    GAMG:  Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39
    time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971
    ExecutionTime = 173.99 s  ClockTime = 174 s
    


  • @东岳200万网格管道流qDNS并行算力测试 中说:

    东岳老师,那怎么调整带宽呢?有什么方法,或者需要更换什么配件吗?



  • Time = 0.1
    
    Courant Number mean: 0.731999 max: 1.22083
    GAMG:  Solving for p, Initial residual = 0.629705, Final residual = 0.00421935, No Iterations 3
    time step continuity errors : sum local = 7.82336e-06, global = -1.9955e-20, cumulative = -1.9955e-20
    Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114208
    GAMG:  Solving for p, Initial residual = 0.0563421, Final residual = 0.000354153, No Iterations 4
    time step continuity errors : sum local = 9.75747e-07, global = 2.11991e-20, cumulative = 1.24408e-21
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117378
    GAMG:  Solving for p, Initial residual = 0.0152771, Final residual = 8.66768e-07, No Iterations 18
    time step continuity errors : sum local = 2.28999e-09, global = -1.80834e-20, cumulative = -1.68393e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.00011696
    ExecutionTime = 4.61 s  ClockTime = 5 s
    
    Time = 0.2
    
    Courant Number mean: 0.731698 max: 1.1641
    GAMG:  Solving for p, Initial residual = 0.75843, Final residual = 0.00508023, No Iterations 3
    time step continuity errors : sum local = 6.71701e-06, global = 1.79247e-12, cumulative = 1.79247e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119141
    GAMG:  Solving for p, Initial residual = 0.0716405, Final residual = 0.000689194, No Iterations 5
    time step continuity errors : sum local = 5.93336e-07, global = 1.79247e-12, cumulative = 3.58494e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117204
    GAMG:  Solving for p, Initial residual = 0.014621, Final residual = 9.75203e-07, No Iterations 25
    time step continuity errors : sum local = 8.46786e-10, global = 1.79247e-12, cumulative = 5.37741e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116758
    ExecutionTime = 9.1 s  ClockTime = 9 s
    

    上面是4核,下面8核

    Time = 0.1
    
    Courant Number mean: 0.731999 max: 1.22083
    GAMG:  Solving for p, Initial residual = 0.629703, Final residual = 0.00603973, No Iterations 3
    time step continuity errors : sum local = 1.11982e-05, global = 8.06587e-21, cumulative = 8.06587e-21
    Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114193
    GAMG:  Solving for p, Initial residual = 0.0566187, Final residual = 0.00046644, No Iterations 4
    time step continuity errors : sum local = 1.28491e-06, global = -2.9891e-20, cumulative = -2.18251e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117396
    GAMG:  Solving for p, Initial residual = 0.0153571, Final residual = 7.70794e-07, No Iterations 20
    time step continuity errors : sum local = 2.0346e-09, global = -1.94705e-20, cumulative = -4.12956e-20
    Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000116978
    ExecutionTime = 5.64 s  ClockTime = 6 s
    
    Time = 0.2
    
    Courant Number mean: 0.731698 max: 1.16413
    GAMG:  Solving for p, Initial residual = 0.75841, Final residual = 0.00736995, No Iterations 3
    time step continuity errors : sum local = 9.74458e-06, global = 8.78067e-13, cumulative = 8.78067e-13
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119176
    GAMG:  Solving for p, Initial residual = 0.073429, Final residual = 0.000652992, No Iterations 6
    time step continuity errors : sum local = 5.62596e-07, global = 8.78067e-13, cumulative = 1.75613e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117203
    GAMG:  Solving for p, Initial residual = 0.0147366, Final residual = 9.38466e-07, No Iterations 23
    time step continuity errors : sum local = 8.14733e-10, global = 8.78067e-13, cumulative = 2.6342e-12
    Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116759
    ExecutionTime = 11.02 s  ClockTime = 11 s
    

  • 版主 副教授

    CPU:  AMD Opteron(tm) Processor 6376
    OS:   debian 10
    OpenFOAM: 8
    
    # cores   Wall time (s):
    ------------------------
    64 164.29
    32 180.06
    16 305.33
    8 407.37
    4 800.44
    2 1981.19
    1 3383.18
    
    CPU: Intel(R) Xeon(R) Platinum 8175M
    OS:  ubuntu 18.04
    OpenFOAM: 8
    # cores   Wall time (s):
    ------------------------
    48 77.08
    24 99.2
    12 123.14
    6 234.82
    3 366.02
    1 1149.63
    
    
    CPU: Intel(R) Xeon(R) E5-2683
    OS:  ubuntu 14.04
    OpenFOAM: 8
    # cores   Wall time (s):
    ------------------------
    28 224.8
    14 214.99
    8 246.61
    4 383.36
    2 721.88
    1 1361
    

  • 管理员

    @bestucan 8175这个可以按照12468163248这个跑一遍不 跟我那个表格里面对不上不好放进去:huahua:


  • 版主 副教授

    @东岳 好的,我回去后就试:huahua:


  • 版主 副教授

    @东岳

    CPU: Intel(R) Xeon(R) Platinum 8175M
    OS:  ubuntu 18.04
    OpenFOAM: 8
    
    # cores   Wall time (s):
    ------------------------
    48 76.67
    32 87
    16 132.63
    8 167.32
    6 234.28
    4 312.67
    2 573.06
    1 1124.36
    

  • 管理员

    @bestucan 已更新 数据挺好的


  • 管理员

    @来瓶热阔落 我最近在测试也发现了这个问题 你换PCG算压力方程试试?但这样跟别的就没有可比性了。但可以看看结果。

    一些CPU确实存在这个问题,目前我没有解决办法,只能加在CPU黑名单里 :135:



  • 
    
    CPU: EPYC 7402 x 2
    OS:  ubuntu 18.04
    OpenFOAM 7
    
    # cores   Wall time (s):
    ------------------------
    48 65.01
    32 68.3
    16 84.09
    8 140.9
    4 283.94
    2 568.89
    



  • 管理员

    @ustbwenwu 多谢,已更新 :140:



  • win10_bash ubuntu18.04
    CPU:  i7-9750H
    RAM: 16G 
    OpenFOAM-6
    # cores   Wall time (s):
    ------------------------
    8 842.92
    4 838.84
    2 846.9
    1 1414.95
    

    感觉有问题啊:143:



  • 心血来潮测一下服务器 为什么会有这样的error 提示? of41能做么

    Reading/calculating face flux field phi
    
    Selecting incompressible transport model Newtonian
    Selecting turbulence model type laminar
    No MRF models present
    
    Creating finite volume options from "constant/fvOptions"
    
    Selecting finite volume options model type meanVelocityForce
    [0] 
    [0] 
    [0] --> FOAM FATAL IO ERROR: 
    [0] keyword meanVelocityForceCoeffs is undefined in dictionary "/home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource"
    [0] 
    [0] file: /home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource from line 20 to line 25.
    [0] 
    [0]     From function const Foam::dictionary& Foam::dictionary::subDict(const Foam::word&) const
    [0]     in file db/dictionary/dictionary.C at line 642.
    [0] [2] 
    [2] 
    [2] --> FOAM FATAL IO ERROR: 
    [2] keyword meanVelocityForceCoeffs is undefined in dictionary "IOstream.momentumSource"
    [2] 
    


  • ubuntu18.04
    CPU: AMD EPYC 7282 16-Core Processor*2
    RAM: 252G 
    OpenFOAM-4.1
    # cores   Wall time (s):
    ------------------------
    32 82.4
    16 121.69
    8 140.26
    4 268.85
    2 498.06
    1 1125.41
    

  • 管理员

    感谢大家 已更新主表


  • 管理员

    @TINGHAIK 虚拟机的都特别慢 bash的应该也会受影响



  • CPU型号:i7-7700
    系统:linux系统
    OpenFOAM版本:OpenFOAM-8
    
    4 594.83
    2 666.07
    1 956.33
    


  • CPU型号: Intel(R) Xeon(R) Gold 6248R
    系统:ubuntu 18.04
    版本:openfoam-8
    
    48 98.12
    32 96.14
    16 128.01
    8 155.18
    4 304.95
    2 542.82
    1 1068.49
    

    没关超线程,物理核48 感觉不合理啊哈哈 明天再弄一次:136:



  • 中午在服务器上测试了一下这个算例,发现算例受核间通信的影响确实很大,在核心数较高时计算时长的收益很小甚至负收益==
    以及8164是老cpu了吧,单核计算能力捉急。。

    CPU型号: Intel(R) Xeon(R) Platinum 8164 CPU @ 2GHz (8)
    系统:    Ubuntu 18.04.5 LTS
    内存:  791160068 kB
    OpenFOAM版本:OpenFOAM-6
    
    192 149.49
    128 51.77
    64 48.97
    32 64.36
    16 111.44
    8 153.07
    4 338.46
    2 611.9
    1 1249.7```

  • 管理员

    8164也不算特别老啊,你这个几个节点?



  • @李东岳 是一个8路服务器,一共208颗物理核心



  • 
    CPU型号: Intel(R) Xeon(R) Gold 6248R
    系统:ubuntu 18.04
    版本:openfoam-8
    
    48 86.08
    32 93.05
    16 125.13
    8 153.34
    4 287.14
    2 524.41
    1 1013.26
    
    

    正常了。。。



  • CPU型号: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
    系统:Linux mint 19.3
    版本:openfoam-6
    
    # cores   Wall time (s):
    ------------------------
    8 552.63
    4 623.67
    2 814.52
    1 1419.05
    
    

  • 管理员

    @huiCfd 额 惨惨惨 :zoule:



  • @李东岳 这台电脑的CPU有点年头了,虽然支持4通道,但是插了两条内存。



  • CPU型号:AMD Athlon X4 860K
    系统:Ubuntu 20.04 (WSL2)
    OpenFOAM版本:OpenFOAM-8
    4 1226.66
    2 1528.62
    1 2310.36
    


  • @stickyrice 我来垫个底:xiezuoye:


  • 管理员

    @huiCfd 楼上比你的还老的来啦 :papa:

    哈哈 不过 @stickyrice WSL2可能对性能也有影响



  • CPU型号:AMD Ryzen 3700X
    系统:Ubuntu 20.04 (WSL2)
    OpenFOAM版本:OpenFOAM-8
    8 391.95
    4 424.99
    2 474.24
    1 817.24

  • 管理员

    @stickyrice wsl2的数据不好横向对比 这个对计算速度有影响 具体多少不清楚 从我虚拟机的体验来看 这个200万网格的算例 在我的服务器上 虚拟机要慢3倍



  • @李东岳 对比表格里的真系统+3900X,我这个WSL2+3700X的单核、双核还有八核都快一点,似乎计算速度打折不明显呢:134: WSL2是不是可以归为虚拟机呢


  • 管理员

    @stickyrice 我查了下这个3900x 3700x 桌面王者啊!单核性能屌炸了。

    wsl我装过一次,编译时间好久,后来没搞了.. https://www.cfd-china.com/topic/4257/是不是有人用wsl跑openfoam?_=1608513399016

    你是怎么安装的openfoam,编译安装?



  • @李东岳 装OpenFOAM是先把两个zip下好了再用cp命令挪到Linux文件系统里,按您的教程编译安装。晚点我单独写个详细的帖请大家看看:xiezuoye:



  • @李东岳 CPU型号:

    Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz   2.39 GHz  (2 个处理器)
    系统:虚拟机
    OpenFOAM版本:OpenFOAM-8
    16 244.9
    8 240.72
    4 378.28
    2 659.48
    1 1245.63
    

    另外一个问题,就是在OpenFOAM-2.1.1下面运行不来对吗


  • 管理员

    @Yu_Tian 多谢分享。
    2.1.1版本太老了,需要对算例做一些改动。



  • @李东岳 李老师,我在用一个求解器,也是找到的别人的代码,是用的2.1.1版本的openfoam,如果想移植到7,或者8版本里,都需要做哪些改动?之前有没有相关的案例可以参考


  • 管理员

    @Yu_Tian 最简单的就是你直接用211 http://dyfluid.com/docs/install.html 最小面有个22x的 不知道能不能用 如果做移植 不知道你那面代码多少 工作量是有一些的 还得看你代码熟练程度

    我建议直接用211



  • CPU: Intel(R) Core(TM) i9-9900K @ 3.60GHz
    RAM: 16GB
    OS: Virtual machine
    OpenFOAM Version: OpenFOAM-8
    8 502.74
    4 539.87
    2 642.91
    1 1015.98



  • @李东岳 不会设置黑色背景,好尴尬:143:



  • 12核是 12 6 3 1吗?


  • 版主 副教授

    @Jacob-Z 1 2 4 8 12。 按2的指数增长,除非到极限


Log in to reply
 


CFD中文网 | 东岳流体学术 | 东岳流体商业 | 吉ICP备20003622号-1