200万网格并行算力测试(OpenFOAM版本)
-
这是一个管道流动的qDNS算例,基于OpenFOAM。使用pisoFoam就可以运行。
如何测试CPU算力
-
下载 https://www.jianguoyun.com/p/DeabU1wQ9s3ZBhjc7LgD 解压,放到任意的文件夹下
-
打开Allrun文件,将其中的
for i in 4 2 1; do
改成你的处理器数量,比如你的处理器是32核,就改成32 16 8 4 2 1
,注意,上下有两处,都要改一下 -
直接在终端输入
./Allrun
即可运行,什么都不用管了。终端不要关闭。 -
看终端的结果,应该类似下图,其中的数字就是结果。
-
回帖,告诉我你的CPU型号和数据,我整理在下表中
系统 CPU 1 2 4 8 16 32 48 64 128 真系统 7742 1370 717 337 205 128 103 62 真系统 8164 (机架式) 1249 611 338 153 111 64 48.9 51 真系统 2699-v4 1146 585 313 194 193 170 173 真系统 6376 3383 1981 800 407 305 180 164 真系统 7502 - - 254 135 100 65 59 真系统 7542 1128 760 272 136 107 79 77 真系统 8175M 1124 573 312 167 132 87 76 真系统 7402 - 568 283 140 84 68 65 真系统 6258R 1091 541 303 150 133 89 87 真系统 6248R 1068 542 304 155 128 96 98 真系统 5218 1247 637 316 205 214 172 真系统 6133(user1) 1258 860 437 329 388 358 真系统 6133(user2) 1410 624 336 183 160 86 真系统 7282 1125 498 268 140 121 82 真系统 7302 1074 503 261 136 101 66 真系统 6130 1241 622 310 157 121 91 真系统 6240R 1245 659 378 240 244 真系统 2680V4 1240 661 328 185 156 138(28核) 真系统 2683V4 - - - - 182 163 真系统 2680V3 1274 648 352 196 168 真系统 2630V3 1453 668 327 209 165 真系统 2680V2 1145 615 350 204 198 真系统 4110 1361 948 500 395 467 真系统 3900X 915 512 382 406 真系统 2620v4 1419 814 623 552 WSL2 3700X 817 474 424 391 WSL2 5950X 615 402 382 340 虚拟机 8400 1114 864 861 900 WSL 11900k 823 488 456 443 虚拟机 5820k 1117 691 542 虚拟机 8750H(user1) 1060 883 910 虚拟机 8750H(user2) 1380 1109 963 虚拟机 苹果M1 830 512 359 虚拟机 2500U 1310 894 877 云图
云图结果类似,结果曾经跟JCP一篇文章对比过,完全符合。更详细的关于算法的可参考 链接文本
-
-
回帖范例:
CPU型号:5900k 系统:linux系统 OpenFOAM版本:OpenFOAM-7 16 400 8 442 4 542 2 691 1 1119
- 注意,一些处理器可能不是整数,那就这样也行:
40 32 16 8 4 2 1
,优先保障有32 16 8 4 2 1..
- 注意,一些处理器可能不是整数,那就这样也行:
-
CPU型号:AMD 霄龙 7742 系统:linux系统 OpenFOAM版本:OpenFOAM-v1912 64 62.23 32 103.27 16 128.03 8 205.02 4 337.93 2 717.1 (不是最佳状态) 1 1370.22(不是最佳状态)
-
@maoyanjun_dut 谢谢!
-
@东岳 持续关注中,排排队,看看哪个最能打
-
CPU型号:i7-8750H 系统:虚拟机 OpenFOAM版本:OpenFOAM-6 4 920.13 2 1098.04 1 572.75
为什么单核的时候反而最快?
-
CPU型号:Intel(R) Xeon (R) Gold 5218 (2) 系统:Linux系统 OpenFOAM版本: OpenFOAM-6 32 172.83 16 214 8 205.85 4 316.65 2 637.1 1 1247.31
-
多谢各位,
@merlinizlf 你重新跑一下试试?或者你看看结果都正确不?
-
好的,我重新再算一遍
-
沾一沾大佬算例的福气
-
CPU型号:i7-8750H 系统:虚拟机 OpenFOAM版本:OpenFOAM-6 4 910.12 2 883.78 1 1060.35
重新跑的结果,这次除了虚拟机什么都没运行。。。
希望能对统计有点用。。。 -
这回结果看起来还好
-
CPU型号:霄龙7302 系统:linux系统 OpenFOAM版本:OpenFOAM-7 32 66.5 16 101.16 8 135.76 4 260.99 2 503.16 1 1074.03
-
CPU型号:E5-2680V3 系统:linux系统 OpenFOAM版本:OpenFOAM-8 16 168.45 8 196.13 4 352.66 2 648.47 1 1274.33
CPU型号:E5-2680V2 系统:linux系统 OpenFOAM版本:OpenFOAM-8 16 198.1 8 204.31 4 350.34 2 615.33 1 1145.8
-
@战气凌霄 多谢
CPU型号:金牌6130 系统:linux系统 OpenFOAM版本:OpenFOAM-7 32 91.22 16 121.43 8 157.92 4 310.89 2 622.44 1 1241.62
-
Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz MemTotal: 262566620 kB CentOS Linux release 7.3.1611 (Core) Using: OpenFOAM-6 40 86.58 32 86.77 16 160.35 8 183.99 4 336.65 2 624.18 1 1410.07
-
多谢 @huningdong ,目前有2个6133的数据,你这个效果不错
-
@东岳 在 200万网格管道流qDNS并行算力测试 中说:
多谢 @huningdong ,目前有2个6133的数据,你这个效果不错
不知道为什么 40和32 比较 完全就没什么差距呀
我再跑一下试试 -
@东岳 是不是把内存配置也写上更容易比较?6133的结果看起来应该是内存条数量不一样
-
@战气凌霄 在 200万网格管道流qDNS并行算力测试 中说:
@东岳 是不是把内存配置也写上更容易比较?6133的结果看起来应该是内存条数量不一样
会不会是内存通道的影响?
猜测 32和40 核心 比较结果 差别不大 是因为 32和用的通道数和40用的一样 所以 限制了计算性能的进一步提升? -
@huningdong 您的和另外一个6133的数据比起来,您的计算速度更快,可能是您把内存通道配满了吧? 至于32核和40核的结果,我觉得确实有可能是因为内存带宽跑满了,不过也不排除进程多了后进程间通信拖累?
补充一下我的数据:CPU型号:E5-2680V3,双路,8x8G DDR4-2133内存 系统:linux系统 OpenFOAM版本:OpenFOAM-8 24 139.36 16 168.45 8 196.13 4 352.66 2 648.47 1 1274.33 CPU型号:E5-2680V2,双路,8x8G DDR3-1600内存 系统:linux系统 OpenFOAM版本:OpenFOAM-8 20 180.27 16 198.1 8 204.31 4 350.34 2 615.33 1 1145.8
-
CPU型号:E5-2699 v4 系统:linux OpenFOAM版本:OpenFOAM-1906 44 173.99 32 170.82 16 193.78 8 194.74 4 313.11 2 585.66 1 1146.46
为什么我的44核心会更慢呢,东岳老师?我最近在算一个200w网格的overset算例,算了半个多月了,算的还是很慢,有什么更好的解决方法吗?@东岳
-
@kennyadapter 我最近也发现有这个问题,主要在压力方程的求解器上,你看下你的44核与32核的每一步的压力方程迭代次数是否一致?
-
这个算例算了多少时间步?
-
@东岳
这是我44核心的log文件的最后几个时间步:之前的迭代次数也差不多:Time = 9.8 Courant Number mean: 0.733368 max: 1.27353 GAMG: Solving for p, Initial residual = 0.0998056, Final residual = 0.00082233, No Iterations 6 time step continuity errors : sum local = 4.80691e-07, global = 1.99762e-20, cumulative = 1.59798e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123775 GAMG: Solving for p, Initial residual = 0.0809999, Final residual = 0.000584259, No Iterations 6 time step continuity errors : sum local = 3.45394e-07, global = 5.63939e-20, cumulative = 1.60362e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120435 GAMG: Solving for p, Initial residual = 0.0189678, Final residual = 9.4483e-07, No Iterations 39 time step continuity errors : sum local = 5.51633e-10, global = 3.45318e-20, cumulative = 1.60707e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638 ExecutionTime = 170.51 s ClockTime = 171 s Time = 9.9 Courant Number mean: 0.73339 max: 1.27886 GAMG: Solving for p, Initial residual = 0.100098, Final residual = 0.000824461, No Iterations 6 time step continuity errors : sum local = 4.82558e-07, global = 8.66441e-20, cumulative = 1.61574e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823 GAMG: Solving for p, Initial residual = 0.0809731, Final residual = 0.000577791, No Iterations 6 time step continuity errors : sum local = 3.42103e-07, global = 4.07868e-20, cumulative = 1.61981e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471 GAMG: Solving for p, Initial residual = 0.0189619, Final residual = 9.31189e-07, No Iterations 39 time step continuity errors : sum local = 5.44568e-10, global = 7.2382e-20, cumulative = 1.62705e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674 ExecutionTime = 172.22 s ClockTime = 172 s Time = 10 Courant Number mean: 0.733413 max: 1.27933 GAMG: Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6 time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6 time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39 time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 173.99 s ClockTime = 174 s End
这是我32核心的:
Time = 9.8 Courant Number mean: 0.733369 max: 1.27363 GAMG: Solving for p, Initial residual = 0.0998053, Final residual = 0.000728134, No Iterations 6 time step continuity errors : sum local = 4.26347e-07, global = 1.16708e-20, cumulative = 5.86486e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123776 GAMG: Solving for p, Initial residual = 0.0810209, Final residual = 0.00054928, No Iterations 6 time step continuity errors : sum local = 3.24835e-07, global = 4.55333e-20, cumulative = 5.9104e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120434 GAMG: Solving for p, Initial residual = 0.0189497, Final residual = 8.88034e-07, No Iterations 38 time step continuity errors : sum local = 5.18253e-10, global = 2.24608e-20, cumulative = 5.93286e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119638 ExecutionTime = 167.35 s ClockTime = 167 s Time = 9.9 Courant Number mean: 0.73339 max: 1.27894 GAMG: Solving for p, Initial residual = 0.100093, Final residual = 0.00073128, No Iterations 6 time step continuity errors : sum local = 4.28756e-07, global = 1.40838e-20, cumulative = 5.94694e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000123823 GAMG: Solving for p, Initial residual = 0.0809929, Final residual = 0.000542798, No Iterations 6 time step continuity errors : sum local = 3.21522e-07, global = -2.66223e-20, cumulative = 5.92032e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120471 GAMG: Solving for p, Initial residual = 0.0189464, Final residual = 8.9418e-07, No Iterations 38 time step continuity errors : sum local = 5.22767e-10, global = 2.26167e-20, cumulative = 5.94294e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119674 ExecutionTime = 169.06 s ClockTime = 169 s Time = 10 Courant Number mean: 0.733413 max: 1.27934 GAMG: Solving for p, Initial residual = 0.0999826, Final residual = 0.000732193, No Iterations 6 time step continuity errors : sum local = 4.29802e-07, global = 1.57268e-20, cumulative = 5.95866e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.0810305, Final residual = 0.00054485, No Iterations 6 time step continuity errors : sum local = 3.22957e-07, global = 2.11478e-20, cumulative = 5.97981e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189583, Final residual = 8.97181e-07, No Iterations 38 time step continuity errors : sum local = 5.24979e-10, global = 2.66036e-20, cumulative = 6.00642e-18 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 170.82 s ClockTime = 171 s End
-
CPU型号:AMD-3900X 系统:Ubuntu18 32g 3000 OpenFOAM版本:OpenFOAM-7 8 406.12 4 382.44 2 512.83 1 915.2
为啥3900x 8核反而慢了
-
@kennyadapter 你这个看起来就是带宽跟不上了
@来瓶热阔落 你看下你8核和4核的最开始时间步的log?类似这种,分别贴一下:Time = 0.1 Courant Number mean: 0.733413 max: 1.27933 GAMG: Solving for p, Initial residual = 0.0999699, Final residual = 0.000818608, No Iterations 6 time step continuity errors : sum local = 4.79761e-07, global = 8.13808e-20, cumulative = 1.63519e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00012387 GAMG: Solving for p, Initial residual = 0.081007, Final residual = 0.000576888, No Iterations 6 time step continuity errors : sum local = 3.41793e-07, global = 5.45899e-20, cumulative = 1.64065e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000120508 GAMG: Solving for p, Initial residual = 0.0189675, Final residual = 9.02577e-07, No Iterations 39 time step continuity errors : sum local = 5.28191e-10, global = 1.77208e-20, cumulative = 1.64242e-17 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.00011971 ExecutionTime = 173.99 s ClockTime = 174 s
-
@东岳 在 200万网格管道流qDNS并行算力测试 中说:
东岳老师,那怎么调整带宽呢?有什么方法,或者需要更换什么配件吗?
-
Time = 0.1 Courant Number mean: 0.731999 max: 1.22083 GAMG: Solving for p, Initial residual = 0.629705, Final residual = 0.00421935, No Iterations 3 time step continuity errors : sum local = 7.82336e-06, global = -1.9955e-20, cumulative = -1.9955e-20 Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114208 GAMG: Solving for p, Initial residual = 0.0563421, Final residual = 0.000354153, No Iterations 4 time step continuity errors : sum local = 9.75747e-07, global = 2.11991e-20, cumulative = 1.24408e-21 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117378 GAMG: Solving for p, Initial residual = 0.0152771, Final residual = 8.66768e-07, No Iterations 18 time step continuity errors : sum local = 2.28999e-09, global = -1.80834e-20, cumulative = -1.68393e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.00011696 ExecutionTime = 4.61 s ClockTime = 5 s Time = 0.2 Courant Number mean: 0.731698 max: 1.1641 GAMG: Solving for p, Initial residual = 0.75843, Final residual = 0.00508023, No Iterations 3 time step continuity errors : sum local = 6.71701e-06, global = 1.79247e-12, cumulative = 1.79247e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119141 GAMG: Solving for p, Initial residual = 0.0716405, Final residual = 0.000689194, No Iterations 5 time step continuity errors : sum local = 5.93336e-07, global = 1.79247e-12, cumulative = 3.58494e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117204 GAMG: Solving for p, Initial residual = 0.014621, Final residual = 9.75203e-07, No Iterations 25 time step continuity errors : sum local = 8.46786e-10, global = 1.79247e-12, cumulative = 5.37741e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116758 ExecutionTime = 9.1 s ClockTime = 9 s
上面是4核,下面8核
Time = 0.1 Courant Number mean: 0.731999 max: 1.22083 GAMG: Solving for p, Initial residual = 0.629703, Final residual = 0.00603973, No Iterations 3 time step continuity errors : sum local = 1.11982e-05, global = 8.06587e-21, cumulative = 8.06587e-21 Pressure gradient source: uncorrected Ubar = 0.13349, pressure gradient = 0.000114193 GAMG: Solving for p, Initial residual = 0.0566187, Final residual = 0.00046644, No Iterations 4 time step continuity errors : sum local = 1.28491e-06, global = -2.9891e-20, cumulative = -2.18251e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000117396 GAMG: Solving for p, Initial residual = 0.0153571, Final residual = 7.70794e-07, No Iterations 20 time step continuity errors : sum local = 2.0346e-09, global = -1.94705e-20, cumulative = -4.12956e-20 Pressure gradient source: uncorrected Ubar = 0.133489, pressure gradient = 0.000116978 ExecutionTime = 5.64 s ClockTime = 6 s Time = 0.2 Courant Number mean: 0.731698 max: 1.16413 GAMG: Solving for p, Initial residual = 0.75841, Final residual = 0.00736995, No Iterations 3 time step continuity errors : sum local = 9.74458e-06, global = 8.78067e-13, cumulative = 8.78067e-13 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000119176 GAMG: Solving for p, Initial residual = 0.073429, Final residual = 0.000652992, No Iterations 6 time step continuity errors : sum local = 5.62596e-07, global = 8.78067e-13, cumulative = 1.75613e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000117203 GAMG: Solving for p, Initial residual = 0.0147366, Final residual = 9.38466e-07, No Iterations 23 time step continuity errors : sum local = 8.14733e-10, global = 8.78067e-13, cumulative = 2.6342e-12 Pressure gradient source: uncorrected Ubar = 0.1335, pressure gradient = 0.000116759 ExecutionTime = 11.02 s ClockTime = 11 s
-
CPU: AMD Opteron(tm) Processor 6376 OS: debian 10 OpenFOAM: 8 # cores Wall time (s): ------------------------ 64 164.29 32 180.06 16 305.33 8 407.37 4 800.44 2 1981.19 1 3383.18
CPU: Intel(R) Xeon(R) Platinum 8175M OS: ubuntu 18.04 OpenFOAM: 8 # cores Wall time (s): ------------------------ 48 77.08 24 99.2 12 123.14 6 234.82 3 366.02 1 1149.63
CPU: Intel(R) Xeon(R) E5-2683 OS: ubuntu 14.04 OpenFOAM: 8 # cores Wall time (s): ------------------------ 28 224.8 14 214.99 8 246.61 4 383.36 2 721.88 1 1361
-
@bestucan 8175这个可以按照12468163248这个跑一遍不 跟我那个表格里面对不上不好放进去
-
@东岳 好的,我回去后就试
-
@东岳
CPU: Intel(R) Xeon(R) Platinum 8175M OS: ubuntu 18.04 OpenFOAM: 8 # cores Wall time (s): ------------------------ 48 76.67 32 87 16 132.63 8 167.32 6 234.28 4 312.67 2 573.06 1 1124.36
-
@bestucan 已更新 数据挺好的
-
@来瓶热阔落 我最近在测试也发现了这个问题 你换PCG算压力方程试试?但这样跟别的就没有可比性了。但可以看看结果。
一些CPU确实存在这个问题,目前我没有解决办法,只能加在CPU黑名单里
-
CPU: EPYC 7402 x 2 OS: ubuntu 18.04 OpenFOAM 7 # cores Wall time (s): ------------------------ 48 65.01 32 68.3 16 84.09 8 140.9 4 283.94 2 568.89
-
@东岳
-
@ustbwenwu 多谢,已更新
-
win10_bash ubuntu18.04 CPU: i7-9750H RAM: 16G OpenFOAM-6 # cores Wall time (s): ------------------------ 8 842.92 4 838.84 2 846.9 1 1414.95
感觉有问题啊
-
心血来潮测一下服务器 为什么会有这样的error 提示? of41能做么
Reading/calculating face flux field phi Selecting incompressible transport model Newtonian Selecting turbulence model type laminar No MRF models present Creating finite volume options from "constant/fvOptions" Selecting finite volume options model type meanVelocityForce [0] [0] [0] --> FOAM FATAL IO ERROR: [0] keyword meanVelocityForceCoeffs is undefined in dictionary "/home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource" [0] [0] file: /home/local.njord/maosui/benchMark/core_8/constant/fvOptions.momentumSource from line 20 to line 25. [0] [0] From function const Foam::dictionary& Foam::dictionary::subDict(const Foam::word&) const [0] in file db/dictionary/dictionary.C at line 642. [0] [2] [2] [2] --> FOAM FATAL IO ERROR: [2] keyword meanVelocityForceCoeffs is undefined in dictionary "IOstream.momentumSource" [2]
-
ubuntu18.04 CPU: AMD EPYC 7282 16-Core Processor*2 RAM: 252G OpenFOAM-4.1 # cores Wall time (s): ------------------------ 32 82.4 16 121.69 8 140.26 4 268.85 2 498.06 1 1125.41
-
感谢大家 已更新主表
-
@TINGHAIK 虚拟机的都特别慢 bash的应该也会受影响
-
CPU型号:i7-7700 系统:linux系统 OpenFOAM版本:OpenFOAM-8 4 594.83 2 666.07 1 956.33
-
CPU型号: Intel(R) Xeon(R) Gold 6248R 系统:ubuntu 18.04 版本:openfoam-8 48 98.12 32 96.14 16 128.01 8 155.18 4 304.95 2 542.82 1 1068.49
没关超线程,物理核48 感觉不合理啊哈哈 明天再弄一次
-
中午在服务器上测试了一下这个算例,发现算例受核间通信的影响确实很大,在核心数较高时计算时长的收益很小甚至负收益==
以及8164是老cpu了吧,单核计算能力捉急。。CPU型号: Intel(R) Xeon(R) Platinum 8164 CPU @ 2GHz (8) 系统: Ubuntu 18.04.5 LTS 内存: 791160068 kB OpenFOAM版本:OpenFOAM-6 192 149.49 128 51.77 64 48.97 32 64.36 16 111.44 8 153.07 4 338.46 2 611.9 1 1249.7```
-
8164也不算特别老啊,你这个几个节点?
-
@李东岳 是一个8路服务器,一共208颗物理核心
-
CPU型号: Intel(R) Xeon(R) Gold 6248R 系统:ubuntu 18.04 版本:openfoam-8 48 86.08 32 93.05 16 125.13 8 153.34 4 287.14 2 524.41 1 1013.26
正常了。。。
-
CPU型号: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz 系统:Linux mint 19.3 版本:openfoam-6 # cores Wall time (s): ------------------------ 8 552.63 4 623.67 2 814.52 1 1419.05