IB host chaining 三节点方案运行 openfoam 效果如何？
-
众所周知，IB网卡咸鱼上100G和10G价格大差不差，但是100G IB 交换机就不好搞了，咱也买不起一整个机柜，插不满交换机，想问下MCX516A或MCX566A这种双口100G网卡，三个节点组个环网，效果怎么样？理论讲单节点总共也有200G带宽了，不知道会不会成为性能瓶颈。既然都上IB卡了，也不能用7302这种700块钱的CPU了，上个7642，货源充足，咸鱼上挑够6块不成问题。手头的工作站加块4口万兆网卡，做管理节点。这样就得到一个4节点集群，顶多5万块钱。这套配置相比超算中心200G IB交换机相连的三个节点性能如何，相比100G交换机呢？有没有土豪赞助点儿经费？
-
What if I ran mpi program over a ring-topology infiniband network?
Running an MPI (Message Passing Interface) program over a ring-topology InfiniBand network could potentially offer some benefits and challenges as well.
Reduced Latency: InfiniBand is known for its low-latency and high-bandwidth capabilities. This would result in a faster communication between the nodes compared to Ethernet based networks.
Enhanced Performance: The high speed offered by InfiniBand could potentially enhance the performance of the MPI program, especially if it involves significant inter-node communication.
Scalability: The ring topology would allow easy scalability. As the nodes increase, the topology would remain unchanged. All nodes would be equally distant from each other without the need to add more switches or links.
Network Efficiency: The ring topology would ensure efficient data transmission. The data packets could travel in either direction on the ring to reach their destination, making the network more efficient.
Fault Tolerance: The ring topology provides a certain level of fault tolerance. If one node or connection fails, the data can travel in the opposite direction to reach its destination.
However, the main challenge would be potential bottleneck situations. While InfiniBand is designed to minimize latency, the ring topology could introduce a bottleneck situation if one node on the ring becomes overwhelmed with the data. This could potentially slow down the entire network, affecting the performance of the MPI program.
Another challenge could be configuring and managing the ring topology network. It might require significant expertise and effort to set up and optimally configure the network, especially if the nodes are physically distributed.
In conclusion, running an MPI program over a ring-topology Infiniband network could provide benefits in terms of speed, efficiency, and flexibility. However, it might also bring potential challenges related to network bottleneck and complexity of network configuration and management.
-
A further question. Is there any research that figured out the performance difference between the star-topology infiniband network with switch and the ring-topology infiniband network without switch for mpi applications.
There has definitely been a significant amount of research aimed at comparing different network topologies, including star and ring, for MPI applications. However, finding specific studies that compare star-topology InfiniBand network with switch and the ring-topology InfiniBand network without switch may be challenging because the actual comparison can depend largely on specific use cases, type of MPI applications, and other factors such as network size, traffic volume, and fault tolerance requirements.
That said, both topologies have their unique advantages.
Star Topology: The central switch can effectively manage and control network traffic, ensuring optimal data routing and reducing chances of network congestion. It also makes the network easier to debug and manage. However, it introduces a single point of failure i.e., if the central switch fails, the entire network may collapse.
Ring Topology: It might offer better performance for smaller networks due to shorter travel paths. However, the absence of a switch could create potential data routing and bottleneck issues, especially for larger networks or heavy network traffic. On the other hand, it does not have a single point of failure and offers better fault tolerance compared to star topology.
While it’s not precisely what you asked for, a good reference to start might be “Performance Analysis of InfiniBand FDR and EDR on Scientific User Cases” by Hakkarinen and Knuutila (2015). Although it does not compare star and ring topologies, it does provide a valuable analysis of InfiniBand performance for scientific applications.
For a detailed and specific comparison, you might need to conduct a custom research or benchmark testing based on your specific MPI applications and network environments.
-
Great! Here's more details. I perform CFD over infiniband network and wonder whether the ring topology is able to achiever better performance and save the cost of infiniband switch if there are 4 computer nodes in the network.
CFD, or Computational Fluid Dynamics, is a complex set of computations that often require high-speed, low-latency inter-node communication. This is particularly the case when using MPI (Message Passing Interface), which is a common parallel programming model used for scientific computation.
The InfiniBand network, being a high-speed, low-latency network, is well suited to this task. However, the topology can play a significant role in the performance.
For a small size of 4 nodes, the ring topology could potentially achieve somewhat similar performance to the star topology given that the path length between most nodes would be relatively small. An additional benefit would indeed be the cost savings from not purchasing an InfiniBand switch, which can be expensive.
However, there are a few considerations to keep in mind:
Performance: While a ring topology might be able to handle the communication needs, it might not always offer the best performance or scalability. A switch in a star network provides intelligent routing, which ensures that data packets find the most efficient path.
Failure Tolerance: If a node in the ring fails, it's harder to bypass it. In contrast, in a star network, because all nodes are connected to the switch, a single node failure won't bring down the communication of all other nodes.
Ease of Troubleshooting: Troubleshooting is also easier in a star network setup as you only need to check connectivity from each node to the switch. In a ring, you would need to check the connectivity between each pair of nodes.
In summary, while you might find that a ring topology could work in a 4-node InfiniBand network, you'd certainly want to perform some tests or simulations with your specific CFD workload to make sure the performance measures up to your expectations, and that you're comfortable with the trade-offs.
-
Fine. Give me more details about performance. Reducing switch is expected to reduce communication delay which is benifit to mpi. To what extend will the smaller delay improve the performance when running CFD application. Is this another advantage of ring topology infiniband network?
The performance improvement due to the reduced communication delay in a ring topology compared to a star topology depends largely on the specifics of your CFD application.
With the InfiniBand protocols, the primary factor that contributes to the delay is not the time it takes for the packet to traverse the physical network (which is typically less than a few microseconds), but the time it takes for the packet to get through the software stack. The overhead of software in sending and receiving messages can be a significant component of the total latency.
MPI-based CFD applications often require large scale, synchronized computing. If your application often needs to exchange small amounts of data between all nodes (such as boundary values or synchronization barriers), then minor latency improvements might accumulate and contribute to significant overall performance improvement.
However, if your CFD application is structured so that most of the time is spent in computation, and communication is relatively infrequent (typical for methods where each node can work on a discrete part of the problem for much of the time), then the improvement might be marginal.
Furthermore, if your CFD application is using collective communication, like MPI_Bcast or MPI_Reduce, or if it's using non-blocking sends/receives, then it may be intelligent enough to take advantage of the network topology to minimise communication delays.
So, it can potentially be an advantage of ring topology, but the magnitude of this advantage is highly dependent on the specifics of your CFD code. Thorough testing will be necessary to calculate the potential benefit in your own case.
