[DirectVerbs] Implement wait-related interfaces in the dv transport

[pakmarkthub]

Prerequisites:

• PR [DirectVerbs] Implement send-related interfaces in the dv transport #106.

This PR:

• implements all wait-related interfaces in mlx5-dv.cpp.
• introduces gds_poll_cq.
• modifies the test applications to use gds_poll_cq instead of ibv_poll_cq.

Notes:

• With this PR, RC and UD are supported when using the dv transport.
• IBV_WR_SEND is the only opcode that is supported. And it works with only non inline data.
• Support for the exp transport is the same as the original -- i.e., all connection types and opcodes.

Presubmit testing:

• gc11 (dv transport).
• Summit (exp transport).

[protoss1235]

I have question, if this update works, it seems libmp need to update too, because it used old structure of libgdsync

[pakmarkthub]

Hi @hongbilu,

Yes, we need to update libmp too. The patch for libmp is not ready yet, however. Do you want to use libmp also?

[protoss1235]

Hi @hongbilu,

Yes, we need to update libmp too. The patch for libmp is not ready yet, however. Do you want to use libmp also?

yeah，very expect for it which must be cool to support ofed 5.4. BTW, i saw that nvshmem and GPUnetIO library also support gdasync too, it seems a little repeated, and I support this project because I feel it is a offical library only for gdasync

[protoss1235]

@pakmarkthub hi，why not use mlx5dv_devx_obj_create instead of mlx5_dv_create_cq? The "struct mlx5_ifc_create_cq_in_bits" seems disappear