ggml-cpu: add RVV vec dot kernels for quantization types

[taimur-10x]

Summary

This PR adds RVV vector dot kernels for a number of quantization types.

Key Changes

• Added the following RVV kernels:

Kernel	VLEN
ggml_vec_dot_iq1_s_q8_K	256
ggml_vec_dot_iq1_m_q8_K	256
ggml_vec_dot_iq2_s_q8_K	128, 256
ggml_vec_dot_iq3_s_q8_K	256
ggml_vec_dot_tq1_0_q8_K	256
ggml_vec_dot_tq2_0_q8_K	256

Testing

Kernels were functionally tested through test-quantize-fns for 128-bit and 256-bit VLENs.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256) with llama-bench.

IQ1_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	3.08	10.23
Tinyllama 1.1B	64	3.08	10.21
Tinyllama 1.1B	128	3.09	10.27
Tinyllama 1.1B	256	3.07	10.22
Tinyllama 1.1B	512	3.09	10.24

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	2.44	7.08
Tinyllama 1.1B	16	2.53	7.04
Tinyllama 1.1B	32	2.65	7.14
Tinyllama 1.1B	64	2.58	7.11
Tinyllama 1.1B	100	2.54	7.11

IQ1_M

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	1.96	8.01
Tinyllama 1.1B	64	1.95	8.05
Tinyllama 1.1B	128	1.96	8.04
Tinyllama 1.1B	256	1.96	8.05
Tinyllama 1.1B	512	1.94	7.98

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	1.71	5.62
Tinyllama 1.1B	16	1.73	5.59
Tinyllama 1.1B	32	1.73	5.64
Tinyllama 1.1B	64	1.72	5.63
Tinyllama 1.1B	100	1.73	5.65

IQ2_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	1.17
Tinyllama 1.1B	64	7.57	1.16
Tinyllama 1.1B	128	8.78	1.14
Tinyllama 1.1B	256	8.57	1.2
Tinyllama 1.1B	512	8.68	1.95

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	1.18
Tinyllama 1.1B	16	3.45	1.02
Tinyllama 1.1B	32	3.25	1.06
Tinyllama 1.1B	64	3.27	1.06
Tinyllama 1.1B	100	3.15	1.04

IQ3_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	1.19
Tinyllama 1.1B	64	7.57	1.25
Tinyllama 1.1B	128	8.78	1.21
Tinyllama 1.1B	256	8.57	1.18
Tinyllama 1.1B	512	8.68	1.12

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	1.22
Tinyllama 1.1B	16	3.45	1.14
Tinyllama 1.1B	32	3.25	1.13
Tinyllama 1.1B	64	3.27	1.13
Tinyllama 1.1B	100	3.15	1.12

TQ1_0

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	2.71
Tinyllama 1.1B	64	7.57	2.72
Tinyllama 1.1B	128	8.78	2.75
Tinyllama 1.1B	256	8.57	2.63
Tinyllama 1.1B	512	8.68	2.68

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	2.50
Tinyllama 1.1B	16	3.45	2.41
Tinyllama 1.1B	32	3.25	2.46
Tinyllama 1.1B	64	3.27	2.51
Tinyllama 1.1B	100	3.15	2.37

TQ2_0

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	4.37
Tinyllama 1.1B	64	7.57	3.89
Tinyllama 1.1B	128	8.78	3.81
Tinyllama 1.1B	256	8.57	3.81
Tinyllama 1.1B	512	8.68	3.79

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	4.22
Tinyllama 1.1B	16	3.45	3.60
Tinyllama 1.1B	32	3.25	3.51
Tinyllama 1.1B	64	3.27	3.16
Tinyllama 1.1B	100	3.15	3.11

Future Work

Subsequent PRs plan to extend existing RVV kernels for quantization types to other VLENs.

[taimur-10x]

@luhenry, @xctan, could this be reviewed please? Thank you.