OpenCl: How do I make use of INT4/8? - opencl

Question

I am looking for ways to improve the efficiency of an algorithm based on OpenCl.
Currently I use float and int datatypes on a Radeon VII card. However, a datatype covering numbers between -8 to +7 would be sufficient.
According to following article the Radeon VII achieves peak performance of 53/110 TFlops when restricting to INT8/INT4, which is much higher than with float which is 14 TFlops.
https://www.pcgameshardware.de/Radeon-VII-Grafikkarte-268194/Tests/Benchmark-Review-1274185/2/
So my question is how can I make use of INT8/4 operations? Simply use datatype char instead of int in OpenCl? Since char is the smallest buildin datatype, how can I even use INT4?

Answer 1

For "int8", that is, 8-bit integers, the OpenCL type is indeed char (signed, -128 to +127) or uchar (unsigned, 0 to 255). Not to be confused with the OpenCL type int8, which is a vector of 8 32-bit integers.
For decent performance you may want to use vector versions of these such as char4 or char16, though this should be driven based on your performance measurements, not guessing.
Note that you'll need to be aware of overflow behaviour, and especially for multiplications, you may need to perform intermediate operations on 16-bit values. (short/ushort/short4/ushort16/etc.) OpenCL also provides "saturating" addition and subtraction and a few other helpful integer built-in functions.
I'm not aware of any "native" support for packed 4-bit integer maths in OpenCL or any of the other GPGPU frameworks, or even any extensions. Maybe someone with experience with this can chip in, but my guess is that you'll effectively need to unpack uchar values using bit shifts and masking, perform your operations on uchar values and then pack back into 4-bit nibbles for storing. The speed boost will likely come from the fact that you can safely multiply using 8-bit logic rather than needing 16 bits to catch the overflow.

Answer 2

I made a test with some kernels to see if there is any difference in performance between int8 and char8:
typedef int8 type_msg;
//typedef char8 type_msg;
#define convert_type_msg(x) convert_int8(x)
__kernel void some_operation(__global type_msg *in_buff,
__global type_msg *out_buff)
{
out_buff[get_global_id(0)] = in_buff[get_global_id(0)] +(type_msg)(2);
}
First, to see what happens on GPU I used CodeXL to get the assembler code.
Here is part of the assembler code where int8 is used:
global_load_dwordx4 v[4:7], v[2:3], off
global_load_dwordx4 v[8:11], v[2:3], off inst_offset:16
v_add_co_u32 v0, vcc, s6, v0
v_mov_b32 v2, s7
v_addc_co_u32 v1, vcc, v2, v1, vcc
s_waitcnt vmcnt(0)
v_add_u32 v8, 2, v8
v_add_u32 v9, 2, v9
v_add_u32 v10, 2, v10
v_add_u32 v11, 2, v11
global_store_dwordx4 v[0:1], v[8:11], off inst_offset
v_add_u32 v2, 2, v4
v_add_u32 v3, 2, v5
v_add_u32 v4, 2, v6
v_add_u32 v5, 2, v7
global_store_dwordx4 v[0:1], v[2:5], off
And here is part of the assembler code where char8 is used:
global_load_dwordx2 v[2:3], v[2:3], off
s_waitcnt vmcnt(0)
v_lshlrev_b32 v4, 8, v3 src1_sel:BYTE_3
v_lshrrev_b32 v5, 8, v3
v_add_u32 v6, 2, v3 src1_sel:WORD_1
v_add_u32 v4, 0x00000200, v4
s_movk_i32 s0, 0x00ff
v_lshlrev_b32 v7, 8, v2 src1_sel:BYTE_3
v_add_u32 v5, 2, v5
v_bfi_b32 v4, s0, v6, v4
s_mov_b32 s1, 0x02010004
v_lshrrev_b32 v6, 8, v2
v_add_u32 v8, 2, v2 src1_sel:WORD_1
v_add_u32 v7, 0x00000200, v7
v_add_u32 v3, 2, v3
v_perm_b32 v4, v5, v4, s1
v_add_u32 v5, 2, v6
v_bfi_b32 v6, s0, v8, v7
v_add_co_u32 v0, vcc, s6, v0
v_mov_b32 v7, s7
v_addc_co_u32 v1, vcc, v7, v1, vcc
v_perm_b32 v3, v3, v4, s1
v_add_u32 v2, 2, v2
v_perm_b32 v4, v5, v6, s1
v_perm_b32 v2, v2, v4, s1
global_store_dword v[0:1], v3, off inst_offset:4
global_store_dword v[0:1], v2, off
I am no expert in assembly language, but as far as I can tell, in both cases there are 8 additions carried out using the v_add_u32 operation. Also char8 seems to require more operations like v_perm_b32 and v_bfi_b32. Maybe someone can explain what these are doing.
The only benefit of using char8 seems to be, that less global memory access is needed. E.g. there is only one global_load_dwordx2 access for char8 but 2 global_load_dwordx4 accesses for int8.
Thus in terms of performance maybe char8 is a little bit slower for computational bounded algorithms but faster for memory bounded algorithms.
To verify the analysis I build up a little experiment, where arithmetic is the bottleneck. To make sure that the compiler does not simplify the for-loop too much, I added some branching inside of it.
typedef int8 type_msg;
#define convert_type_msg(x) convert_int8(x)
//typedef char8 type_msg;
//#define convert_type_msg(x) convert_char8(x)
__kernel void some_complex_operation(__global char8 *in_buff,
__global char8 *out_buff)
{
type_msg res = in_buff[get_global_id(0)];
for(int i=0; i<1000000; i++)
{
res += select((type_msg)(-1), (type_msg)(4), res<(type_msg)100);
}
out_buff[get_global_id(0)] =(type_msg) res;
}
On my system the average time (running 100 times) for
int8 is 0.0558 sec
char8 is 0.0754 sec
short8 is 0.0738 sec
long8 is 0.1105 sec
So char8 consumes roughly 35 % more time. This confirms the observation, that in assembly language more instructions are generated for char8. However, some professional explanation for the addtional assembly statements would be nice.