o ^iGi@s ddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZejejeZejed gZd d Zed d ZGdddeZddZddddddZddddddZdZ ddZ!ddZ"Gdd d eZ#Gd!d"d"e Z$dS)#N)Path)knobs) GPUTarget) GPUDriver) _allocation)compile_module_from_src)TensorDescriptorincludec sddl}|dkr dSddlddlm}m}m}mmm}Gfdddj } |||||||}z dj }Wn t yLYdSw|g|_||_dd} fd d } ||| | rut| SdS) NrLinux)c_charc_intc_size_tc_void_pc_char_pPOINTERcseZdZdfdfgZdS)z8_find_already_mmapped_dylib_on_linux..DlPhdrInfo dlpi_addr dlpi_nameN)__name__ __module__ __qualname___fields_)rrrU/var/www/html/RAG/RAG_venv/lib/python3.10/site-packages/triton/backends/amd/driver.py DlPhdrInfosrz libc.so.6ics@|jj}tt|}|jvr||tt|dSdS)Nrr) contentsrrosfsdecodenamememmoveminlen)infosizedatarp)ctypeslib_namemax_path_lengthrrcallback3s  z6_find_already_mmapped_dylib_on_linux..callback)platformsystemr&r r r rrr Structure CFUNCTYPECDLLdl_iterate_phdr Exceptionargtypesrestypecreate_string_bufferrr string_at) r'r*r r r rr callback_tr/pathr)r)rrr&r'r(r$_find_already_mmapped_dylib_on_linuxs(     r7c sdtjj}r|rtj|r|Std|dt}|r7tj|r-|Std|dg}tj tj t d}tj|rM|S| |ddl }|}|}|jrf|g|}|D]}tj |dd}tj|r}|S| |qhtd }|r|d D]} tj | } tj| r| S| | qtd } | rtj | d} tj| r| S| | z%td d g} | rtj | d} tj| r| WS| | Wn tjtfyYnwtd}|rtj |d}tj|r|S| |tddgjdd}fdd|D}|D]}tj|r9|S| |q,tj d}tj|rP|S| |tdd|)Nzlibamdhip64.sozTRITON_LIBHIP_PATH 'z' does not point to a valid zmemory mapped 'z'' in process does not point to a valid librtorchLD_LIBRARY_PATH:HIP_PATH hipconfigz--path ROCM_PATHz/sbin/ldconfigz-pignore)errorscs&g|]}|r|dqS))stripendswithsplit).0liner'rr s&z2_get_path_to_hip_runtime_dylib..z/opt/rocm/lib/zcannot locate z after attempted paths )ramd libhip_pathrCrr6exists RuntimeErrorr7joindirname__file__appendsitegetsitepackagesgetusersitepackagesENABLE_USER_SITEgetenvrD subprocess check_outputdecoderBCalledProcessErrorFileNotFoundError splitlines)env_libhip_path mmapped_pathpaths local_librQ site_packages user_siter6env_ld_library_pathdf env_hip_path hip_lib_pathhip_root env_rocm_path rocm_lib_pathlibslocsloccommon_install_pathrrGr_get_path_to_hip_runtime_dylibAs                  rncs$eZdZfddZddZZS)HIPUtilscs"t|dstt|||_|jS)Ninstance)hasattrsuperro__new__rp)cls __class__rrrss zHIPUtils.__new__cCsLt}ttjtd}|d|d}t|dt d}|j |_ |j |_ dS)Nzdriver.cz/*py_libhip_search_path*/r hip_utilssrcr include_dirs) rnrrr6rMrN read_textreplacerrz load_binaryget_device_properties)selfrJrymodrrr__init__s  zHIPUtils.__init__)rrrrsr __classcell__rrrurros rocCs8|ddkrdSdddddddd d d d d d d d d |S)Nr*hipDeviceptr_tint8_tint16_tint32_tint64_tuint8_tuint16_tuint32_tuint64_tdouble)i1i8i16i32i64u1u8u16u32u64fp16bf16fp32f32fp64r)tyrrr ty_to_cpps( rrrr)rrrrr pack_fp16 pack_bf16 pack_fp32 pack_fp64 piiiKKOOOOOcsdd}fddfddfddd d t||D}d fd d |D}t|}dt|}ttt|d}dd t|D}t |dkrfdddd| Dnd }g}| D]&\}} | dkrwqn| t vr| t | d|qn| t | d|qnd|} g} | D].\}} | ddkr| d|dq| t vr| d|dq| dkr| d|qdd | D} t} ttt |}dd | D}| d| dd| d t | dkrd| nd d!d|d"|d#|d$d%fd&d | Dd'|d(|d)d%| d*d+d,d | Dd-t | dkrJdd| nd d.}|S)/NcSsg}|D]S}t|trR|drR|dd}td|}|d|td|D]}|dq,|dt|D]}|d q=t|D]}|dqIq||q|S) N tensordesc,rztensordesc<([^[>]*)rrrr) isinstancestr startswithcountrematchgrouprPrange) signatureoutputsigndimdtype_rrr_expand_signatures        z(make_launcher.._expand_signaturecst|tr dt|S|S)Nr)rtuplerMmap)r)_serialize_signaturerrrs z+make_launcher.._serialize_signaturecsJt|trdt|}d|dS|ddkrdS|dkr!dSt|S)Nr[]rrz PyObject* constexprrrrMrrrval_extracted_typerrrs   z&make_launcher.._extracted_typec sdt|trdt|}d|dS|ddkrdS|dkr!dSdd d d d d ddddd t|S)N()rrOrrclbhiLBHIK) rlongrrrrrrrrrr format_ofrrr s(    z make_launcher..format_ofcSi|]\}}||qSrr)rEidxsrrr z!make_launcher..rcsg|]}|qSrr)rErrrrrH sz!make_launcher..rcSrrr)rErrrrrr$rr, css|] \}}d|VqdS)z&_argNrrErrrrr %sz make_launcher..rz argrptr_infoz.dev_ptr_arg_storagec Ss<g|]\}}|tvrt|d|dt|d|dqS) _argz _storage = z(_argz);)FLOAT_STORAGE_TYPEFLOAT_PACK_FUNCTIONrrrrrH:s "cSs"g|] \}}|dkrd|qS)rz&argrrrrrrHDs"z&global_scratchz&profile_scratcha\ #define __HIP_PLATFORM_AMD__ #include #include #include #include #include #include // The list of paths to search for the HIP runtime library. The caller Python // code should substitute the search path placeholder. static const char *hipLibSearchPaths[] = {"a"}; // The list of HIP dynamic library symbols and their signature we are interested // in this file. #define HIP_SYMBOL_LIST(FOR_EACH_ERR_FN, FOR_EACH_STR_FN) \ FOR_EACH_STR_FN(hipGetLastError) \ FOR_EACH_STR_FN(hipGetErrorString, hipError_t hipError) \ FOR_EACH_ERR_FN(hipModuleLaunchKernel, hipFunction_t f, \ unsigned int gridDimX, unsigned int gridDimY, \ unsigned int gridDimZ, unsigned int blockDimX, \ unsigned int blockDimY, unsigned int blockDimZ, \ unsigned int sharedMemBytes, hipStream_t stream, \ void **kernelParams, void **extra) \ FOR_EACH_ERR_FN(hipModuleLaunchCooperativeKernel, hipFunction_t f, \ unsigned int gridDimX, unsigned int gridDimY, \ unsigned int gridDimZ, unsigned int blockDimX, \ unsigned int blockDimY, unsigned int blockDimZ, \ unsigned int sharedMemBytes, hipStream_t stream, \ void **kernelParams, void **extra) \ FOR_EACH_ERR_FN(hipPointerGetAttribute, void *data, \ hipPointer_attribute attribute, hipDeviceptr_t ptr) // The HIP symbol table for holding resolved dynamic library symbols. struct HIPSymbolTable { #define DEFINE_EACH_ERR_FIELD(hipSymbolName, ...) \ hipError_t (*hipSymbolName)(__VA_ARGS__); #define DEFINE_EACH_STR_FIELD(hipSymbolName, ...) \ const char *(*hipSymbolName)(__VA_ARGS__); HIP_SYMBOL_LIST(DEFINE_EACH_ERR_FIELD, DEFINE_EACH_STR_FIELD) }; static struct HIPSymbolTable hipSymbolTable; bool initSymbolTable() { // Use the HIP runtime library loaded into the existing process if it exits. void *lib = dlopen("libamdhip64.so", RTLD_NOLOAD); // Otherwise, go through the list of search paths to dlopen the first HIP // driver library. if (!lib) { int n = sizeof(hipLibSearchPaths) / sizeof(hipLibSearchPaths[0]); for (int i = 0; i < n; ++i) { void *handle = dlopen(hipLibSearchPaths[i], RTLD_LAZY | RTLD_LOCAL); if (handle) { lib = handle; } } } if (!lib) { PyErr_SetString(PyExc_RuntimeError, "cannot open libamdhip64.so"); return false; } typedef hipError_t (*hipGetProcAddress_fn)( const char *symbol, void **pfn, int hipVersion, uint64_t hipFlags, hipDriverProcAddressQueryResult *symbolStatus); hipGetProcAddress_fn hipGetProcAddress; dlerror(); // Clear existing errors const char *error = NULL; *(void **)&hipGetProcAddress = dlsym(lib, "hipGetProcAddress"); error = dlerror(); if (error) { PyErr_SetString(PyExc_RuntimeError, "cannot query 'hipGetProcAddress' from libamdhip64.so"); dlclose(lib); return false; } // Resolve all symbols we are interested in. int hipVersion = HIP_VERSION; uint64_t hipFlags = 0; hipDriverProcAddressQueryResult symbolStatus; hipError_t status = hipSuccess; #define QUERY_EACH_FN(hipSymbolName, ...) status = hipGetProcAddress(#hipSymbolName, (void **)&hipSymbolTable.hipSymbolName, hipVersion, hipFlags, &symbolStatus); if (status != hipSuccess) { PyErr_SetString(PyExc_RuntimeError, "cannot get address for '" #hipSymbolName "' from libamdhip64.so"); dlclose(lib); return false; } HIP_SYMBOL_LIST(QUERY_EACH_FN, QUERY_EACH_FN) return true; } static inline void gpuAssert(hipError_t code, const char *file, int line) { if (code != HIP_SUCCESS) { const char* prefix = "Triton Error [HIP]: "; const char* str = hipSymbolTable.hipGetErrorString(code); char err[1024] = {0}; snprintf(err, 1024, "%s Code: %d, Messsage: %s", prefix, code, str ); PyErr_SetString(PyExc_RuntimeError, err); } } #define HIP_CHECK(ans) { gpuAssert((ans), __FILE__, __LINE__); } static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, hipStream_t stream, hipFunction_t function, hipDeviceptr_t profile_scratchz>) { hipDeviceptr_t global_scratch = 0; void *params[] = { z }; if (gridX*gridY*gridZ > 0 && launch_cooperative_grid) { HIP_CHECK(hipSymbolTable.hipModuleLaunchCooperativeKernel(function, gridX, gridY, gridZ, z*num_warps, 1, 1, shared_memory, stream, params, 0)); return; } if (gridX*gridY*gridZ > 0) { HIP_CHECK(hipSymbolTable.hipModuleLaunchKernel(function, gridX, gridY, gridZ, ae *num_warps, 1, 1, shared_memory, stream, params, 0)); } } typedef struct _DevicePtrInfo { hipDeviceptr_t dev_ptr; bool valid; } DevicePtrInfo; static PyObject* data_ptr_str = NULL; static inline DevicePtrInfo getPointer(PyObject *obj, int idx) { DevicePtrInfo ptr_info; hipError_t status = hipSuccess; ptr_info.dev_ptr = 0; ptr_info.valid = true; if (PyLong_Check(obj)) { ptr_info.dev_ptr = (hipDeviceptr_t)PyLong_AsUnsignedLongLong(obj); return ptr_info; } if (obj == Py_None) { // valid nullptr return ptr_info; } PyObject *ret = PyObject_CallMethodNoArgs(obj, data_ptr_str); if (!ret) { PyErr_SetString(PyExc_TypeError, "Pointer argument must be either uint64 or have data_ptr method"); ptr_info.valid = false; goto cleanup; } if (!PyLong_Check(ret)) { PyErr_SetString(PyExc_TypeError, "data_ptr method of Pointer object must return 64-bit int"); ptr_info.valid = false; goto cleanup; } ptr_info.dev_ptr = (hipDeviceptr_t)PyLong_AsUnsignedLongLong(ret); if (!ptr_info.dev_ptr) goto cleanup; uint64_t dev_ptr; status = hipSymbolTable.hipPointerGetAttribute(&dev_ptr, HIP_POINTER_ATTRIBUTE_DEVICE_POINTER, ptr_info.dev_ptr); if (status == hipErrorInvalidValue) { PyErr_Format(PyExc_ValueError, "Pointer argument (at %d) cannot be accessed from Triton (cpu tensor?)", idx); ptr_info.valid = false; // Clear and ignore HIP error (void)hipSymbolTable.hipGetLastError(); } ptr_info.dev_ptr = (hipDeviceptr_t)dev_ptr; cleanup: Py_DECREF(ret); return ptr_info; } static uint16_t pack_fp16(double f) { uint16_t result; // from https://github.com/python/pythoncapi-compat/blob/5e317108f872c904eb726cb8d560dcadbdf88a72/pythoncapi_compat.h#L482-L492 #if 0x030600B1 <= PY_VERSION_HEX && PY_VERSION_HEX <= 0x030B00A1 && !defined(PYPY_VERSION) _PyFloat_Pack2(f, (unsigned char*)&result, 1); #else PyFloat_Pack2(f, (char*)&result, 1); #endif return result; } static uint16_t pack_bf16(double f) { float f32 = (float)f; uint32_t u32 = *(uint32_t*)&f32; return (uint16_t)(u32 >> 16); } static uint32_t pack_fp32(double f) { float f32 = (float)f; return *(uint32_t*)&f32; } static uint64_t pack_fp64(double f) { return *(uint64_t*)&f; } static PyObject* launch(PyObject* self, PyObject* args) { int gridX, gridY, gridZ; uint64_t _stream; uint64_t _function; int launch_cooperative_grid; PyObject *profile_scratch_obj = NULL; PyObject *launch_enter_hook = NULL; PyObject *launch_exit_hook = NULL; PyObject *kernel_metadata = NULL; PyObject *launch_metadata = NULL;  cs$g|]\}}|d|dqS)r; rrrrrrHs$z if(!PyArg_ParseTuple(args, "a,", &launch_cooperative_grid, &gridX, &gridY, &gridZ, &_stream, &_function, &profile_scratch_obj, &kernel_metadata, &launch_metadata, &launch_enter_hook, &launch_exit_hook z)) { return NULL; } a // extract kernel metadata int num_warps, num_ctas, shared_memory, clusterDimX, clusterDimY, clusterDimZ; if (!PyArg_ParseTuple(kernel_metadata, "iiiiii", &num_warps, &num_ctas, &shared_memory, &clusterDimX, &clusterDimY, &clusterDimZ)) { return NULL; } // extract launch metadata if (launch_enter_hook != Py_None){ PyObject* ret = PyObject_CallOneArg(launch_enter_hook, launch_metadata); if (!ret) return NULL; Py_DECREF(ret); } hipDeviceptr_t profile_scratch = 0; if (profile_scratch_obj != Py_None) { DevicePtrInfo profile_scratch_info = getPointer(profile_scratch_obj, -1); if (!profile_scratch_info.valid) { return NULL; } profile_scratch = profile_scratch_info.dev_ptr; } // raise exception asap rc Ss>g|]\}}|ddkrd|d|d|d|d ndqS)rrzDevicePtrInfo ptr_infoz = getPointer(_argrz); if (!ptr_infoz.valid) return NULL;rrrrrrrH=s>z; _launch(gridX, gridY, gridZ, num_warps, num_ctas, launch_cooperative_grid, clusterDimX, clusterDimY, clusterDimZ, shared_memory, (hipStream_t)_stream, (hipFunction_t)_function, (hipDeviceptr_t)profile_scratcha); if(launch_exit_hook != Py_None){ PyObject* ret = PyObject_CallOneArg(launch_exit_hook, launch_metadata); if (!ret) return NULL; Py_DECREF(ret); } if(PyErr_Occurred()) { return NULL; } Py_RETURN_NONE; } static PyMethodDef ModuleMethods[] = { {"launch", launch, METH_VARARGS, "Entry point for all kernels with this signature"}, {NULL, NULL, 0, NULL} // sentinel }; static struct PyModuleDef ModuleDef = { PyModuleDef_HEAD_INIT, "__triton_launcher", NULL, //documentation -1, //size ModuleMethods }; PyMODINIT_FUNC PyInit___triton_launcher(void) { if (!initSymbolTable()) { return NULL; } PyObject *m = PyModule_Create(&ModuleDef); if(m == NULL) { return NULL; } data_ptr_str = PyUnicode_InternFromString("data_ptr"); if(data_ptr_str == NULL) { return NULL; } PyModule_AddFunctions(m, ModuleMethods); return m; } ) enumeratevaluesrM_BASE_ARGS_FORMATrlistfilterboolrDr!itemsrrPrrnr) constantsr warp_sizer args_formatformat args_list arg_decl_listrr arg_declsinternal_args_listfloat_storage_declsrJparamsryr)rrrr make_launchers   ,     tvx|VWZ^wx%rcsfdd}|S)zN Replace all tensor descriptors with the base ptr, shape, and strides cs|dtt}|ttd}g}|D]%}t|tr4||jg|j|j|jdk|j|jq| |qg||RS)Nnan) r!rrrextendbaseshapestridespaddingrP)args meta_argsraw_kernel_args final_argsarglauncherrrinnerrs 2 z,wrap_handle_tensor_descriptor..innerr)rrrrrwrap_handle_tensor_descriptorms rc@seZdZddZddZdS) HIPLaunchercstdrjnt}fddfdd|D}ddjD}t|||jtdtd}t d d | D}|rFt |j n|j |_ |j |_ |j|_|j|_dS) Nrcst|tr jj|fS|SN)rrfn arg_namesindex)x)ryrrsz&HIPLauncher.__init__..csi|] \}}||qSrrrErvalue)arg_idxrrrsz(HIPLauncher.__init__..cSrrrr rrrrr__triton_launcherrxcss$|] }t|to |dVqdS)rN)rrr)rErrrrrs"z'HIPLauncher.__init__..)rqrdictrrrrrrzanyrrlaunchlaunch_cooperative_gridprofile_scratch_sizeprofile_scratch_align)rrymetadatarrrhas_tensor_desc_argr)r ryrrs  zHIPLauncher.__init__c sHfdd}||j|jtj}|j|j||g|RdS)Ncs4|dkr}||}|}|||SdSNr)get)r#align allocator grid_size alloc_sizealloc_fngridXgridYgridZstreamrrallocate_scratchs   z.HIPLauncher.__call__..allocate_scratch)rrr_profile_allocatorrr) rrrrr functionrr!profile_scratchrrr__call__s  $zHIPLauncher.__call__N)rrrrr%rrrrrs rcsjeZdZfddZddZeddZdedefd d Zd d Z d dZ ddZ ddZ ddZ ZS) HIPDrivercstt|_t|_dSr)rrrroutilsr launcher_cls)rrurrrs  zHIPDriver.__init__cCsddl}|jSr)r9cudarr9rrrget_device_interfaceszHIPDriver.get_device_interfacecCs6zddl}|jo|jjduWStyYdSw)NrF)r9r) is_availableversionhip ImportError)r9rrr is_actives  zHIPDriver.is_activerreturncCst|Sr)r)rrrrrmap_python_to_cpp_typesz HIPDriver.map_python_to_cpp_typecCsB|}|j|}tjjp|d}|d}td|dd|S)NarchwarpSizer.r;r)get_current_devicer'r~rruntime override_archrrD)rdevicedevice_propertiesr3rrrrget_current_targets  zHIPDriver.get_current_targetcCsddl}|d|S)Nrr))r9r8r5r*rrrget_active_torch_devicesz!HIPDriver.get_active_torch_devicecCsddlm}|S)Nr)do_bench)triton.testingr<)rr<rrrget_benchmarkers zHIPDriver.get_benchmarkercCs&ddl}d}|jt|d|jddS)Nrir))rr8)r9emptyint)rr9 cache_sizerrrget_empty_cache_for_benchmarksz'HIPDriver.get_empty_cache_for_benchmarkcCs |dSr)zero_)rcacherrr clear_caches zHIPDriver.clear_cache)rrrrr+ staticmethodr0rr2r:r;r>rCrFrrrrurr&s  r&)% functoolsrrVrpathlibrtritonrtriton.backends.compilerrtriton.backends.driverrtriton.runtimertriton.runtime.buildrtriton.tools.tensor_descriptorrr6rNrealpathrOrMrzr7 lru_cachernobjectrorrrrrrrr&rrrrsL       0 _