Through direct numerical simulations (DNS), we have studied the effect of irregular multi-scale roughness (triangular elements in 2D and conical in 3D) on heat transport and statistics of the coherent structures in Rayleigh-Benard convection (RBC) for a working fluid of Prandtl number Pr = 0.7. The roughness geometry used in the present work entails a significant number of length scales of roughness elements. Contrary to the previous studies [1], the inclusion of distinct roughness elements in the present work has allowed us to sustain the enhanced Nusselt-Rayleigh number, Nu(Ra), scaling exponent even in the higher Rayleigh range. It has been observed that different roughness elements behave differently at a particular thermal forcing. The distinct flow dynamics in different regions associated with rough surfaces (tip, throat, and valley) reveal how they regulate heat transfer rate at different Ra. The three different irregular roughness configurations, characterized by their maximum heights (5%,10%, and 20% of the cell height) reveal the dependency of onset of enhanced heat flux (exponent) regimes, which is quantified by the critical Ra_c . Taller the roughness elements are, smaller the Ra_c is. While the transformation of double-multiple roll state is responsible for enhanced heat flux in the intermediate and tallest roughness cases, activation of numerous roughness elements enhances Nu in the smallest case. The 3D case in a cubical box also confirms the augmented heat transport properties and altered flow structures due to surface roughness. Increase in the volume fraction of thermal plumes is found to be responsible for augmented heat flux. Also, mean vertical profiles of horizontal velocities reveal that roughness controls the preferred orientation of large-scale rolls.