GPGPU 2023

As GPUs become more and more popular in cloud computing, it is important to address one feature that they lack compared to CPUs: secure execution environment. Currently, users of cloud computing must choose between running their program in a secure environment on CPUs, sacrificing performance and parallelism, or running it on GPUs, sacrificing security. In this talk I will discuss some of the challenges in providing Trusted Execution Environment (TEE) on GPUs. First, I will discuss two secure memory architectures, counter-mode encryption and direct encryption, for GPUs and show that we need to architect secure memory differently from it for CPUs. As GPUs are designed for high-throughput computation, its secure memory needs to deliver high bandwidth. Furthermore, with counter-mode encryption, the memory traffic resulting from the metadata, i.e., the counters, MACs (message-authentication codes), and integrity tree, may cause significant performance degradation, even in the presence of metadata caches. Moreover, the sectored cache structure adopted by GPUs leads to multiple sequential accesses to the same metadata cache line, which necessitates the use of MSHRs (miss-status handling registers) for meta-data caches. Finally, unlike CPUs, separate/partitioned metadata caches perform better than unified metadata caches on GPUs. I will discuss some possible solutions to the above problem. Besides of providing TEEs on GPUs, we must also consider how CPU and GPU TEEs interact. Without co-designing them, we impose a separate memory encryption domain from the host TEE, causing a very substantial slowdown for communicating data from/to the host. I will discuss a flexible GPU memory encryption design that relies on software memory encryption aided by small architecture support. LITE's flexibility allows GPU TEE to be co-designed with CPU to create a unified encryption domain. We show that GPU applications can be adapted to the use of LITE encryption APIs without major changes.

An auto-tuning framework for GPU devices is presented for tuning OpenCL application kernels. The GPU tuner employs multi-objective optimization methodology to improve the performance and power consumption of applications. The proposed approach explores a user defined solution space efficiently comprising of possible tunable algorithmic with the hardware counter variations through code transformations. The methodology targets GPU code tuning situations where performance and energy consumption are critical and an optimal trade-off between performance and energy is unavoidable. The proposed framework, MOKAT is evaluated for 2D Convolution kernel targeting NVIDIA Tesla GPU. Various hardware counter and algorithmic variations such as loop unrolling, caching, workgroup size and memory utilization are employed for evaluation. MOKAT utilizes a non-dominated Genetic Algorithm (GA) with hardware power sensor data for application code transformation through code rewrite and validation. The proposed technique is applicable to both off-line and in-loop tuning of kernels. The final Pareto optimal configuration code utilized around 30% less power and 4% faster execution time. The analysis on the first and last GA generation shows the convergence of optimization, and standard deviation is improved by 45%.

The emerging new applications, such as data mining and graph analysis, demand extra processing power at the hardware level. Conventional static task scheduling is no longer able to meet the requirements of such complicated applications. This inefficiency is a major concern when the application is supposed to run on a Graph- ics Processing Unit (GPU), where millions of instructions should be distributed among a limited number of processing cores. A non-optimal scheduling strategy leads to unfair load distribution among the GPU’s processing cores. Consequently, while busy cores are stalled due to the lack of resources, waiting for their data from the main memory, other cores are idle, waiting for busy cores to complete their tasks. This leads to a significant number of stall and idle cycles, limiting the GPU performance to a large extent. Our study introduces LATOA, a Load-Aware Task Offloading and Adoption method that tackles this problem by reducing both stall and idle cycles. LATOA is the first study moving from static to dynamic task scheduling based on run-time information obtained from the Miss Status Holding Register (MSHR) tables. In LATOA, all processing cores are dynamically tagged with critical, neutral, or relaxed states. Then, irregular warps with low locality properties are detected and offloaded from critical cores (going to the stall state) to relaxed ones (going to the idle state). Based on our experiments, LATOA reduces the number of stall cycles on average by 24% and increases the neutral states on average by 38%. In addi- tion, with negligible hardware overhead, LATOA improves system performance and power efficiency on average by 26% and 7%, respectively.

It is well-known that memory bandwidth is a critical resource in modern GPUs and can be the bottleneck for many workloads. However, high on-chip communication bandwidth is also necessary to ensure that off-chip bandwidth is not bottlenecked by on-chip bandwidth. In this talk, I will explore the challenges of bandwidth challenges internal to the GPU including the on-chip interconnect. In particular, I will re-visit the design of GPU NoC and how bandwidth hierarchy needs to be properly considered to avoid bandwidth bottlenecks in a GPU. The two important characteristics of on-chip communication are bandwidth and latency; I will describe the unique on-chip bandwidth and latency characteristics and their implication on overall performance. In addition, I will describe how high-bandwidth on-chip bandwidth can be exploited for high-bandwidth side-channel attacks.

SYCL is a promising programming model for heterogenous computing across vendors’ devices. In this paper, we study whether SYCL can be applied to an automotive workload and its portability on heterogeneous computing platforms. We explain the automotive benchmark, describe our implementations and optimizations of the benchmark, and evaluate the performance of the benchmarks using SYCL and other programming models on heterogeneous devices. The study also allows us to have a better understanding of portability of the benchmark across these platforms.

GPU architectures have become popular for executing general-purpose programs which rely on having a large number of threads that run concurrently to hide the latency among dependent instructions. This approach has an important cost/overhead in terms of low data locality due to the increased pressure on the memory hierarchy of the many threads being run concurrently and the extra cost of storing and managing the on-chip state of those many threads. This paper presents SOCGPU (Simple Out-of-order Core for GPU), a simple out-of-order execution mechanism that does not require register renaming nor scoreboards. It uses a small Instruction Buffer and a tiny Dependence matrix to keep track of dependencies among instructions and avoid data hazards. Evaluations for an Nvidia Tesla V100-like GPU show that SOCGPU provides a speed-up of up to 2.3 in some machine learning programs and 1.38 on average for a variety of benchmarks, while it reduces energy consumption by 6.5%, with only 2.4% area overhead.

Modern GPUs benefit from a sizable Register File (RF) to provide fine-grained thread switching. As the RF is huge and accessed frequently, it consumes a considerable share of the dynamic energy of the GPU. Designing a large, high-throughput RF with low energy consumption and area for GPUs is challenging. In this paper, an energy-efficient hierarchical RF design for GPUs, called Malekeh, is introduced. Malekeh keeps registers in energy- efficient small caches and maximizes cache efficacy by using light- weight policies and supporting adaptive algorithms. The policies’ effectiveness is improved by leveraging register reuse distance in- formation provided by the compiler as a hint. Malekeh reduces the RF reads by 48.5% and dynamic energy by 29.1%. It also improves performance by 9.6% with a negligible overhead of 0.04% in area.

General Purpose Graphics Processing Units (GPGPU) power most of the top systems in HPC. The total capacity of scratchpad memory has increased by more than $20$x in the last decade. However, existing optimizations for stencil computations using temporal blocking have yet to aggressively exploit the large capacity of scratchpad memory. This work uses the 2D Jacobian 5-point iterative stencil as a case for studying the utilization of the large scratchpad memory. Unlike the existing research that tile the domain in a thread block fashion, we tile the domain so that every single tile is large enough to employ all the available scratchpad memory in the GPU. Hence, we process several time steps inside a single tile before offloading the result back to global memory. The evaluated result shows our performance is comparable to state-of-the-art implementations, yet our implementation is much simpler, and does not need auto-generation of code.

Multi-GPU systems are considered as one of the most promising scalable accelerator systems. There have been several studies that tackled communication and scheduling efficiencies with a small-scale multi-GPU system (mostly with four GPU modules). In this paper, we examine scalability by increasing the number of GPUs. Our observations show that multi-GPU systems are yet to be scalable, mainly due to Non-Uniform Memory Access (NUMA) effects; furthermore, the state-of-the-art aggressive page distribution is one of the main reasons that increase slow remote accesses.