WoolyAI - Hypervise & Maximize GPU Infra

Hi, We’re a small team of OS, virtualization, and ML engineers, and after three years of development, we’re thrilled to launch the beta of our CUDA abstraction layer! We decouple the Kernel Shader execution from applications that use CUDA into a Wooly Abstraction layer. In this abstraction layer, we compile these to a new binary, and Shaders are compiled into a Wooly Instruction Set. At runtime, Kernel Shader launch events initiate a transfer of Shader over the network from a CPU host to a GPU host, where they are recompiled. Their execution is managed by Wooly Server software to achieve maximum GPU resource utilization, isolation between workloads, and cross-compatibility with hardware vendors before being converted to be passed on to the respective GPU hardware runtime and drivers. In principle, the wooly abstraction layer is similar to an Operating System, which sits on top of the hardware and enables the most efficient and reliable execution of multiple workloads. We built a GPU Cloud service(WoolyAi Acceleration Service) using this abstraction layer with "Actual GPU Resources Used" billing, NOT "GPU Time Used" billing. Looking forward to getting your feedback and comments.

Hey @manisha_arora6 congrats on the launch! Very interesting product. I checked out your demo and I have a couple questions: - How does the abstraction layer’s performance compare to native CUDA execution? Have you run any benchmarks showing efficiency, particularly for deep learning training and inference? - Since the system transfers shaders over the network before execution, what is the impact on latency? How does the recompilation process affect real-time workloads, especially for inference-heavy applications? Lastly, good call on billing for usage vs. time! I think that’s the way to go for new serverless frameworks. 👏🏼