豆豆友情提示：这是一个非官方 GitHub 代理镜像，主要用于网络测试或访问加速。请勿在此进行登录、注册或处理任何敏感信息。进行这些操作请务必访问官方网站 github.com。 Raw 内容也通过此代理提供。
Skip to content

Fix JIT cache race condition with multi-process compilation #302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Gregory-Pereira wants to merge 1 commit into
base: main
Choose a base branch
from
+33 −0
Open

Fix JIT cache race condition with multi-process compilation #302

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions csrc/jit/compiler.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include <nvrtc.h>
#include <regex>
#include <string>
#include <sys/file.h>

#include "../utils/exception.hpp"
#include "../utils/format.hpp"
Expand All @@ -18,6 +19,29 @@

namespace deep_gemm {

Copy link
Copy Markdown

@afurm afurm Apr 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exception-unsafe fd in constructor

Line 28: fd_ = open(lock_path.c_str(), O_CREAT | O_RDWR, 0666);

If open() succeeds but DG_HOST_ASSERT on the next line throws, the constructor never finishes and the destructor never runs — fd leaks. Use try-catch in constructor body, close fd in catch before re-throwing; or use initializer-list pattern that throws from open() directly so no valid fd exists if later assertions fail.

// RAII cross-process file lock for JIT compilation
class FileLock {
int fd_ = -1;

public:
explicit FileLock(const std::filesystem::path& lock_path) {
fd_ = open(lock_path.c_str(), O_CREAT | O_RDWR, 0666);
DG_HOST_ASSERT(fd_ >= 0 and "Failed to open lock file");
int ret = flock(fd_, LOCK_EX);
DG_HOST_ASSERT(ret == 0 and "Failed to acquire file lock");
}

~FileLock() {
if (fd_ >= 0) {
flock(fd_, LOCK_UN);
close(fd_);
}
}

FileLock(const FileLock&) = delete;
FileLock& operator=(const FileLock&) = delete;
};

class Compiler {
public:
static std::filesystem::path library_root_path;
Expand Down Expand Up @@ -106,6 +130,15 @@ class Compiler {
if (const auto& runtime = kernel_runtime_cache->get(dir_path); runtime != nullptr)
return runtime;

// Cross-process lock to prevent concurrent compilation of the same kernel
make_dirs(cache_dir_path / "locks");
const auto lock_path = cache_dir_path / "locks" / fmt::format("kernel.{}.{}.lock", name, get_hex_digest(kernel_signature));
FileLock lock(lock_path);

// Re-check after acquiring lock
if (const auto& runtime = kernel_runtime_cache->get(dir_path); runtime != nullptr)
return runtime;

// Create the kernel directory
make_dirs(dir_path);

Expand Down