Home Massive hack hits AI servers, exploits Ray framework vulnerability

Massive hack hits AI servers, exploits Ray framework vulnerability

Researchers have found that thousands of servers, running AI infrastructure, have been hacked in an active attack campaign. This attack targets a reported vulnerability in Ray, a computing framework used by the likes of OpenAI, Uber, and Amazon.

According to analysts at Oligo, a cybersecurity firm, the vulnerability allows attackers to take over the companies’ computing power and leak sensitive data. The blog post stated that this flaw has been under active exploitation for the last seven months, affecting sectors like education, cryptocurrency, biopharma and more.

They claim that a trove of sensitive information from compromised servers has been leaked, targeting entities such as OpenAI, Hugging Face, Stripe, and Slack, as well as cloud environments like Amazon’s AWS and Microsoft Azure.

Oligo reports that it discovered hundreds of compromised clusters, with hackers allegedly installing cryptocurrency miners on compromised infrastructure. The researchers explain that attackers choose to compromise these machines because they can obtain valuable sensitive information, and GPUs are very expensive and difficult to obtain.

The company reports that GPU on-demand prices on AWS can reach an annual cost of $858,480 per machine, which means the total amount of machines and computing power that might have been compromised is estimated to be worth almost a billion dollars. Attackers have also installed reverse shells, which are text-based interfaces that allow for remote server control.

In a statement, it continued: “When attackers get their hands on a Ray production cluster, it is a jackpot. Valuable company data plus remote code execution makes it easy to monetize attacks—all while remaining in the shadows, totally undetected (and, with static security tools, undetectable).

What is the Ray AI framework?

Ray, an open-source unified compute framework, simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.

These applications generally operate on large clusters of servers. A central dashboard serves as an interface for displaying and managing active tasks and applications. Among the programming interfaces accessible via this dashboard is the Jobs API. It enables users to dispatch a list of commands to the cluster through a straightforward HTTP request that doesn’t require authentication.

In November 2023, analysts from the security outlet Bishop Fox spotted a similar vulnerability in Ray, tracked as CVE-2023-48022. Bishop Fox senior consultant Berenice Flores Garcia wrote in a blog post: “In the default configuration, Ray does not enforce authentication. As a result, attackers may freely submit jobs, delete existing jobs, retrieve sensitive information, and exploit the other vulnerabilities described in this advisory. “

Anyscale response

A spokesperson for Anyscale, the developer behind Ray, confirmed to ReadWrite that there were a number of issues it was fixing. In a statement, they said: “In light of reports of malicious activity, we have moved quickly to provide tooling to allow users to verify proper configuration of their clusters to avoid accidental exposure.”

They also said the firm was providing a client-side script and server-side code and that it had pre-configured the defaults of the client-side script to reach out to a server it has set up, simplifying the process of determining whether or not ports are unexpectedly open.

Last October, the company had initially denied the reports stating that four of the five reported bugs had already been fixed in November. They disputed the term “vulnerability,” referring to it as a bug instead.

It added: “We recognize that reasonable minds can differ on this issue, and consequently have decided that, while we still do not believe that an organization should rely on isolation controls within Ray like authentication, there can be value in certain contexts in furtherance of a defense-in-depth strategy, and so we will implement this as a new feature in a future release.”

Featured image: DALL-E

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Suswati Basu
Freelance journalist

Suswati Basu is a multilingual, award-winning editor and the founder of the intersectional literature channel, How To Be Books. She was shortlisted for the Guardian Mary Stott Prize and longlisted for the Guardian International Development Journalism Award. With 18 years of experience in the media industry, Suswati has held significant roles such as head of audience and deputy editor for NationalWorld news, digital editor for Channel 4 News and ITV News. She has also contributed to the Guardian and received training at the BBC As an audience, trends, and SEO specialist, she has participated in panel events alongside Google. Her…

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.