When Replicas Go Rogue - A Deep Dive into Cloudflared Replicas Exploitation Scenarios

Posted Dec 7, 2024

By Yan Sandman 17 min read

I’ve been using Cloudflare tunnels for various applications for quite some time. Recently, I noticed a warning box that might have been there all along, but I had never paid attention to it before.

“Store your token carefully. This command includes a sensitive token that allows the connector to run. Anyone with access to this token will be able to run the tunnel.”
The message seems pretty obvious, but it made me wonder—how bad could it be if the connector token were stolen? And what could a threat actor achieve with it?

This blog post shows how I turned a ‘What can go wrong?’ theory into practice.

Introduction

Cloudflare ZTNA or Cloudflare Access is a part of Cloudflare One SASE (Secure Access Service Edge) It is a product designed to secure applications, data, and devices, offering features such as global authentication, secure web gateways, remote browsing, data loss prevention, and email security.
This blog post touches on attack methods around 2 crucial areas of Cloudflare Zero Trust:

Cloudflare Tunnel: This feature enables secure access to resources without opening ports or assigning public IPs. Instead, a lightweight agent, Cloudflared (and now also WARP for Linux devices), is installed on the host to create outbound connections to Cloudflare’s global network, ensuring the application remains protected from exposure to inbound traffic.
Cloudflare Access: Self-hosted applications - This feature allowing secure publication of internal tools and applications to the Internet by adding an authentication layer between the end user and the origin server.

Here is a diagram that shows how it is all orchestrated together:

credit: Cloudflare https://developers.cloudflare.com/cloudflare-one/applications/configure-apps/self-hosted-apps/

This blog post provides a brief technical overview of how the Cloudflare Tunnel client (formerly known as Argo Tunnel) operates. It will then explore various methods that threat actors could use to access Cloudflared connector tokens and delve into how they might exploit the Cloudflared connector replicas feature once such a token is compromised. Finally, the post will present a practical AiTM (Adversary-in-the-Middle) attack scenario, showcasing session hijacking through a step-by-step process. The discussion will conclude with recommended strategies for mitigation, detection, and remediation.

How Cloudflare Tunnel works

A tunnel is a secure, outbound-only pathway that connects your origin to Cloudflare’s global network. This connection is established through a client installed on the origin server, and there are two options for such a client:

Cloudflared Tunnel Client: Written in Go and open-source, this client is developed by Cloudflare to provide secure tunneling.
Cloudflare WARP Connector: Designed for site-to-site, bidirectional, and mesh networking. Currently, in Beta, it supports Linux devices exclusively at this time.

In this blog post, I will focus specifically on the Cloudflared Tunnel Client.

“The Cloudflared Tunnel Client is a command-line utility for Cloudflare Tunnel. It acts as a tunneling daemon, securely proxying traffic from Cloudflare’s network to your origin (e.g., a web server). This daemon sits between Cloudflare’s network and your origin, enabling Cloudflare to route client requests to your server without the need to open firewall ports. This allows your origin to remain as secure and closed as possible.”(Offical Cloudflare Tunnel GitHub Repository: https://github.com/cloudflare/cloudflared).

That’s how it would look in a normal HTTP setup:

credit: Cloudflare https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/

I will now cover two main concepts: Authentication & Cloudflared replicas

Authentication: When configuring a tunnel for the first time, a unique JWT token associated with that tunnel is generated.
This token contains three key components:

Cloudflare One Account ID: Identifies the associated account.
Configured Tunnel UUID: Uniquely identifies the tunnel.
Base64 Encoded Tunnel Secret: A unique secret token.

The structure of the JWT token looks like this:

{
"a":"<Cloudflare One Account ID>",
"t":"<Tunnel UUID>",
"s":"<Base64 Encoded Tunnel Secret>"
}

On both Linux and Windows, the token is passed as an argument to the executable.

On Linux: It is included in the ExecStart parameter of the Cloudflared service.
On Windows: It is stored in the registry key of the Cloudflared service as part of the ImagePath value.

Notably, this token is not tied to a specific machine. As long as the operating system supports the Cloudflared client, any machine can use this token to run the Cloudflared service. This brings us to the second concept: Replicas.

Replicas: Cloudflared replicas are Cloudflare Tunnel’s solution for ensuring availability and failover.
This feature allows you to install the same Cloudflared tunnel on multiple servers. If one origin server fails, the system continues to operate seamlessly. another one with a connector replica installed can serve as a backup and ensure the redundancy and availability of the service

Replicas are the functionality upon which much of this attack theory is based. However, from an attacker’s perspective, they have a downside: traffic forwarding to servers is entirely controlled by Cloudflare, and neither the client nor the tunnel owner can influence or manage this behavior. As stated in the official documentation: “By design, replicas do not offer any level of traffic steering (random, hash, or round-robin). Instead, when a request arrives at Cloudflare, it will be forwarded to the replica that is geographically closest. If that distance calculation is unsuccessful or the connection fails, we will retry others, but there is no guarantee about which connection is chosen.”

Cloudflare ZTNA from an Adversary’s Perspective

Now that we have a better understanding of the concepts and how they function, let’s discuss the potential security issues that could arise. During the technical overview, I mentioned some concepts that may present security challenges. In this section of the blog post, we will explore a possible attack scenario and demonstrate one of them.

How Threat Actors Could Access Cloudflared Connector Tokens

For any of the attacks I’m about to mention to be successful, one crucial prerequisite must be met: obtaining the JWT token associated with the tunnel. There are multiple ways to achieve this, some arising from poor credential management and Identity and Access Management (IAM) practices, while others involve slightly more technical depth. At a high level, the known methods through which these credentials can be accessed by a threat actor include:

Common Credential Theft Methods:

Credentials Stolen via Infostealer Infections: Threat actors target employees with Infostealer malware to extract credentials.
Poor Secret Management Practices: Mismanagement of secrets, allowing unauthorized access to sensitive credentials.
Compromised Cloudflare Accounts: Attackers gaining access to Cloudflare accounts.
Other Unknown Methods: Additional attack vectors specific to unique environments.

A more complex method for threat actors is directly stealing the JWT token from the machine hosting the tunnel. Since the purpose of tunnels is to reduce ingress traffic and rely almost exclusively on egress traffic, the attack surface around the server is significantly reduced.

There are several methods I’m aware of for extracting tokens from Linux and Windows systems. Each method may require different permission levels, depending on the target system and its specific configurations.

Command Line Arguments The JWT is included in the command line of both Linux and Windows versions of the agent. Below are demonstrations for each system.

Linux

Windows

Command History: While Windows has more complexity around logging commands, many Linux distributions have a history file in the user’s home directory by default. If no precautions were taken and the default command from the Cloudflare Tunnel setup was simply copied and pasted, it is likely that the history file will contain the token.

Linux

Service Configuration files & Registry: As part of the agent installation, the JWT is defined in the service configuration. On Linux systems, it is included in the service configuration files, while on Windows systems, it is stored in the registry.

cloudflared.service configuration file

Cloudflared service registry

Memory Dumps: The JWT token also resides in memory. By dumping the memory, we can retrieve its value using utilities like strings.

Dumping Cloudflare memory in Windows.

Grepping for JWT in the Windows memory dump.

Grepping for JWT in the Linux memory dump.

What Threat Actors Can Do with Cloudflare Connector Tokens

As mentioned in the introduction and technical background of this blog post:
“Anyone with access to this token will be able to run the tunnel.”
That is quite alarming. But what exactly can an attacker do if they gain control of a tunnel? Here comes the fun part of the post: the attack scenarios.
The scenarios I’m about to discuss revolve around the Cloudflare Tunnel, which is protected by Cloudflare Access. Once a replica tunnel is installed on a malicious server, it could appear to the end user as if it were the original server. For instance, imagine this very blog post running behind a tunnel whose credentials have been compromised. You might read this post and decide to explore more content on this website. You send your GET request, but it could be routed to a malicious server—completely unknown to you as the end user. In the case of this blog post, the risk might seem minimal since it’s a static blog with no sensitive functionality. But imagine if this were a web application storing sensitive information, implementing authentication processes, or even serving as an internal company page for downloading internal tools. In such cases, the implications could be devastating.
The good news is that the traffic is distributed randomly across the replica servers, meaning the attacker cannot control when a specific user is routed to their malicious server. Additionally, there are mitigations available in HTTP setups to help with session hijacking scenarios, which will be presented later in this blog post.
The bad news is that the lack of control doesn’t eliminate the attack’s potential impact. Even without precise targeting, the attacker can still execute this attack and cause significant damage. For instance, will your users—trained to spot phishing attacks through suspicious URLs—be able to identify this? Or will these incidents merely be reported to your support and IT teams as “temporary connectivity problems with Cloudflare”?

When I tried to threat model this scenario, I considered multiple scenarios that might be implemented. Each has its pros and cons—some are effective, while others are less so. The scenarios include:

DoS attacks on a service by installing as many replicas that will lead to nowhere.
Redirection of users to malicious websites
Serving malicious client-side scripts or malware to the application’s users

However, the most interesting scenario I thought of was an Adversary-in-the-Middle (AiTM) attack, where the attacker intercepts traffic randomly and hijacks as many sessions as possible.

Practical AiTM Attack Scenario: Intercepting CF_Authorization Tokens to Hijack Cloudflare Access Application Sessions

The attack works as follows: When a user authenticates to the Cloudflare Access application with its agentless authentication setup (this means any authentication method other than WARP), they receive two tokens: CF_AppSession and CF_Authorization. Using these tokens, they can access the application behind Cloudflare Access.

This is possible under the default settings because an option called CF_Binding Cookie is turned off. Turning on this option mitigates the attack.

Default Cookie settings.

When a replica tunnel is installed, the traffic intended for the origin server can also be randomly sent to the malicious replica. This behavior is designed to avoid availability issues and serve as a failover mechanism. I exploited this behavior to create a web server that randomly intercepts some of the requests meant for the origin server. This enables the hijacking of Cloudflare Access sessions and the capture of other sensitive application data.

It is worth mentioning that the CF_Binding cookie will prevent Cloudflare Access session hijacking but will not prevent the interception of other data from the application running on the origin server. To overcome the fact that the malicious server cannot run the same application, each time a user request is sent to that server, it redirects the user multiple times until the request eventually reaches the original origin server.

The animated diagram below demonstrates the flow of the attack.

Here is how it works step by step:

When an unauthenticated user sends a request to the host behind Cloudflare Access, they are automatically redirected to the Cloudflare Access page.
In that case, the user performs an email OTP authentication, which is the authentication method chosen during the access policy setup.

A successful authentication will return a JWT token called CF_Authorization, and a CF_AppSession.
The JWT token is built in the following way:

{
  "aud": [
 "3ac7587a992c63b2113f43bc59ca0e4f206b7cc8547d20c9426bca83e9534c0d"
  ],
  "email": "<Authenticated Email>",
  "account_id": "<Cloudflare Account ID>",
  "exp": 1732377100,
  "iat": 1732290700,
  "nbf": 1732290700,
  "iss": "https://<Team Name>.cloudflareaccess.com",
  "sub": "5d7094ed-13f3-50ce-bae9-42b7631af439",
  "identity": {
 "email": "<Authenticated Email>",
 "idp": {
   "id": "2e34e2e3-18c1-41be-9371-1cb99b3c33e8",
   "type": "onetimepin"
 },
 "geo": {
   "country": "<country code>"
 },
 "user_uuid": "<user_uuid>",
 "account_id": "<account_id>",
 "iat": 1732290700,
 "ip": "<IP Address>",
 "auth_status": "NONE",
 "common_name": "",
 "service_token_id": "",
 "service_token_status": false,
 "is_warp": false,
 "is_gateway": false,
 "device_id": "",
 "mtls_auth": {
   "cert_issuer_dn": "",
   "cert_serial": "",
   "cert_issuer_ski": "",
   "cert_presented": true,
   "common_name": "",
   "auth_status": "NONE"
 },
 "version": 2
  },
  "type": "org",
  "identity_nonce": "mSf6ZRve8BTQ9Oy8"
}

The user will then be redirected to the protected application Then, there will be a redirect to the protected application, and any request to that application will include the CF_Authorization token that will be sent to the origin server
In my scenario, I have also included a second layer of simple authentication to the application itself, user name + password that returns a session token to access the app:
There are now three cookies under the application:
- CF_AppSession: The Cloudflare Access application session.
- CF_Authorization: The retrieved cookie from a successful authentication.
- session: The application’s session cookie.
Each of these tokens will be included in every request and may also be sent to the malicious server, potentially in one or more of the requests. The malicious server’s role is to redirect the client as many times as possible until the client eventually reaches the original server.
For debugging purposes, I added a token called a ‘redirect token’ for the client that connected to the malicious server.
Here’s how it appears from the AiTM server:
The session tokens are logged on the malicious server:
Using the session tokens from a previous authenticated client results in successful authentication:

Below is a short video demonstrating the attack in real-time, showing interactions with the original server, the malicious rogue server, and the victim client.

Applying the technique to an automated PoC

To simplify the process, I have created an automated PoC that installs all dependencies, sets up the tunnel connector, and runs the web server using arguments derived from the cloudflared.service log file.

It is available in my GitHub repository:cloudflared_aitm_poc

Mitigation, Detection, and Remediation

This part of the blog post will discuss ways to mitigate this type of attack, how to detect it, and the steps to remediate in case such an attack is detected.

Mitigating Session Hijacks in Cloudflare Access Applications

As mentioned earlier, there is a mitigation against this type of attack: CF Binding Cookie Here’s how it works: This cookie is added to the user’s browser once they successfully authenticate to the Cloudflare Access application. It is used to authenticate with the Cloudflare global network. However, after successful authentication, this cookie is stripped off by Cloudflare before reaching the origin server.

It is important to note that this mitigation only addresses session hijacking within Cloudflare applications. The request will still be intercepted. For tunnels that are not protected behind Cloudflare Access, implementing measures such as continuous token refresh can help prevent session hijacking. However, it is crucial to understand that the mitigation for all attack scenarios lies in proper secret and IAM management. These measures may vary depending on the system and setup, whether in a public cloud environment or an on-premises configuration so they won’t be covered in this blog post.

Detection Strategies for Potential Cloudflared Connector Exploitation

Unfortunately, while attempting to locate logs after exploitation, I couldn’t find any reliable sources for viewing past logs from Cloudflare connectors.

For now, the most effective method I have found involves querying the Cloudflare One API for Cloudflared tunnels. It is important to note that this method only provides real-time information. Below is the API request that returns all relevant information. This can be executed using a Lambda function, a cron job, or any other method of your choice at regular intervals.

The detection logic depends on the workloads of the target environment. If new connector replicas are created with a public IP address that you do not recognize, it could indicate a potential attack.

curl --request GET \
  --url https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/cfd_tunnel \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer {API_KEY}'| jq '.result[] | {id, name, status, created_at, deleted_at, connections}'


{
  "id": "TUNNEL_ID",
  "name": "TUNNEL_NAME",
  "status": "healthy",
  "created_at": "2024-11-21T20:55:15.521851Z",
  "deleted_at": null,
  "connections": [
    {
      "colo_name": "The Cloudflare data center used for this connection.",
      "uuid": "UUID",
      "id": "ID",
      "is_pending_reconnect": false,
      "origin_ip": "The Connector Public IP",
      "opened_at": "2024-12-04T13:35:05.685970Z",
      "client_id": "Connector ID",
      "client_version": "2024.11.1"
    },
    {
      "colo_name": "The Cloudflare data center used for this connection.",
      "uuid": "UUID",
      "id": "ID",
      "is_pending_reconnect": false,
      "origin_ip": "The Connector Public IP",
      "opened_at": "2024-12-04T13:35:05.957915Z",
      "client_id": "Connector ID",
      "client_version": "2024.11.1"
    },
    {
      "colo_name": "The Cloudflare data center used for this connection.",
      "uuid": "UUID",
      "id": "ID",
      "is_pending_reconnect": false,
      "origin_ip": "The Connector Public IP",
      "opened_at": "2024-12-04T13:35:06.943615Z",
      "client_id": "Connector ID",
      "client_version": "2024.11.1"
    },
    {
      "colo_name": "The Cloudflare data center used for this connection.",
      "uuid": "UUID",
      "id": "ID",
      "is_pending_reconnect": false,
      "origin_ip": "The Connector Public IP",
      "opened_at": "2024-12-04T13:35:07.956288Z",
      "client_id": "Connector ID",
      "client_version": "2024.11.1"
    }
  ]
}

Remediation: rotation of a Compromised Token

Follow Cloudflare’s guidance on the rotation of a compromised token and rotation of a token without service disruption.

Vendor Response

The vendor has updated the documentation for rotating compromised tokens and rotating tokens without service disruption and has requested that they be included as references in the blog post.

https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#rotate-a-token-without-service-disruption
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#rotate-a-compromised-token

Disclosure Timeline

2024-11-22 Report of a potential attack method submitted through Cloudflare’s bug bounty program on HackerOne.
2024-11-24 Initial report mistakenly closed by the HackerOne triage team.
2024-11-26 Proof of concept submitted, prompting further review by Cloudflare.
2024-11-27 Coordination with Cloudflare began. Blog publication delayed at their request.
2024-12-04 Cloudflare confirmed updated documentation on Tunnel token security and token rotation. Requested inclusion of links to updated documentation in the blog post.
2024-12-08 Blog post published with the requested reference links.

Conclusion

Compromised tokens in Cloudflare Tunnels can lead to serious security issues, including session hijacking and data interception. Mitigating these risks involves enabling features like CF_Binding cookies, regularly rotating tokens, and monitoring for unauthorized replicas through API queries. By taking these steps, you can effectively reduce the likelihood of exploitation and ensure your applications remain secure.

References

https://developers.cloudflare.com/reference-architecture/static/cloudflare-evolving-to-a-sase-architecture.pdf
https://blog.cloudflare.com/zero-trust-sase-and-sse-foundational-concepts-for-your-next-generation-network/?utm_medium
https://developers.cloudflare.com/cloudflare-one/
https://developers.cloudflare.com/cloudflare-one/applications/configure-apps/self-hosted-apps/
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/tunnel-useful-terms/
https://community.cloudflare.com/t/tunnel-cannot-route-to-geographically-closest/537965/4
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#rotate-a-token-without-service-disruption
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#rotate-a-compromised-token

Cloudflare Security, Cloudflared, AiTM

This post is licensed under CC BY 4.0 by the author.