Encrypted Traffic Analysis for Threat Detection

Threat Hunting across Encrypted Surfaces

Almost all modern business communications occur over the internet. The digitization of almost all business processes and the increasing reliance on the cloud has led to almost all communications being encrypted. While the increasing level of encryption is a major improvement in data security, it has created a ‘blind spot’ for threat hunting.

Attacks can now be easily hidden among legitimate encrypted traffic. Most of the time, threat hunting involves reviewing the content of the traffic being transmitted. With encrypted traffic, the content is hidden and cannot be reviewed.

Encrypted traffic analysis shifts the focus from the content to the context. It focuses on the elements of the packet which are transmitted along with the encrypted content.

For threat hunting, encrypted traffic can be compared to Schrödinger’s cat. Until the traffic is decrypted, threat analysts have no way of knowing whether the traffic contains the data in its encrypted form, or the data has been tampered with.

The good news is that most of the time when threats are discovered they are due to poorly configured or misimplemented security controls. An old and unsupported protocol is likely to be discovered before a genuine attack is found.

What Is Encryption in Network Traffic?

Communication between systems is facilitated via the transmission of data across a network. Before it is transmitted, the data may be encrypted. Only the intended recipient can decode the encrypted data.

Once data is encrypted, it is unreadable to unauthorized parties. Even if an unauthorized party is able to intercept the traffic, it will be in an encrypted form. The presence of traffic in an encrypted form simplifies threat hunting.

Since traffic is easy to encrypt, but hard to decrypt, focus shifts away from deciphering data to analyzing behavior of other elements, like who communicates with whom, how frequently, for how long. Which communication protocols are used? These behaviors are encrypted and are not indicative of the data being communicated. These are the main concepts of encrypted traffic analytics.

Network Traffic Analysis (NTCA) provides a host of benefits including but not limited to:

• Although there are a plethora of logs created and handled by endpoints, much of that communication goes silent. In encrypted environments, that communication often happens over the network. NTCA provides that visibility.
• Mobilizing that communication creates visible behavioral patterns. For example, communication can be detected before threats are perceived by other, less encrypted means.
• Easier and quicker investigations are the byproduct of behavioral and communicative patterns.
• Previously obscured communication is brought to light by behavioral communication pattern analysis.
• Quickly identifying and responding to communicative patterns can help thwart lateral movement.
• Quicker audits. Sometimes what’s on paper and what’s in practice don’t align. With traffic data, you can see what’s actually in use.

Reality Checks

You can find the biggest discrepancies between what you expect to see and what you actually see in the world of policy. Let’s say there’s a policy to block certain protocols. You may find that, in practice, that policy is completely ineffective. There are many reasons this may happen. It may be that a protocol is being surreptitiously blocked by a service using an encrypted channel.

Policy gaps are cause for concern. Finding a policy discrepancy can mean that there is a risk. These are worth investigating and remedying.

How to Analyze Encrypted Network Traffic

This requires a shift in focus. Generally, you’d first identify the threats you want to protect against and then look for the relevant indicators.

Start with your policy. Determine what it is that you need to validate and then verify if you are positioned to do so. This usually means you need a surface to observe encrypted traffic.

Next, come up with a theory. For example, “No internal hosts should communicate using TLS 1.0.” From your theory, identify what you need to validate using encrypted traffic analysis.

• Traffic should not use the wrong protocol – Plain HTTP over an encrypted channel should not occur.
• Conduct an application session – Often, the first session establishes the context of the communication. Encryption should not always be assumed.
• Review encrypted traffic to validate the presence of out-dated or weak algorithm suites.
• Determine the version of TLS/SSL that is being used.

Identify C2 Traffic in Encrypted Communications

Command and control is the communication link between an attacker and the machine he compromises. C2 communications usually occur after an attack payload is delivered to the compromised machine. C2 communications are also referred to as beaconing.

Beaconing communications occur over encrypted channels. To further obfuscate communications, attackers introduce random delays.

Previously, to identify beaconing communications, an attacker only had to ensure that a payload was delivered at periodic intervals. Now, an analyst has to identify normal and expected communication patterns. Once normal communication patterns are identified, a deviation from those patterns likely represent an attack.

A few signals are consistent:

• Check-ins to destinations that are rarely/never checked in to by the user, with randomized interval
• Sessions that have an incredibly consistent number of participants
• Long-lasting connections with hosts that normally create short-lived connections
• Self-signed certs, short-lived/temporary certs, and mis-matched server certs

Jitter can conceal individual packets, but over a longer period of time, it can’t conceal the overall pattern. Some behavioral patterns are tough to represent using logic and rules, which is why behavioral patterns in encrypted traffic are helpful to discover other potential threats.

JA3 and JA3S pattern matching

The JA3 pattern is created using a hash of the TLS client hello message. The JA3S pattern is created using a hash of the TLS server hello message.

• Due to TLS implementation by malicious actors staying consistent over multiple pieces of malware, and the repetitive nature of malicious activities, TLS sessions using malware are easily identifiable.
• As mentioned previously, randomizing the order of extension fields can result in an altogether new JA3 hash. Other patterns and methods, such as JA4, have been introduced to account for randomization of extension fields.

Supporting Compliance and Audits

Auditors and cyber insurance companies are increasing their scrutiny. Next year, they will require more stringent protections for data in transit.

This requires a deep analysis of transport layer protocols and the strength of the ciphers. Translation is required to interpret these findings for a non-technical audience.

This can be simplified with a network detection and response platform that provides encrypted traffic analysis. The solution should produce actionable recommendations that enable organizations to address audit and compliance requirements.

NetWitness provides network detection and response capabilities, and enables security teams to perform traffic analysis to meet their compliance obligations.

• Gain insight into encrypted sessions to identify threats
• Use ML to analyze encrypted data flows
• Detect oddities in encrypted traffic
• Create reports that meet your legal or regulatory obligations
• Automate your work processes

Being able to analyze encrypted data traffic flow has profoundly changed the way security personnel perform their duties. Rather than simply waiting for threat indicators to be alerted to possible attacks, security personnel can now hunt for threats using sophisticated techniques.

We have described the benefits of incorporating encrypted traffic analysis in your threat management framework. Using this analysis, you can intercept threats that are otherwise concealed within encrypted data.

It is common for organizations to implement data encryption. While doing so, privacy of customers is protected. However, data encryption creates blind spots for an organization to identify threats. Incorporating encrypted data traffic analysis can help organizations to cut threats concealed within encrypted data. The primary objective of using encryption is protected in that data is protected from being accessed by unauthorized persons, including attackers.