Detecting Phishing attempts with DNSTWIST

DNSTWIST for SOC & CTI

DNSTWIST

Dnstwist is a tool designed to identify domains that might be used in phishing attacks or other malicious activities by generating a comprehensive list of variations on a given domain using the following techniques:

• Addition: Adds characters to the domain name.
• Bitsquatting: Uses bit errors to generate similar-looking domain names.
• Cyrillic: Utilizes Cyrillic characters to create visually similar domains.
• Homoglyph: Employs characters that look alike but have different meanings.
• Hyphenation: Adds hyphens to generate new permutations.
• Insertion: Inserts characters into the domain name.
• Omission: Omits characters from the domain name.
• Repetition: Repeats characters in the domain name.
• Replacement: Replaces characters in the domain with others.
• Subdomain: Adds or alters subdomains.
• Transposition: Changes the order of characters in the domain name.
• Vowel-Swap: Swaps vowels within the domain name.
• Dictionary: Utilizes dictionary words to generate new permutations.

By comparing these generated domains to the existing ones, dnstwist helps uncover domains that could be employed to impersonate or mimic legitimate sites for phishing.

For the Cyber Threat Intelligence (CTI):

Using the list of generated similar domains, CTI teams can proactively detect similar domain registrations, which may be indicative of an attacker possibly preparing a phishing campaign. The CTI team can then monitor the services associated with the similar domain, as well as any potential future associations or developments related to that domain.

The registration check is generally performed at a lower frequency (sometimes delayed by several hours or even a day, depending on the WHOIS service used by the CTI), so there might be a gap in visibility if an attacker moves quickly. In such instances, using dnstwist in close collaboration with the SOC team is necessary.

For the Security Operational Center (SOC):

The SOC uses the list of generated similar domains to identify actual network requests to these domains within the logs collected on the SIEM. This monitoring allows for real-time detection, potentially identifying successful phishing attempts against an organization’s users

This process use various logs that contain the domain name information, including but not limited to DNS logs, proxy logs, email sender/recipient domain names, EDR DNS requests…

By synergizing the capabilities of both CTI and SOC through the use of similar domains, organizations can construct a more resilient and responsive defense mechanism against phishing attacks.

For the Redteam:

Dnstwist serves as a valuable tool for generating creative phishing ideas for the redteam/bad guys but as dnstwist is a now a well-known tool used by many security teams, crafting a relevant phishing domain that is not in its default generated list can provide a significant advantage. It enables the red team to mimic legitimate domains while avoiding detection of similar domain.

(To initially search for domain names owned by your target company, searching the company name on shodan, bgp and github commits (emails) will give you valuable informations)

Detection of the tool DNSTWIST

Before diving into the details of configuring dnstwist within your environment and deploying detection use cases, it’s important to understand the kind of events the tool dnstwist itself can generate and how to detect them. As previously explained, dnstwist can be leveraged by both the blue team for defensive purposes and the red team for offensive strategies. Here’s what to expect in terms of behavior detection from dnstwist:

User-Agent String:

• The default user-agent string used by dnstwist for some of its requests is constructed as follows: USER_AGENT_STRING = 'Mozilla/5.0 ({} {}-bit) dnstwist/{}'.format(sys.platform, sys.maxsize.bit_length() + 1, __version__). Therefore, the appropriate detection method would be to search for the string Mozilla/5.0 (*-bit) dnstwist in the field http_user_agent in your proxy logs. (or whatever field name you are using for user agent in your SIEM)

CommandLine script argument:

• Search for the commandline argument --fuzzers following by one of the algorythm. It is the only argument that is subject to a low false positives rate detection:
• --fuzzers addition
• --fuzzers bitsquatting
• --fuzzers cyrillic
• --fuzzers homoglyph
• --fuzzers hyphenation
• --fuzzers insertion
• --fuzzers omission
• --fuzzers repetition
• --fuzzers replacement
• --fuzzers subdomain
• --fuzzers vowel-swap
• --fuzzers transposition
• --fuzzers dictionary

SMTP requests:

• The function _mxcheck in the code below is designed to perform a mail exchange (MX) record check by attempting to send an email to a user on the domain(s) :

def _mxcheck(self, mx, from_domain, to_domain):
 from_addr = 'randombob1986@' + from_domain
 to_addr = 'randomalice1986@' + to_domain
 try:
 smtp = smtplib.SMTP(mx, 25, timeout=REQUEST_TIMEOUT_SMTP)
 smtp.sendmail(from_addr, to_addr, 'And that\'s how the cookie crumbles') # sender,recipient,body
 smtp.quit()
 except Exception:
 return False
 else:
 return True

This function will assess whether the domain has a valid MX record and can receive emails by sending a test email. However, it relies on hardcoded user names for the email, specifically using the strings randombob1986@ or randomalice1986@. You can detect these by examining the sender or recipient fields in your email logs.

• Monitoring for a high volume of SMTP requests from a single source in a short period of time can provide additional insight and support in this context.

Frequency and Timing of DNS Requests:

(default behavior but can be opt out with --format list):

• Detect a single source IP address making a high number of DNS requests within a short period of time ! Dnstwist will make thousands of DNS requests in a very short period of time (1m-5m depending on the DNS server contacted and th lengh of your domain)

Request to Non-Existent Subdomains:

• Detect a single source IP address making a high number of requests to non-existent subdomains in a short period of time (especially subdomains of your domain list)

SOC - SIEM Integration & Automation

Below are some proposed implementations of DNSTWIST:

Simple Splunk Integration (dns fuzzing without dns requests) :

• 1. Execute the dnstwist script directly from the SIEM (creating a splunk app is necessary if you are using Splunk Cloud)
• 2. Saving the results of dnstwist for each given domain into a lookup table (csv file)

API ACCESS

To obtain the currently active, registered, or resolving similar domains linked to an IP address, we can create a separate API access for Dnstwist on an isolated network. This approach ensures that we don’t request the potential phishing domain from within our secure environment !

Such an API access could be tailored to meet the needs of both CTI and SOC use cases.

I’ve created two example scripts to illustrate how this process can work. You can find them here: https://github.com/mthcht/Purpleteam/tree/main/Logging/dnstwist

• request_dnstwist.py: This client script will interface with the API endpoint of dnstwist_api.py. By providing a domain, it will request the corresponding similar domain lists and save them to a CSV file. This file can later be leveraged within the SIEM for crafting detection rules.
• dnstwist_api.py: Run this on a server where dnstwist is installed, ideally in an environment isolated from your enterprise network. This script will handle dnstwist operations as requested by a client, allowing it to be utilized for various scan operations required by CTI.

Below is our test example with dnstwist_api.py, just a simple Flask app:

Get mthcht’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The dnstwist_algo endpoint is specifically designed to carry out DNS fuzzing operations on a given domain and return the results as a list without resolving the actual domain names. This is the fast approach and may not necessarily require a dedicated API server for this, it can be executed directly by the SIEM/SOAR. (as illustrated in the first integration example ‘Simple Splunk Integration’)

@app.route('/dnstwist_algo', methods=['POST'])
def get_dnstwist():
    print(request.data, request.json)  # Print request data for debugging
    domain = request.json['domain']
    try:
        result = subprocess.run([DNSTWIST_BINARY,'--format','list',domain], stdout=subprocess.PIPE, text=True, check=True)
        twisted_domains = result.stdout
        return Response(twisted_domains, mimetype='text/csv')
    except subprocess.CalledProcessError as e:
        return Response(str(e), status=500)
    except Exception as e:
        return Response(str(e), status=400)

The dnstwist_resolved endpoint carries out DNS fuzzing operations on a specific domain and subsequently resolves all the similar domain names. This process involves thousands of DNS requests, and the results are then returned to the client script. It’s a more comprehensive approach that provides detailed information about the domain’s DNS informations.

@app.route('/dnstwist_resolved', methods=['POST'])
def new_endpoint():
    print(request.data, request.json)
    domain = request.json['domain']
    nameservers = '8.8.8.8,8.8.4.4' # Change this for your own dns servers (using google dns can be faster if your own servers can't handle thousands of dns requests in a short period of time)
    try:
        result = subprocess.run([DNSTWIST_BINARY,'--format','csv','--registered','--nameservers',nameservers,domain], stdout=subprocess.PIPE, text=True, check=True)
        twisted_domains = result.stdout
        return Response(twisted_domains, mimetype='text/csv')
    except subprocess.CalledProcessError as e:
        return Response(str(e), status=500)
    except Exception as e:
        return Response(str(e), status=400)

Our test example with request_dnstwist.py (client making the request to the API):

import requests
import logging
import csv
import sys
import argparse

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
def get_method_and_header(algo, resolved, dnstwist_api_server):
    if resolved:
        method = f"{dnstwist_api_server}/dnstwist_resolved"
        header = ["metadata.algotype", "dest_nt_domain", "dest_ip", "metadata.dns_aaaa", "metadata.mail_server", "metadata.dns_ns","metadata.original_domain"]
        algotype = 'resolved'
    else:
        method = f"{dnstwist_api_server}/dnstwist_algo"
        header = ["dest_nt_domain", "metadata.original_domain"]
        algotype = 'algo'
    return method, header, algotype
def query_dnstwist(method,domain):
    try:
        response = requests.post(method, json={'domain': domain})
        response.raise_for_status()
        return response.text
    except requests.RequestException as e:
        logging.error(f"An error occurred while querying dnstwist for {domain}: {e}")
        return None
def save_to_csv(algotype,header, data, domain):
    filename = f"SOC_DNSTWIST_{domain}_List.csv"
    try:
        with open(filename, 'w', newline='') as file:
            writer = csv.writer(file)
            writer.writerow(header)
            if algotype == 'algo':
                rows = data.split('\n')
            if algotype == 'resolved':
                rows = data.split('\n')[1:]
            for line in rows:
                if line:
                    writer.writerow(line.split(',') + [domain])
        logging.info(f"Results saved to {filename}")
    except Exception as e:
        logging.error(f"An error occurred while saving to CSV for {domain}: {e}")
def main():
    parser = argparse.ArgumentParser(description='Client script for dnstwist.')
    parser.add_argument('domains', nargs='+', help='List of domains to process')
    parser.add_argument('--algo', action='store_true', help='Execute the algo function')
    parser.add_argument('--resolved', action='store_true', help='Execute the resolved function (will make thousands of dns requests)')
    args = parser.parse_args()
    dnstwist_api_server = 'http://127.0.0.1:443'
    if not args.domains:
        logging.error("Please provide at least one domain as an argument")
        return
    if not args.resolved and not args.algo:
        args.algo=True
    for domain in args.domains:
        logging.info(f"Querying dnstwist for domain: {domain}")     
        method, header, algotype = get_method_and_header(args.algo, args.resolved, dnstwist_api_server)
        result = query_dnstwist(method,domain)
        if result:
            save_to_csv(algotype, header, result, domain)
        else:
            logging.error(f"Failed to retrieve data from dnstwist for {domain}")
if __name__ == '__main__':
    main()

Examples usages:

python3 request_dnstwist.py --algo <mydomain>
python3 request_dnstwist.py --resolved <mydomain>
python3 request_dnstwist.py --algo <mydomain>,<mydomain>,<mydomain>,<mydomain>
python3 request_dnstwist.py --resolved <mydomain>,<mydomain>,<mydomain>,<mydomain>

Replace with the domain names owned by your company.

If the path I want to monitor for phishing is intra.laposte.fr, and I want to use the ‘resolved’ function (specifically to monitor the registered and active similar domains), the content of my lookup table (as of 2023/08/15) would be:

FYI The dnstwist project also provides a API we can use here: https://github.com/elceef/dnstwist/tree/master/webapp

DNSTWIST API Server connected to the SIEM:

• (0). The list of the enterprise domains is maintained on the SIEM and a scheduled search will frequently use this list with our script/application to generate the similar domains list
• 1. The SIEM is contacting our API service (through the installation of our own application or via a custom Splunk search command)
• 2. The API endpoint for dnstwist receives the request from our SIEM for a domain search and executes dnstwist to generate the list of similar domains. Depending on the configuration we choose, dnstwist will also try to resolve all the domains from the domain list, resulting in thousands of DNS requests being executed!
• 3. The execution of dnstwist is finished, and the results are returned in CSV format to our application/script making the request on the SIEM.
• 4. The application/script saves the content of similar domains in a lookup table available on the SIEM.
• (5). Detection rules on the SIEM use the lookup table of similar domains to detect potential phishing attempts from the enterprise network.

SOAR connected to the SIEM and DNSTWIST API

• (0). The list of the enterprise domains is maintained on the SIEM
• 1. The SOAR requests the SIEM (API access) to retrieve the content a the lookup table, which contains the list of legitimate domains of our enterprise.
• 2. The SIEM return the content of the lookup to the SOAR
• 3. The SOAR is contacting our API service with a playbook
• 4. The API endpoint for dnstwist receives the request from the SOAR for a domain search and executes dnstwist to generate the list of similar domains. Depending on the configuration we choose, dnstwist will also try to resolve all the domains from the domain list, resulting in thousands of DNS requests being executed!
• 5. The execution of dnstwist is finished, and the results are returned in CSV format to the playbook on the SOAR.
• 6. The SOAR push the similar domain list in a lookup table on the SIEM
• (7). Detection rules on the SIEM use the lookup table of similar domains to detect potential phishing attempts from the enterprise network.

SOAR connected to the SIEM and DNSTWIST API (2):

• (0). The list of the enterprise domains is maintained on the SOAR
• 1. The SOAR is contacting our API service with a playbook
• 2. The API endpoint for dnstwist receives the request from the SOAR for a domain search and executes dnstwist to generate the list of similar domains. Depending on the configuration we choose, dnstwist will also try to resolve all the domains from the domain list, resulting in thousands of DNS requests being executed!
• 3. The execution of dnstwist finishes, and the results are returned in CSV format to the playbook on the SOAR.
• 4. The SOAR will handle the SIEM search by himself: Directly execute a splunk search via API with the similar domains names.
• 5. The Splunk searches finishes, and the results are returned to the SOAR (or the SIRP used for your enterprise)

Gitlab:

In all the examples I’ve illustrated above, the SOAR could be substituted with Gitlab CI/CD using scheduled jobs to regularly update the lookups on the SIEM.

CTI — Alerts:

On the CTI side, when an alert is triggered for a domain registration that matches one of the similar domains generated with dnstwist, an automatic action can be initiated.

This may include blocking the domains and corresponding IP addresses on the proxy, firewall, or mail server within the infrastructure, using the SOAR, for example.

These identified phishing domains by the CTI team can then be forwarded to the SIEM, resulting in a critical alert for any attempts to contact these domains or their associated IP addresses.

Additional checks can be performed with dnstwist by the CTI’s automated tasks to pinpoint an actual phishing site. By using the arguments — phash , — lsh or — ssdeep, the similarity of the website can be analyzed.

More details on the phishing detection approach can be found in the dnstwist GitHub repository https://github.com/elceef/dnstwist

SOC — Detection rules

Once you’ve successfully set up the automated process to maintain an updated list of similar domains on your SIEM, you can start the hunt:

Below is a straightforward SPL search using the list of similar domains. It’s designed to alert you when a user accesses a phishing domain name within the proxy logs:

`my_proxy_logs` src_ip IN (192.168.0.0/16,10.0.0.0/8,172.16.0.0/12)
  NOT [|inputlookup SOC_Laposte_domains_List.csv | fields - metadata.*]
  | lookup SOC_DNSTWIST_laposte.fr_List.csv dest_nt_domain OUTPUT metadata.original_domain as original_domain
  | where isnotnull(original_domain)
  | stats
    values(action)
    values(src_ip)
    values(dest_ip)
    values(dest_port)
    values(original_domain)
    values(url)
    values(http_method)
    values(http_user_agent)
    sum(bytes_in) as total_bytes_in
    sum(bytes_out) as total_bytes_out
    count by _time dest_nt_domain
  | rename values(*) as *
  | convert ctime(_time)

Example of a results from our detection rule:

The outcome of the search triggers an alert that requires investigation by an analyst.

The same logic can be applied across various logs, such as firewall,proxy,email and DNS for detecting the destination IP address (if discovered by dnstwist) and the associated domain names.

If an alert comes from the CTI detection at the registration process, it must also be automated to have the domain name saved in a lookup table on the SIEM for immediate detection and a higher severity.

Some recommandations:

• Domain Registration Monitoring and access: CTI and SOC teams for monitoring and identifying potential impersonating domains using dnstwist and Whois services. (examples use cases i illustrated)
• Multi-Factor Authentication (MFA): Encourage or require MFA across all user accounts on authentication portals. This helps protect user accounts even if credentials are compromised through a phishing attack.
• Legal Actions: Take legal measures against typosquatters, especially if you hold a trademark.
• Email Inspection: Implement a solution to scan and log URLs in incoming emails, enabling a more robust and proactive hunting against phishing attempts.
• SSL Certificates: Ensure the consistent use of SSL certificates to facilitate site ownership verification for users
• Usurpation: Implement timely notifications and alerts to prevent domain expiration and potential hijacking by malicious actors. Regularly monitor your domains’ renewal status to retain control.

Conclusion:

By implementing dnstwist, SOC can achieve real-time detection and alerting for suspicious domains within network/mail activity logs, while CTI teams can proactively monitor domain registrations indicative of potential phishing campaigns in preparation.

If your SOC/CTI team isn’t already leveraging dnstwist for phishing detection, I hope this post can convince you to consider it. Be aware that false positives are possible, especially if the length of your domain name is short or exceptionally common.

Update 2024/07: I added a script with generated lookups to illustrate the hunt here https://github.com/mthcht/awesome-lists/tree/main/Lists/Phishing/DNSTWIST

Careful triage is essential during threat hunting sessions before deploying a detection rule for this purpose, happy hunting !