Know the Truth


We believe that the only true test for security products is one that replicates the real world. While third party antivirus efficacy reports and sandboxed product demos are useful, they fail to take into account the rapidly changing tactics of attackers. Cylance’s goal is to empower System Administrators, IT specialists and engineers with the tools, knowledge and malware to test any vendor’s product in a safe and secure manner.

*Available in the U.S., expanding to other regions soon.

What is the Cylance® Threat Testing Framework?

Test For Yourself is a framework that provides a safe testing environment for malware protection products. We feel that the only valid way to test this is using fresh malware samples in a safe but realistic scenario tailored to your environment.

When it comes to endpoint protection effectiveness, test for yourself. Download the book to learn how to set up your environment and test the products for real-world conditions.


You would not build a house without a plan, and you should not detonate malicious software without one either. These articles will help provide you with a blueprint to test antivirus products.


In order to capture the best results, you will want to use the best tools available. We provide you with information regarding tools that will allow you to analyze test results in a safe and effective manner.


Testing with old, well known malware is a surefire recipe for a false sense of security. Here we provide instructions to obtain fresh malware samples that represent the most current threats.


  • Testing for Dummies: Offline Test

  • Testing for Dummies: Random Mutation

  • Testing for Dummies: Holiday Test

  • Testing for Dummies at a Glance - Infographic

  • Virtual Machine Setup – Best Practices

    Malware variants exist that halt execution and evade detection by employing Virtual Machine (VM) evasion methods. However, those samples are becoming few and far between as more companies employ virtual environments in production – which means that malware creators can’t be confident that the VM that their software is evading is not actually their intended target.

    There are many benefits to testing in a virtual environment. For example, a security analyst can quickly snapshot a VM and conduct tests with the confidence that any changes made are easily reverted by restoring to the VM state captured in that snapshot.

    In practice, accurate results are a product of recreating production environments as accurately as possible. This can be achieved with software that will virtualize a physical machine, or you can create a fresh VM with a base OS image. Note: if a VM is created from an OS image, make sure to install all software that is in the base image to best mirror the production environment.

    The VM should next be updated to include all recent security patches. Make sure to remove or disable any other AV product that you do not wish to evaluate. This step is critical in order to demonstrate the true ability of the product being tested. Network isolation should not be of concern during this portion of the set up as the VM should be clean and free of any malware.

    Once the virtual environment has been established, install the AV of your choice for testing and ensure it has been updated with the policies you intend to run within your environment. Check to ensure the AV product is up to date and running the policy of choice. Virtualization software, such as VMware, should also be updated to the most recent version.

    IMPORTANT: Isolating the VM from the physical device hosting it will ensure that if the malware infection is undetected, it is fully contained. Device isolation is accomplished via the VM’s settings and network configuration. Always check the VM settings to make sure that if any folders are shared, the permissions are set to Read-Only (RO). Also check to see whether the VM permits drag and drop and/or copy and pasting; make sure those features are disabled.

  • Types of Networking Interfaces

    Typically, virtualization Frameworks provide three types of networking interfaces: network address translation (NAT), host-only or custom network and bridged.

    • A NAT interface will allow access to the physical network by sharing the address of the hosting machine. This setup provides access to the Internet through the physical infrastructure. Typically, your physical device will be assigned the address of X.X.X.1 on the NAT network. This puts the physical device and your VM on the same virtual network through virtual interfaces on the physical device. This means that the VM can only communicate out of the virtual network. The hosting device is performing NAT and the external devices to the virtual network have no routing information back to the VM. This network configuration is best used to allow VMs access to the physical network, while limiting inbound network traffic.
    • A Host-Only or Custom Network interface will allow for communications between all devices located on that network segment. It is important to note that these devices should not have access to the Internet or physical network. To better isolate the VM, the physical device’s virtual interface can be removed from these network segments. This configuration is best used to set up a virtual network that will be isolated from the physical network.
    • A Bridged interfaced will place the VM’s virtual network directly in the physical network. This configuration allows all devices on the physical network to communicate with the VM. There are almost no circumstances that this configuration should be used while testing an AV product or any malware.
  • Choosing the Best Interface

    It is strongly recommended to always test on a network that is separated from production. Host-Only or Custom Network interfaces are established if the hosting device is segmented from the production network and no Internet connectivity is necessary. Configure a NAT interface if Internet connectivity is required for testing.

    For more complex and secure setups, a hybrid of the two interfaces can be used. For instance, the testing virtual machine (VM) and an OS Firewall VM such as PfSense could be placed on a Host-Only or Custom Network segment. Once this is established, configure the Firewall VM to have an additional interface attached to the NAT network. The Firewall VM should be configured to route AV test VM traffic from the Host-Only or Custom Network through to the NAT network, while strict firewall rules should be in place to ensure that traffic does not communicate with any of your physical devices.

    Once networking has been set up, start the task manager within the VM and leave the process list visible during testing. A snapshot of the VM should be taken once all available updates have been applied. At this point, it is safe to introduce malware to the VM.

  • Introducing Malware to the Environment

    Malware can be introduced onto an endpoint via a number of different channels. This could be through the network, on infected USB drives or as malicious files stored in shared folders. Always zip and password-protect malware while transferring malicious files between devices.

    Scan the malware folder with the AV product under evaluation once the malware is placed on the test virtual machine (VM). This should result in a static conviction of the files. It is important to keep in mind that not all files are statically malicious and thus require a second stage to ascertain their designed intent, which is initiated by executing the malicious file. Prior to executing the malware, the analyst should take another snapshot of the VM just in case a sample infects the VM undetected by the AV product. In this case, the VM can be reverted but work will not be lost.

  • The Formula for Success

    Testing AV products can be performed in a safe and secure manner if the tester follows best practices. Testing in a VM that is isolated from the host device, as well as isolated from the physical network ensures that a security analyst can detonate malware safely and in a manner that yields the most accurate test results.

    These test results allow security professionals to properly vet available solutions for their networks and devices, and it is therefore imperative that these tests reflect the environment that the tested products will eventually protect. It cannot be recommended enough that test environments accurately mimic your organization’s production environments.

  • Safely Handling Malware

    Let's state the obvious, the "mal" in malware is an abbreviation of "malicious". We'll spare you the dictionary definition, but the short version of the story is that it is out to hurt your machine. It therefore stands to reason that you should be extremely – emphasis: extremely – careful while handling malware. Here are a few measures you can take

    1. 1.Always keep files zipped and protected with the industry-standard password 'infected'. This helps to ensure that files are not accidentally clicked and that in order to access them, one has to implicitly acknowledge that the contents are malicious
    2. 2. Never send malware samples via e-mail. E-mail provides opportunities for samples to be released to unintended parties. There is also the risk intention to share testing resources will be construed as an attempt to infect the recipient. Organizations do deploy anti-malware measures on mail servers, and this could get you flagged. It's better to share via repositories or carefully secured USB drives
    3. 3. Keep a working directory and a storage directory in your test environment. This ensures that you are being intentional about the malicious files you are testing. To maintain hygiene, be sure to follow the following rules:
      1. a. Move malware you intend to test to your working directory
      2. b. Only detonate malware from your working directory
      3. c. Always move malware you do not intend to test back to storage
      4. d. Consider removing or altering file extensions (see below)
    4. 4. Remove file extensions or add an invalid file extension to malicious files. In Windows Explorer, be sure to have file extensions visible. You can simply delete file extensions by highlighting them. This, however, is pretty unwieldy as you might be dealing with hundreds or possibly thousands of files. To do so in the command window:
      1. a. On your keyboard, hold the Windows key and press the 'R' key
      2. b. In the run dialog, type "cmd.exe", a command window should open
      3. c. Alternatively, you can type "cmd" in the Start menu search bar and the command prompt option should show in the results
      4. d. Change directories to the directory in which you're storing malware. For this tutorial, we'll use "C:\malware"
        1. i. Type: "cd C:\[desired path]", in this case, "cd C:\malware"
      5. e. Enter the command "dir", which will list all the files that are in that directory. Verify that these are the malware files you wish to rename
      6. f. We are going to add words to the end of a file extension, which is going to cause Windows to consider this a type of file that can't be executed. For this example, we'll assume that we have a directory of .exe files
        1. i. Type: "ren *.exe *DoNotRunMalware"
        2. ii. The files would then have an extension of .DoNotRunMalware, E.G.: malwareSample_1.DoNotRunMalware
        3. iii. If you were to double click on this file, Windows would not run it and show a prompt asking which application you'd like to open it with
      7. g. Wait until the cursor reappears. If you are renaming many files, this may take a few minutes
      8. h. Enter the "dir" command again and verify that the file names have been changed
      9. i. Perform the reverse to rename all of the files to .exe
        1. i. Type: "ren *.DoNotRunMalware *.exe"
      10. j. To rename a single file, enter the following:
        1. i. For the file malware1.DoNotRunMalware, " ren malware1.DoNotRunMalware malware1.exe"
    5. 5. Work in an AV excluded directory if you operate malware on your host. We strongly recommend against operating malware unless you're in a virtualized environment. Be sure to exclude the directory in which the malware resides if you're going to do so. Be very, very careful if you do this
    6. 6. Remove executable rights from the directory you store malware. This provides an extra layer of protection in that you cannot accidentally detonate stored malware. This is especially helpful if you decide against changing the file extensions. Here is a great tutorial on how to do this

  • Mutating Samples

    Mutating malware is the process of changing existing malicious software without significantly altering its functionality. This is often performed to change that piece of malware's hash (also known as the message digest). Mutation allows malware to evade signature-based antivirus (AV) solutions as these often rely heavily upon a collection of hashes in order to identify threats.

    As a security professional, learning to mutate malware will allow you to better vet endpoint protection solutions as you can create unique malware – from a hash and signature perspective – for your tests.


  • Virtualization

    Virtualizing your test environment is ideal – bordering on necessary – and provides several advantages.

    • 1. You can easily snapshot a baseline (uninfected) image. This eases restoring a compromised machine to an uninfected state, thus decreasing downtime between tests.
    • 2. Set up multiple operating systems on one machine. You can establish several test environments on one physical machine, which can include different operating systems – I.E.: Windows 7, Windows 8, OS X, etc. - as opposed to requiring several physical machines.
    • Windows and Linux Hosts

      VMware Workstation Pro
    • Apple OS X Hosts

      VMWare Fusion Pro
  • Monitoring Tools

    Monitoring tools allow you to discover if your machine has been infected and gives you a glimpse into how the malware is affecting your test machine.

    • Windows monitoring tools

      DiskMon is an application that logs and displays all hard disk activity on a Windows system.

      Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity.

      Portmon for Windows: Portmon is a utility that monitors and displays all serial and parallel port activity on a system.

      Windows Performance Monitor is a Microsoft Management Console (MMC) snap-in that combines the functionality of previous stand-alone tools including Performance Logs and Alerts, Server Performance Advisor and System Monitor. It provides a graphical interface for customizing Data Collector Sets and Event Trace Sessions.

    • Apple OS X monitoring tools

      Activity Monitor: Apple’s description “shows the processes that are running on your Mac, so you can see how they affect your Mac's activity and performance”.

    • Terminal commands

      top: Display and update sorted information about processes.

      iotop: Display top disk I/O events by process.

      Nettop: Displays a list of sockets or routes.

  • Network Monitoring Tools

    Network monitoring tools allow you to examine if and how malware communicates over a network, which includes the Internet. Most pieces of malicious code include some sort of network functionality, and this can give you a good sense of how a remote attacker communicates with infected machines

    • Suggested tool

      WireShark: is the world's foremost network protocol analyzer.
  • Compression Tools

    Compression tools will allow you to open archived files in order to parse the contents. Malware often includes several components packaged (compressed) into a single archive file. On Windows, .zip files are commonly found, but numerous other formats exist.

    • Windows Tools

      7-Zip: 7-Zip is an open source file archive tool which allows password protection and encryption.
    • Mac OS X

      iZip: is a file archive tool which allows password protection and encryption.
  • Mutation Tools

    1. 1. PESpin v1.33: PESpin is free and provides an easy to use graphical interface for mutating one sample at a time. Unfortunately, the download link is broken. Search for an alternate download source and compare the hash to verify file integrity.

      1. a. pespin133.rar (SHA256 c0531ef573eb7127d32545ae9c70da796f1fecac488360bb42d4aa7e6bcc1c78)
    2. 2. Hyperion v1.2: As an innovative AV evasion technique, this crypter creates a weak key that isn’t saved, then brute forces it at runtime. To compile on Windows install MinGW, then choose the GNU C++ compiler package.

      1. a. Note: you’ll need to run the compiled crypter from the root of the Hyperion directory in order for dependencies to load correctly.


Cylance gathers an untold number of real-world malware samples every day for use in the Cylance Threat Testing Framework. These samples are refreshed every 24 hours to provide testers with the latest threats we’ve seen. However, testers DO NOT need to rely on our samples for testing. There are a number of other options to choose from. For those that would like Cylance to provide you with samples and a copy of CylancePROTECT® to test, please fill out this form and a Cylance expert will unlock access to our malware repository.

We are currently working on improving this experience. Please check back later.

We use cookies to provide you a relevant user experience, analyze our traffic, and provide social media features. Read More