Test Scenarios
There were several testing scenarios designed to measure
system and user impact.
Resource Consumption Test
35
30
1. Designed to determine what system resources were
being utilized by the installed product after installation but
sitting idle.
• RAM usage is determined by adding the in-use RAM
utilized by new processes/services added to the system
during the install.
• Disk usage is determined by measuring the primary
hard drive Used Space in bytes – from drive properties;
before the installation then after the installation and all
updates have been applied.
Resource Consumption
2500
2000
1500
1000
25
20
15
10
5
0
Benign File Creation 500 (sec)
Brand Y
Brand X
Benign File Creation 1000 (sec)
Benign File Copy mpress packed 500 (sec)
Cylance PROTECT 1.2.1410.60
Malware Detection and Cleaning
1. Malware Samples Copy 300
• This test is designed to judge any system impact by
the security software when copying 300 samples
of malware from a network share. This test utilized
300 samples of malware consisting of 100 random
ransomware samples, 100 random malware samples
greater than 3MB in size, and 100 ransomware
samples that have been packed.
2. Detect and Clean 300
500
0
Brand Y
Benign Files
40
RAM (MB)
Brand X
DISK (MB)
Cylance PROTECT 1.2.1410.60
Benign Files
1. Benign File Creation – 500
• This test is designed to judge any system impact by the
security software on creating new files on the disk. A
simple FOR loop that copies the Windows Media Player
setup file (setup_wm.exe) 500 times renaming the
copied file to setup_wm###.exe. A simple timing batch
script was used to time the creation of these files.
2. Benign File Creation – 1,000
• This test is designed to judge any system impact by the
security software on creating new files on the disk. A
simple FOR loop that copies the Windows Media Player
setup file (setup_wm.exe) 1,000 times renaming the
copied file to setup_wm###.exe. A simple timing batch
script was used to time the creation of these files.
3. Benign File Copy – MPRESS packed 500
• This test is designed to judge any system impact by
the security software when copying 500 files from an
external USB 3 hard drive. This test utilized 500 copies
of the Windows Media Player setup file (setup_wm.exe)
that were all packed with a common packer – MPRESS.
• This test is designed to judge system impact during the
detection and cleaning of the 300 samples copied to the
system from the previous test. Time to detect and clean
was determined by starting the PERFMON data monitor
at the start of the file copy process and was stopped
when the security software had stopped scanning the
files and CPU load returned to approximately 0%.
Malware Detection and Cleaning
2000
1500
1000
500
0
Malware Sample Copy 300 (sec)
Brand Y
Brand X
Detect and Clean 300 (sec)
Cylance PROTECT 1.2.1410.60
3. CPU Impact
• CPU average and max load were determined by creating
a PERFMON data set to gather CPU load (in percentage)
while the installed security software detected and
cleaned the 300 samples of malware.
Competitive System Resource Impact Testing
3
CPU Usage
The biggest differences in the impact on the end-user system
was seen when copying 300 malware samples to the hosts.
100%
80%
60%
40%
20%
0%
The biggest performance differences are in detection
and cleaning.
CPU avg load during D&T 300 (% of CPU)
Brand Y
Brand X
CPU MAX load during D&T 300 (% of CPU)
Cylance PROTECT 1.2.1410.60
Test Results
Brand Y and Brand X have entirely different approaches to
scanning and detecting malware.
Brand Y uses a driver to inject itself into all read/write functions
whether on a local disk, removable storage, or network drive.
Brand Y calls this feature on-access protection and is enabled
by default. Many AV vendors commonly use this method and
it is a major driver in users complaining that the AV is making
their box crawl to a halt. When performing common tasks, such
as unzipping large archives, or copying many files, this type of
system severely impacts performance. As you can see from
the results, in most cases Brand Y’s scanning during these file
creation and copy events, on average, took 16 times longer than
on a CylancePROTECT system.
As Brand Y injected itself in the copy process, it began
detecting malware as soon as the file copy started. While it
was the fastest at detecting and cleaning some of the 300
samples, taking only 203 seconds, the CPUs were essentially
totally consumed during this process. The average CPU load
during the 300-file copy was over 79% and maxed out at
100%. It should also be noted that Brand Y only had to perform
cleaning on 2/3 of the files – as they missed detecting 99 of
the 300 samples. Compared to CylancePROTECT, Brand Y
was 20% quicker in detecting and cleaning 67% of the samples,
however Brand Y’s average CPU utilization during this time
was nearly 20 times that of CylancePROTECT doing the
same task.
Brand X’s results were significantly different than Brand
Y’s. As Brand X does not inject itself into the file copy, the
actual copying of the files happens very quickly. However,
once their deferred scanning begins, their CPU consumption
dramatically increases to an average of 33.6% of the CPU
(maxing out at 100%), and it took them nearly 1,500 seconds
to detect and clean 94.3% of the 300 samples. Compared to
CylancePROTECT, Brand X used over eight times the CPU and
took nearly six times longer to detect and clean the samples.
Brand X however, uses a method called Deferred Scanning.
This is similar to what CylancePROTECT does with the File
Watcher feature — keeps track of files written to disk and
then queues them up for scanning at a later time. This results
in little immediate user interruption, as you can see from the
results where Brand X and CylancePROTECT were pretty
much equal in the file creation/copy tests.
CylancePROTECT utilized, on average, only 4% of the CPU,
which maxed at 23.4%, and took 277 seconds to detect
and quarantine 99.7% of the samples. There was one file
remaining out of the 300. This file was a corrupt file — not a
valid WIN32 application when it was executed.
Brand X and Brand Y use significantly more system
resources (at idle) than CylancePROTECT.
These are other observations seen during the test.
Other Observations During the Test
Both Brand X and Brand Y use significantly more resources on
a host than does CylancePROTECT. CylancePROTECT sits idle
at about 107MB of RAM and about 380MB of hard disk space
consumed with two running processes.
Brand Y
Sitting idle, Brand Y consumes over six times the RAM used
and over five times the disk space (nearly 2GB) consumed by
the installed applications and updates. Brand Y also has 13
processes running.
• Upon reboot, the on-access scanning process consumes
significant CPU resources while loading data. On the
low-end laptop, hard disk access was significant for many
minutes after log in.
Bran
Please complete the form to gain access to this content
