How To Use Windows Performance Toolkit to Troubleshoot a Hight Antivirus Software Activity

The first time I began using Windows 10 on my Laptop, I’ve noticed that there is a lot of CPU activity the first minutes after the system’s startup …

The Windows Task Manager shows me that this CPU activity is caused by my Sophos Antivirus !

And I was asking my self, why my Antivirus is using CPU like that ? I guess he is doing his job, by scanning for viruses or other threats …

But, I was wondering, why it do that each time I start Windows ?

So, I decided to use Windows Performance Toolkit to figure this out.

  • First, I’ve restarted my Laptop.
  • Just after I logged in Windows, I’ve started Windows Performance Recorder (WPR) to record a trace.
  • When the High CPU activity happened, I waited a few seconds to allow WPR to record a trace.
    See this post to learn more about recording a trace with WPR https://goo.gl/77vEqb

In WPA I’ll use CPU Precise graph for my investigations.

WPA_Sophos_02.PNG

You can see from the graph, the Antivirus’ High CPU activity, that represent 60% of total CPU activity over time.And by moment it reachs 100%.

My Theory was; There is a process that are doing some thing on the system, may be accessing files and the antivirus is checking this activity. But in the graph I don’t see any other process with a high CPU activity !

To get my answer, I’ll use, the Readying process and the the Waits Columns.
Why ?

The Readying Process, is the Process that make the New Process (in our case SavService.exe) Ready to execute in the CPU. In simple words, the Readying Process tells the New Process: Hi, I finished running, and it’s your turn tu run in the the CPU.

The Waits columns, informs us about the total time the New Process (Sophos AV) spent waiting on the Readying process to finish executing in the CPU, to get chance to execute in turn.

So finding the Readying process with the high waits time, will probably reveals to us the culprit.

WPA_Sophos_Waits_01.PNG

There are many Thread running in the Savservice.exe process context. I’ll expand, the first ones and check for the readying process with the highest waits time.

WPA_Sophos_Waits_02

One process appears with a high wait time. It’s CompatTelRunner.exe process…
How you can interpret that ?

The Sophos Antivirus Thread 4412, waits a total of 28 s, on CompatTelRunner.exe Threads to finish running to get in the CPU to run code.

The same thing is happening for the other Antivirus’ threads.
So there is a big chances that the CompatTelRunner process is causing all this Antivirus activity.

The next step is to identify the CompatTelRunner process and how it starts.

You can do a google search on CompatTelRunner key word and get results telling you that this is the Microsoft Compatibility Telemetry program that collect data of how frequently you use features and applications, system files, and likely more other stuff. more often it causes a high CPU or disk activities. And it’s recommanded that you should disable it.

But let’s continue our troubleshooting as if this not a known issue, just to show the steps to go through to solve this kind of issue using WPT.

So, to identify the Images and the CompatTelRunner.exe processes I’ll uses the Images table and the processes table;

Images_01

In the Images table, you can read in the “file description” column “Microsoft Compatibility Telemetry”. You can read more about this program here http://www.zdnet.com/article/windows-10-telemetry-secrets/

After that I need to know, how this program is started, by using the processes table.

Processes_01

In the “Command line” column, there is this parameter “DoScheduledTelemetryRun”. It seems like a scheduled task.

Now I’ve to go to the scheduled tasks and disable this task. But how to find it, quickly, between the others Windows tasks ?

There is command line that will extract all the scheduled tasks and copy them in a csv file. So that I can import it in an Excel file and get the information I need.

This command is “schtasks /query /fo LIST | clip.exe”
By disabling this task the problem was solved.

*************************************************************************************

Do you want to learn more about Windows Performance Toolkit and how to use it for troubleshoot Windows Performance issues ?
Get a lifetime access to my course for only 10$ with 30 days money back guarantee.
Just click the link https://goo.gl/86bB1D and start learning a new troubleshooting skill…Limited time offer !

*************************************************************************************

 

 

Windows 10 Slow Boot

Since I’ve upgraded my system from Windows 7 to 10 on my HP ProBook 6570b with 16 gigabytes of RAM and an SSD disk, I’m experiencing a slow boot when “restarting” the computer. Windows shows a black screen with the Windows 10 logo in the middle for several seconds.
A trace capture taken with Windows Performance Recorder shows that the “Pre Session Init” boot phase, has taken 22 Seconds. Which is big duration for this phase!

Windows 10 SlowBoot

So what’s going on ?

Using Region of Interest graph and choosing the “Thread Activities” view, you can see that “Boot-PnP-SystemStart-Phase” takes 20 Seconds and the Thread 8 of system process 4 and is involved in this delay.

Windows 10 Slow boot

Let’s take a look at CPU Usage Sampled Table to see what Thread 8 is doing during this amount of time;

Windows 10 Slow boot

When digging in the call stacks, you can see “IopLoadDriver” function call that load Drivers. At this point I can assume that the delay is related to driver loading operation.

So let’s move forward and look for other clues. At the bottom we can see an interesting functions calls related to the Hash validation. These functions are called from CI.dll module which is the Code Integrity Module responsible of Drivers’ signature checking at the boot phase.

We can see this more clearly in the “Generic Events” table. By expanding “ValidateFileHash” taskname under “Microsoft-Windows-CodeIntegrity” column, you can see that the Hash validation starts at 2,433540242s for “\Device\HarddiskVolume2\Windows\System32\drivers\dxgkrnl.sys” file and ends at 22,569235172s !

Windows 10 Slow boot

For all the other drivers the code integrity check operation took a few milliseconds !

So the question is; why it took so long for checking “dxgkrnl.sys” file integrity ?

dxgkrnl.sys is a Microsoft DirectX graphic driver; At the begining I thought that my dxgkrnl.sys was not up to date (10.0.10586.672), beacause I was running Windows 10 1511 version.

Windows 10 Slow boot

So I’ve applied all the latest updates for Windows 10 version 1511; the dxgkrnl.sys files version was updated to “10.0.10586.873”, but the boot time remains the same !

My next step was upgrading to Windows 10 Creator Update, whithout this solving my problem !

I’ve also the same problem on my other HP ZBook Laptop with 16 gigabytes of RAM and an SSD disk !

On the Internet many people reported this kind of issue, but none gave a definitive solution !

The story continues …

What I did next is take a trace from another Windows 10 PC which boot normally, without delay, and make a comparison between the two traces. I created a comparative window and put in the CPU Precise Graph. I drilled in the call stacks of Thread 8 and try to catch any difference between the two call stacks. And what I saw in the top of the stack seemed interesting. There are calls to functions in the Wof.sys module that are not present in the call stack of the computer without boot delay!

Windows 10 Slow boot

Windows 10 Slow boot

Wof.sys is a File System Filter Driver. And according to MSDN page https://msdn.microsoft.com/en-us/windows/hardware/drivers/ifs/what-is-a-file-system-filter-driver- ;
A File system filter Driver is is an optional driver that adds value to or modifies the behavior of a file system. A file system filter driver can filter I/O operations for one or more file systems or file system volumes. Depending on the nature of the driver, filter can mean log, observe, modify, or even prevent. Typical applications for file system filter drivers include antivirus utilities, encryption programs, and hierarchical storage management systems.”

So my guess is when a file I/O is performed on dxgkrnl.sys file it’s intercepted by a program which try to do something with !

So pushing my curiosity further, I took another trace with the “WDF Driver Activity” option enabled.

Windows 10 Slow Boot

The generic Events table shows me a new task which perform just before Dxgkrnl driver initialization. A code integrity applied vmbkmclr.sys driver.
vmbkmclr.sys is a part of Hyper-V backup integration components for VMs. Armed with this information, I made a test by uninstalling the hypervisor from my machine and now the boot time is faster !

The Pre Session Init boot phase goes from 22s down to 8s.

Windows 10 Slow boot

Now the big question is; The Hypervisor is really the culprit ? or a misconfiguration of my system ? As a reminder, I’m facing the same issue on my other windows 10 machine, with the Hyper-V service running on it.

So this is just a workaround not a definitive fix.

 

 

 

 

 

 

Windows Tips and Tricks

  1. Copy a file path to the Clipboard

I use this tip to copy a path of a script into command line without having to type the whole path or when uploading files attachement in Outlook …etc.

To get the file’s path to the Clipboard: hold down the Shift key, right-click the file or folder you want, then select the newly revealed “Copy as Path” option.

tips_copyaspath

Windows store installation error

Introduction:

A few days ago I was facing a strange behavor on my Laptop; I was unable to open the windows’ calculator. It appears briefly and the disapears quickly. And I was so busy to tackle this problem, till yesterday when I took the Bull by the horns.

The solution:

For those looking forward, go to step 5.

  1. I took a trace with Procmon and I found this:

Calculator

It seems something was broken; the Calculator was unable to load some DLLs from “C:\Program Files\WindowsApps\Microsoft.WindowsCalculator_10.1608.2213.0_x64__8wekyb3d8bbwe\” location.

2. I’ve done some Internet research and the suggestions tend to the reinstallation of the calculator using the Windows store App; unfortunaly, even the Windows store App was do not work properly 😦

3. To Re-install the Windows store application, I’ve used this powershell command:

Get-AppXPackage *WindowsStore* -AllUsers | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register “$($_.InstallLocation)\AppXManifest.xml”}

and guess what! I’ve got the following error:

powershellerror80070002
I got here two errors code: 0x80073CF9 and 0x80070002.

4. To have the corresponding error message use the following command:

winrm helpmsg ERROR_CODE
winrmhelpmsg
The last error message informs us that the setup failed beacause of a missing file. And if we read carrefuly the full error message we are prompted to use the following command to get more details;

Get-AppxLog -ActivityID 9535eb18-6e56-0000-8304-3695566ed201

The result is a log of the setup process and at the end we get this information:

Opening the registry key Software\Microsoft\Windows\CurrentVersion\Appx\AppxAllUserStore\\S-1-5-21-705754729-2888756721-3051348587-1047 fails with the following error message 0x80070002

So what is S-1-5-21-705754729-2888756721-3051348587-1047 ?

It’s a user SID (Security Identifier). So I’ve a user SID missing in the registry! but what user this SID relates to?

The following command gives me the answer: wmic useraccount get sid,name

wmic_sid

The SID in question is not on my system. It seems that is a SID of a domain’s user that was not properly deleted after the computer unjoigned from the domain.

5. At this point, two options are available:

  • Install the Windows store for the current user only with this powershell command: Get-AppXPackage *WindowsStore*| Foreach {Add-AppxPackage -DisableDevelopmentMode -Register   “$($_.InstallLocation)\AppXManifest.xml”}
    (Without -Allusers parameters)
  • Add the missing registry key and run the powershell setup command for all computer’s users.

I’ve shoosed the seconde one to be able to install Windows store for all users’ acounts, and after that I was able to install the Calculator from the Windows Store.

PowerPoint Opening file error

Few days ago a user reported to me a powerpoint opening file error. He has received an email with a powerpoint file as attached piece, and when he tried to open it he got this message:

PowerPoint found a problem with content in filename.pptx.
PowerPoint can try can attempt to repair the presentation.
if you trust the source if this content, click repair.

PowerPoint_error_Msg_01

And even when he clicked on the repair button he got an other message “Sorry…PowerPoint can’t read filename.pptx

 

PowerPoint_error_Msg_02

So, I told him to send me the file and I opened it successfuly on my computer.

First deduction, the problem is not related to the powerpoint file but in the PC’s configuration.

First of all, I did some search on google and I found some workarounds that suggest to disable the “Enable protected view for files originating from Internet” option in the “Trust Center > Protected View” Menu.

Office_TrustCenter_Menu

By doing so, it works but it’s not safe to let this option disabled. So I decided to deal diffrently with this problem.

To figure out what is the problem, I’ve used Process Monitor and recorded a trace when opening the file on the user’s PC.

First thing to do in Procmon, create a filter on the process name = POWERPOINT.EXE. Thus we restrict our investigation scope to the process we are interested in.

ProcMon_Filter_on_PPT

 

Next, we need to apply a filter on the result by going to the menu “Tools > Count values Occurences”

We need to look in the events by applying a filter on those that we consider interesting. I mean events like “Access denied, Name Invalid, Path not found and sharing violation”.

ProcMon_Occurences_Result_01

After a few investigations, I’ve found something interesting when applying the “Path not found” filter. PowerPoint is looking for “MSPPT.OLB” file in “C:\Program Files\Microsoft Office\Root\Office16\”

ProcMon_PathNotFound_Filter

Knowing that all our computers are using MS Office 2010 or 2013. Why PowerPoint 2013 is trying to open a file that seems to belong to an Office 2016 installation that should not to be here.
By asking the user this question, he admitted that he has installed Office 2016 to test and removed it after.

Now we need to find where PowerPoint take the Office16 path reference instead of Office15.

by doing a search using a key word “Office16” we find this:

Office16_ref

Office16_ref_01

The Office16 path is registered in the registery base.

When opening the regedit utility and seeking for the indicated key, we can notice that above the “2.c” directory exists two others “2.a” and “2.b” each one tip repectively on the following path: “C:\Program Files (x86)\Microsoft Office\Office14\MSPPT.OLB” for Office 2010 and “C:\Program Files\Microsoft Office\Office15\MSPPT.OLB” for Office 2013.

So my guess is; when PowerPoint looks for the MSPPT.OLB’s path it moves to the last folder under “HKEY_CLASSES_ROOT\TypeLib\{91493440-5A91-11CF-8700-00AA0060263B}”. And as Office 2016 have been deleted by the uninstaller, the folder “C:\Program Files\Microsoft Office\Office16” do not exist any more.

By deleting the “HKEY_CLASSES_ROOT\TypeLib\{91493440-5A91-11CF-8700-00AA0060263B}\2.c” key, PowerPoint opened the file without problem.

 

Process Monitor is a powerfull tool that can help system administrators to solve some tricky issues.

Thank you for reading.

Detect and remove ransomware

Yesterday When checking the firewall logs, something scary has captured my attention.

I’ve seen a local IP adress trying to get connected with a Command and Control (CnC) server. these servers are known to deliver the encryption key to ransomware which encrypts user files and ask the user to pay for the unencryption key.

fortunatly, our firewall has banned this connection.

K_Ctrl_Sec_Log_modif

As you can see, in the firewall’s security log there is a connection attempt to the IP 208.100.26.234 on the TCP 80 port, every 30 minutes. The IPS module of the firewall has detected this traffic as malicius and has dropped the packet.

you had better to have a good up to date firewall to protect your network.

So now, I’ve the IP of the comprmised PC, but how to identify the process sending these packet to the CnC server? I simplly applied the Mark Russinovich formula “When in doubt… run process monitor“. I did and after a few moment of capturing, I ‘ve done a search on the 208.100.26.234 and I got this:

ProcMon_01

The culprit is rafdpklsxd.exe. it has, probably infected the computer through a mail sent to the user with an attached file like MS Word containing a macro. we have been bombed with this kind of email the past days.

So if you are in charge of managing the security in your compagny, you must educate your users to not open attached files in emails sent by unknown senders.

But a question nagging me. Why the anti-virus hasn’t detected and stopped this malware? Unfortunately the antivirus wasn’t working properly on that PC. A chance that the Firewall has done his job properly otherwise this user would have lost his files.

In conclusion, take sariously the ransomware threat by educating your users, having a good firewall and be sure your antivirus is working properly.

You can find here informations about the ransomwares and how to prevent an infection;

https://www.lexsi.com/securityhub/un-nouveau-vaccin-dynamique-contre-locky/

http://past.is/atdag

http://past.is/atdae

 

 

 

How to use WPR to record Windows boot process

 

In this article I will show you how to use “Windows Performance Toolkit” to make a trace of Windows boot sequence, in order to troubleshoot slow logon.

First of all, you need to download the Software Development Kit (DSK) https://dev.windows.com/en-us/downloads/windows-10-sdk

After running the sdksetup.exe you should select one of the following options:

SDK setup

The first option will install Windows performance tool kit on the computer running the setup. The second one, will allow you to download an offline setup files that can be executed on an other computer.

For our purpose we will chose the first one.

Click next and accept the license agreement.

SDK setup 01

Click on “Windows Performance Toolkit” and then install.

Reboot your computer to finish the setup.

To make a trace of your boot sequence, type “wpr” from the windows start menu and then click on “Windows Performance Recorder”.

run_wpr

wpr01

On the “Performance scenario” menu choose “Boot”.

wpr02
Type “1” for the numbers of iterations and then click the sart button.

wpr03

Select the path where the trace file (.etl) will be saved and click on the “Save” button.

wpr04

After you click on the OK button your system will reboot and “Windows Performance Recorder” will record all the boot phase.

After you open your windows session WPR will end the trace and will save the file in the specified path.

Boot_trace_inprogress

Generaly the trace file will be a hundred of Mb till Giga bytes.

So if you want to share your trace or send it by e-mail, don’t forget to compress it.