How to Quickly Identify CPU Spikes’ causes ?

In this short post I want to share with you how to find quicly what cause your CPU spikes.

If you hear your CPU’s fan loudly, it’s a sign of High CPU usage. Usualy we open Windows Task manger to see which process is causing CPU spikes, but when the culprit process is svchost you are not too advanced!
Svchost.exe is a process that hosts windows services and the Task Manager doesn’t give you the possibility to know which service is running inside svchost.exe process !
You have to use instead Process Explorer instead.

procexp_m

In this Example svchost.exe is using 21% of my CPU. To have more details, I just clicked on the Process name and get this Windows.

 

procexp_01_m
The “Thread” panel shows the thread running in this process; in this case the Thread named “wuauserv” is responsible of my CPU spike.
So what is wuauserv? to get more information about this Thread I’ve clicked on the “Module” Button which showed me the following Window;

procexp_04

 

It become obvious that the culprit causing CPU spikes is the Windows Update service.

So, using the right tools can quicly helps you understand what’s happening in your system.

Windows 7 slow logon troubleshooting

Hello,

In this article we will discuss how to troubleshoot Windows slow logon using Windows Performance Recorder (WPR) and Windows Performance Analyzer (WPA).

The case:

Windows logon takes about 1 minute to display the desktop, on a DELL Latitude Laptop, with a I7 CPU and SSD hard disk.

To deal with this issue, I first started Windows Performance Recorder to create boot trace.

You can find in this article how to install and use Windows Performance Recorder https://zinetek.wordpress.com/2015/12/16/how-to-use-wpr-to-record-boot-sequence/

Analyzing the trace

After the system reboot, and the trace created I open it in Windows Performance Analyzer (WPA).

First, lets take a look to the system configuration. Go to tab trace, then system and then General.

WPA_sysconfig

Next click storage

WPA_Sysconfig_storage

As we can see, this Laptop is an I7 3Ghz CPU, 8 Gb of RAM, and SSD hard drive. it becomes obvious that with these specs, 1 min is too long to logon.

So now lets go deeper in our analyzis. first, Add Boot phases graph in the analysis Window.

wpa_boot_phase

We can see that the major delay in the boot trace comes from the winlogon phase (59 s).

Many operations can occur during the WinlogonInit phase; PnP services, Network subsystem, Group policies processing, credentials input can all lead to a delay.

Before we dig in the analysis, lets take a look to the computation and storage graph to find out where to look first. Disk and CPU performance are the must popular causes of slow logon.

According to the two graphics, there is no issue in the cpu or disk usage. So we have to look elsewhere.

WPA_Computation

WPA_Storage

Now we are going to add the Generic events graphic and expand the “Microsoft-Windows-Winlogon”

Genericevnt_winlogon

The “DisplayWelcomScreen” aka CTRL+ALT+DEL was available at 6,84 sec of the trace and the user entered the combination in the keyboard at 7,46 sec of the trace (less than 1 sec for the whole operation).

Next in the “RequestCredentials” task, the user entered his username and password in 4,22 sec (11,68 – 7,46).

all others tasks doesn’t took a lot of time except the “RestoringNetCoonections” that took more than 66 sec. I think we caught our culprit.

So the OS tries to reconnect the network drives during the winlogon process.

After the desktop shows up, I opened the windows explorer and we can see for network drives with a red cross which means that windows was unable to reconnect them. Those drives are mapped on a remote shares.So when the VPN connection is disturbed due to a bad Internet connection, then user experience this dekay when logging in.

Lecteurs réseau 02_modif

To be certain that this is the root cause of our logon’s delay, I disconnected all the network drives and made a new boot trace and here is the result:

WPA_Winlogon_afterresolution

Now that the root cause of the delay was found, the question was, how to make the network drives available to the user without causing major delay in the winlogon process?

The answer I came up with, was to create a script that runs after the user has opened his session.

 

identifying suspicious network activity by using Wireshark and Process monitor

Intoduction:

In this post I’ll discuss on how to identify suspicious network traffic using Wireshark and Process monitor.

The case:

I’ve started the packet capture on my PC with WireShark and one thing captured my attention. I’ve seen many packets intended to a pc that doesn’t exist any more in the network sent by the file server!

NBNS_query_boulwa_Modif

So why the file server with the ip adress 10.x.x.6 is sending NBNS queries (NetBios Name Service) to the host BOULWA-XP asking for his IP adress?

WireShark can show me packets sent from the file server to the specific host, but it can’t tell me which program or service running in the file server that is responsible for this trafic.

To find this program or service I’ve used Process Monitor from SysInternals tool. So I started the capture for a few seconds, then I did a search on the string “BOULWA-XP”. In the result we can see the process name at the origine of the query, in this case it’s spoolsv.exe. Next I’ve applied a filter to have only the traces related to spoolsv.exe

procmon_search_boulwa_modif

spoolsv_filter

spoolsv_filter_01_modif

In the filtred trace, we can see also the spoolsv.exe process accessing the “HKCU\Printers\Connections\,,BOULWA-XP,Microsoft XPS Document Writer” registry key. This means that there is a connection to the printer “Microsoft XPS Document Writer” on the host BOULWA-XP. It can be verified by opening printers location in the control pannel.

panneau_cfg_impr_modif

So by deleting this printer from the control pannel, the network traffic related on this printer disapears from the network.

Cannot scan to shared folder using canon IR copier

The case:

We have a corporate Canon IR copier with scan over network feature enabled. But we cannot scan to the shared folder at the smb server.

Carnet_adresses_modif

So, lets find why.

At the beginning I suspected a permission issue on the share, so I checked the permissions that seems correct, everyone is given read and write acces to the share.

Second, I started process monitor on the computer who hosts the share, but it did not give me any clue. So I deduced that the issue is related to the network and I have to do a captue the packets on the Canon’s Ethernet interface.

To do so, I set up the SPAN (Switch Port Analyzer) feature on the Cisco Catalyst Switch to copy all the network trafic from the Canon’s interface to another interface on which I plugged a PC with Wireshark installed on it.

Here is the wireshark’s capture result:

Wireshark_netbios_issue_modif

We can see the canon copier sending Netbios broadcasts asking for the IP address of the host named AL1XXXX7P (The smb server). As it didn’t receive any response he sends a dns query to the DNS Server asking for the IP of the AL1XXXX7P.DOMAIN.LOCAL host, and the DNS server replies with the IP 10.x.x.13.
Next the copier sends a Netbios query to the 10.x.x.13 host trying to initiate a communication on UDP port 137. the smb server replies that he can’t communicate on port 137.

Ok, now we know why the scanned documents are not sent to the the smb share. So Why the SMB server cannot use his UDP 137 port?

The reason is that, a few days ago, I’ve enabled the “Disable Netbios over TCP/IP” option of the network interface. By enabling Netbios over TCP/IP again I was able to send scans again from the Canon copier to the SMB Server.

NIC_Netbios_disabled

In the following screenshot the Wireshark trace when the the Netbios communication betwen the two devices succeed:

Wireshark_netbios_success_modif

Case of unexplained LDAP requests

In this post I will explaine how I troubleshooted an unusual LDAP requests on the network.

The case:

A computer member of windows domain is sending LDAP requests massively and constantly on the network, to all the domain controllers of all sites.

The computer is on site A (Subunet 192.168.1.0/24) and sends UDP packets on port 389 to the domain controllers located on the following sites:

– B (subnet: 192.168.2.0/24)

– C (Subnet: 192.168.3.0/24)

– D (Subnet: 192.168.4.0/24)

The requests appears on the entreprise Firewall, and the IT manager of one of the remote sites, starts complaining about this unusual udp packets sends to his domain controllers. So I decided to find out what is causing these massive LDAP requests.

Kerio_ldap_modif

The investigation:

In the firewall, I’ve terminated all the UDP requests on port 389 (LDAP) initiated by the host 192.168.1.89. I started Process Monitor (procmon) and waited untill the host starts sending UDP packets to the domain controllers. After that I stopped the procmon’s capture and started my investigation.

The big question was: which process is sending this LDAP requests? And why is it doing it?

So in process monitor, I set a filter on the Path column that contain 389 value.

Filter_path_389

The result of applying the filter is the following:

NB: for confidentiality reasons I hid the original IP addresses and assume that subnets are 192.168.x.x

Filter_path_389_result1

We can notice that the “Lsass.exe” process is sending UDP requests on port 389 to the local and remote domaine controllers.

What is lsass.exe process in fact? Lsass.exe for Local Security Authority Sbsystem is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. It verifies users logging on to a Windows computer or server, handles password changes, and creates access tokens. It also writes to the Windows Security Log (https://en.wikipedia.org/wiki/Local_Security_Authority_Subsystem_Service).

So now we have to find which process is calling lsass.exe to authenticate on the domain controllers?

The answer is found upon above in the trace.

lsaas_read_sophos_regkey

The lsass.exe process is accessing a registry key “SophosSAUALGUCT120000” which is in fact the concatenation of “SophosSAU” for Sophos Antivirus Update program and the netbios name of the Windows PC.

Having this clue between hands, I stopped the Sophos Update service from service manger and observe if there is any LDAP request sent. And there was no LDAP query sent for a while. But when I start the Sophos Update Service the LDAP reapprars quickly.

The culprit was found, but what was causing Sophos Update Service sending this unusual LDAP requests? After having a deep look to the sophos installation in the PC it revealed the the installation was damaged. After reinstalling the Sophos antivirus again the problem was solved.

Conclusion:

Without tools like Process monitor, it would be difficult to troubleshoot this kind of issues.

You can download pocess monitor by clicking: https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx

Troubleshoot access issue to a shared folder using Sysinternals tools

Introduction:

In this post I will show how to troubelshoot an access issue to a shared folder using Sysinternals tools.

The case:

In Symantec Backup Exec, I wanted to add a new storage device as a share. The shared folder was created in a Synology NAS, with domaine administrator rights including my username. But I got the following error: “The path seems to be invalide. Check the server and the paths names”.

Chemain inaccessible_modif

So I tried to access the shared folder with Windows Explorer…and it works!

Nas_Share_From_Explorer

So why Symantec BE is unable to access this share?

To discover it, I run Process Monitor and I take a trace when I clic on the Next button.

In process monitor I do a search on the “\\NASSYNOLOGY\Backup” string and the first result give me a clue. The process “BackupExecManagementService.exe” trying to access the “\\NASSYNOLOGY\Backupp” share is getting “ACCESS DENIED” as a result.

Procmon_access_denied_modif

The next step is to determine which user is running the BackupExecManagementService.exe process. To do so, I use the SysInternals’ Process Explorer tool.

In process Explorer, right clic on BackupExecManagementService.exe, then clic on properties and then on Security tab to get the user name who ran this process.

ProcExp_BE_access_user_modif

Effectively this user were not granted the access rights on the shared folder. By giving it the correct access rights the problem was solved.

You can dowload the Sysinternals suite here: https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx