VMWare Horizon, slow performance with FltPerformSynchronousIo during IRP_MJ_CLEANUP

Hi, all,

We have a VMWare Horizon set up, regular desktop, that attaches shared
folders.
Our filter performs a query for file attributes, delete pending flag and
other “file infos” during IRP_MJ_CLEANUP. We have tested several cases, and
any call via FltPerformSyncronousIo causes extreme performance slowdowns.

Anyone have a clue how to avoid it, or found a workaround? The calls
have to be during CleanupPreOp :frowning: FltPerformSynchronousIo is used instead
of FltQueryInformationFile, to avoid RDPDR bugcheck, since FQIF performs
async I/O.

Kind regards, Dejan.

Noone?

Kind regards, Dejan.

On Wed, Jul 18, 2018 at 10:59 PM Dejan Maksimovic <
xxxxx@gmail.com> wrote:

Hi, all,

We have a VMWare Horizon set up, regular desktop, that attaches shared
folders.
Our filter performs a query for file attributes, delete pending flag
and other “file infos” during IRP_MJ_CLEANUP. We have tested several cases,
and any call via FltPerformSyncronousIo causes extreme performance
slowdowns.

Anyone have a clue how to avoid it, or found a workaround? The calls
have to be during CleanupPreOp :frowning: FltPerformSynchronousIo is used instead
of FltQueryInformationFile, to avoid RDPDR bugcheck, since FQIF performs
async I/O.

Kind regards, Dejan.

Slow performance with high or low CPU utilization?

Have you tried “poor man’s sampling” and just break in with the debugger
periodically during the slowdown to see what the threads are doing?

Is it an option to switch back to FltQueryInformationFile and see if the
perf improves? It would be interesting to know.

-scott
OSR
@OSRDrivers

No CPU utilization whatsoever.
The environment is extremely hard to make, so I am testing remotely. I
would not even know how to get them to set up WinDBG for the machine :slight_smile:
This is not a VMWare Workstation/Player machine, but a desktop
virtualization.

Hmm, I might try that… get back to you tomorrow.

Kind regards, Dejan.

On Wed, Jul 25, 2018 at 6:07 PM Scott Noone <
xxxxx@lists.osr.com> wrote:

> Slow performance with high or low CPU utilization?
>
> Have you tried “poor man’s sampling” and just break in with the debugger
> periodically during the slowdown to see what the threads are doing?
>
> Is it an option to switch back to FltQueryInformationFile and see if the
> perf improves? It would be interesting to know.
>
> -scott
> OSR
> @OSRDrivers
>
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Changing to FltQueryInformationFile does nothing :frowning: The slowdown is still
present.
The difference is extreme. Without the call, it takes 2-3 seconds max. to
list the shares, but with the call (either sync or async) it takes
sometimes 3-4 minutes.

Kind regards, Dejan.

On Wed, Jul 25, 2018 at 6:07 PM Scott Noone <
xxxxx@lists.osr.com> wrote:

> Slow performance with high or low CPU utilization?
>
> Have you tried “poor man’s sampling” and just break in with the debugger
> periodically during the slowdown to see what the threads are doing?
>
> Is it an option to switch back to FltQueryInformationFile and see if the
> perf improves? It would be interesting to know.
>
> -scott
> OSR
> @OSRDrivers
>
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Weird. What network file system is being used? Also, are you above or below
the VMware minifilters?

I’d try putting ProcMon at a very low altitude and see what happens when
your filter does the query. Comparing that to a “normal” system might be
interesting.

It’s more cumbersome, but you can generate crash dumps from snapshots:

http://www.vmwarearena.com/how-to-generate-crash-dump-for-vmware-virtual-machine-guest-os-hung-issues/

In the absence of being able to actually single step this, grabbing a few
snapshots during the long delay might be interesting. My hope would be to
find a call stack or pattern that points to the anomalous behavior.

-scott
OSR
@OSRDrivers

There aren’t VMware minifilters, it appears as an MUP device, when I
connect shares via VMWare Horizon. The shares are similar (to the user,
they are identical) to Remote Desktop shares of the host machine.

Yes, ProcMon below and above my minifilter is the first thing tested
always, and nothing changes for the IRP_MJ_CLEANUPs, not even the duration
seems to change.
If the filter is not present, or FileStandardInformation is not queried
during IRP_MJ_CLEANUP PreOp, then the duration showed by ProcMon is in
microseonds, whereas, when the query is done, it is in milliseconds (IIRC,
ProcMon shows the duration in seconds, i.e. 1.0 is 1 second, right? So our
0.001-0.004 duration is 1-4 milliseconds)
It is not a hang, again, just a major delay, compared to regular processing.

Kind regards, Dejan.

On Fri, Jul 27, 2018 at 10:40 PM Scott Noone <
xxxxx@lists.osr.com> wrote:

> Weird. What network file system is being used? Also, are you above or
> below
> the VMware minifilters?
>
> I’d try putting ProcMon at a very low altitude and see what happens when
> your filter does the query. Comparing that to a “normal” system might be
> interesting.
>
> It’s more cumbersome, but you can generate crash dumps from snapshots:
>
>
> http://www.vmwarearena.com/how-to-generate-crash-dump-for-vmware-virtual-machine-guest-os-hung-issues/
>
> In the absence of being able to actually single step this, grabbing a few
> snapshots during the long delay might be interesting. My hope would be to
> find a call stack or pattern that points to the anomalous behavior.
>
> -scott
> OSR
> @OSRDrivers
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>