Data corruption because of unexpected paging read

Gavin_Ye · November 28, 2014, 4:57pm

I am creating a file system driver which is similar to fastfat in WDK smaples.
And I got problem when I doing following test, and data is corrupted.

I did some search in this forum, but I cannot find a explanation why following issue happens.
Have anyone seen similar problem or has any idea how it happened?
Really appreciate it!!!

#########################
Test Description

Create new file map.txt
Write 512 bytes (Memory is filled with ‘A’, cache write)
Write Another 512 bytes (Memory is filled with ‘A’) (file size should be 1024, cache write)
Close file
Open the file again
Map the file (CreateFileMapping) with size 2048
CreateFileMapping(hFile, NULL, PAGE_READWRITE, 0, 2048, NULL);
Set memory with ‘B’ from 1024 to 2048.
Unmap
Close file

PROBLEMS:
In step 6, a IRP_MJ_SET_INFORMATION request is sent to extend EOF from 1024 to 2048.
In this request, our FSD call CcSetFileSize to update file size in cache (I notice there is a CcPurgeCacheSection after FSD call CcSetFileSize), and this cause a page fault, and a paging read is issued.
The problem is that there is no paging write before this — ValidDataLength is 1024 because of previous cache write, but ValidDataToDisk is 0 because there is no paging write yet, so the first 1024 bytes (two 512 bytes write) data get lost because fsd cannot read it from disk.
At last, the paging write comes, but FSD already lost the first 1024 bytes, and data corruption happens.
(If test on normal file system, e.g. ntfs, there is no such paging read)

#########################
ProcMon Info
Test on normal file (Without our FSD) - C:\TestData\map.txt

FASTIO_NETWORK_QUERY_OPEN - FAST IO DISALLOWED
IRP_MJ_CREATE - NAME NOT FOUND
Desired Access: Read Attributes
Disposition: Open
Options: Open Reparse Point
Attributes: n/a
ShareMode: Read, Write, Delete
AllocationSize: n/a
IRP_MJ_CREATE ? SUCCESS
Desired Access: Generic Write, Read Attributes
Disposition: OverwriteIf
Options: Synchronous IO Non-Alert, Non-Directory File
Attributes: N
ShareMode: None
AllocationSize: 0
OpenResult: Created
IRP_MJ_WRITE ? SUCCESS
Offset: 0, Length: 512, Priority: Normal
FASTIO_WRITE ? FAST IO DISALLOWED
Offset: 512, Length: 512
IRP_MJ_WRITE ? SUCCESS
Offset: 512, Length: 512, Priority: Normal
FASTIO_MDL_WRITE_COMPLETE ? SUCCESS
Offset: 0, MDL: 0xfffffa8002ff7f40
FASTIO_MDL_READ_COMPLETE ? SUCCESS
MDL: 0xfffffa8003154e90
IRP_MJ_CLEANUP ? SUCCESS
IRP_MJ_CLOSE ? SUCCESS
IRP_MJ_CREATE ? SUCCESS
Desired Access: Generic Read/Write
Disposition: OpenIf
Options: Synchronous IO Non-Alert, Non-Directory File
Attributes: N
ShareMode: None
AllocationSize: 0
OpenResult: Opened
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION ? FILE LOCKED WITH WRITERS
SyncType: SyncTypeCreateSection, PageProtection:
FASTIO_QUERY_INFORMATION ? SUCCESS
Type: QueryStandardInformationFile
AllocationSize: 4,096
EndOfFile: 1,024
NumberOfLinks: 1
DeletePending: False
Directory: False
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
FASTIO_QUERY_INFORMATION ? SUCCESS
Type: QueryStandardInformationFile
AllocationSize: 4,096
EndOfFile: 1,024
NumberOfLinks: 1
DeletePending: False
Directory: False
IRP_MJ_SET_INFORMATION ? SUCCESS
Type: SetEndOfFileInformationFile, EndOfFile: 2,048
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION - SUCCESS
FASTIO_ACQUIRE_FOR_CC_FLUSH SUCCESS
IRP_MJ_WRITE ? SUCCESS
Offset: 0
Length: 4,096
I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O
Priority: Normal
FASTIO_RELEASE_FOR_CC_FLUSH SUCCESS
IRP_MJ_CLEANUP ? SUCCESS
IRP_MJ_CLOSE ? SUCCESS

#########################
ProcMon Info
Test file on shadow volume (with our FSD) - \Device\HarddiskShadowVolume2\TestData\map.txt

FASTIO_NETWORK_QUERY_OPEN - FAST IO DISALLOWED
IRP_MJ_CREATE - NAME NOT FOUND
Desired Access: Read Attributes
Disposition: Open
Options: Open Reparse Point
Attributes: n/a
ShareMode: Read, Write, Delete
AllocationSize: n/a
IRP_MJ_CREATE ? SUCCESS
Desired Access: Generic Write, Read Attributes
Disposition: OverwriteIf
Options: Synchronous IO Non-Alert, Non-Directory File
Attributes: N
ShareMode: None
AllocationSize: 0
OpenResult: Created
IRP_MJ_WRITE ? SUCCESS
Offset: 0, Length: 512, Priority: Normal
FASTIO_WRITE ? SUCCESS
Offset: 512, Length: 512
IRP_MJ_CLEANUP? SUCCESS
IRP_MJ_CREATE ? SUCCESS
Desired Access: Generic Read/Write
Disposition: OpenIf
Options: Synchronous IO Non-Alert, Non-Directory File
Attributes: N
ShareMode: None
AllocationSize: 0
OpenResult: Opened
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
SyncType: SyncTypeCreateSection, PageProtection:
FASTIO_QUERY_INFORMATION ? SUCCESS
Type: QueryStandardInformationFile
AllocationSize: 4,096
EndOfFile: 1,024
NumberOfLinks: 1
DeletePending: False, Directory: False
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
FASTIO_QUERY_INFORMATION ? SUCCESS
Type: QueryStandardInformationFile
AllocationSize: 4,096
EndOfFile: 1,024
NumberOfLinks: 1
DeletePending: False
Directory: False
IRP_MJ_SET_INFORMATION ? SUCCESS
Type: SetEndOfFileInformationFile, EndOfFile: 2,048
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION ? SUCCESS
IRP_MJ_READ ? SUCCESS
Offset: 0
Length: 2,048
I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O
Priority: Normal
FASTIO_ACQUIRE_FOR_CC_FLUSH ? SUCCESS
IRP_MJ_WRITE ? SUCCESS
Offset: 0
Length: 4,096
I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O
Priority: Normal
FASTIO_RELEASE_FOR_CC_FLUSH ? SUCCESS
IRP_MJ_CLEANUP ? SUCCESS
IRP_MJ_CLOSE ? SUCCESS

OSR_Community_User · November 28, 2014, 5:48pm

VDL management can be surprisingly complicated.

What value are you sending into CcSetFileSize for VDL? From what you’ve described it sounds like you are setting the VDL beyond the EOF, which would tell the cache manager there’s now data within that region. To maintain coherency it would either need to discard that partial page (deferring to the file system to have the correct data) or fault in the correct data (though that will happen at some point anyway).

I’m not sure what you mean when you say “I notice there is a CcPurgeCacheSection after FSD call CcSetFileSize” - I certainly don’t see one in the FASTFAT code for setting EOF and there’s no obvious reason why one would purge (discard) dirty data in the cache under such circumstances.

Tony
OSR

Gavin_Ye · November 28, 2014, 6:17pm

Hi Tony,

Thank you for the reply in Holiday.

The fsd doesn’t set VDL beyond EOF. When the Set EOF request comes, the VDL is 1024 (The cache write set this), the ValidDataToDisk is 0 – because there is no paging write yet, the current EOF is 1024, and New EOF is 2048.
CcPurgeCacheSection is not in fsd code. Actually, I saw this when I debug the driver. In Set EOF request, fsd call CcSetFileSize, and this function cause CcPurgeCacheSection called (I guess it is called by Cache Manager), and then it get a page faut, so a paging read is issued by Virtual Memory Manager.

Here are some rules that I think should happen with file size changing:
Can you help me check if there is anything wrong?

ValidDataLength:

When write data (cache, noncache, paging) and end position of write is greater than current ValidDataLength.
The VDL must be less than or equal to FileSize.

FileSize:

SetFileInfo: EOF (Without AdvanceOnly Flag)
Cache write or Noncache Write (Not Paging) which needs to extend file size.

AllocationSize:

SetFileInfo: Allocation
Cache write or Noncache Write (Not Paging) which needs to extend allocation size.

ValidDataToDisk:

Paging write, when end of write is greater than current ValidDataToDisk. It must be less than or equal to FileSize.

In my case, the ValidDataToDisk might be smaller than ValidDataLength, is this suspicious?
When the paging read comes, there is no data has been written to disk yet, so the ValidDataToDisk is zero, and VDL is 1024.

OSR_Community_User · November 29, 2014, 2:19pm

What you should see is a flush first; the purge happens after the flush. Thus, I’d expect you would see an IRP_MJ_WRITE (IRP_PAGING_IO) on the region from 0 to 1023 first. Then the purge discards the contents of the page.

Are you not seeing an IRP_MJ_WRITE? If not, walk through this call (CcSetFileSize/CcSetFileSizeEx) and try to ascertain why the flush isn’t doing anything but the purge is (the flush is done by Mm but the Purge is done in Cc albeit with some Mm interaction).

The IRP_MJ_WRITE should push the VDL out (Alloc/EOF cannot change for paging I/O but VDL does).

Tony
OSR

Gavin_Ye · December 1, 2014, 1:56am

Thank you, Tony.
Let me try.

OSR_Community_User · December 1, 2014, 5:56am

>cache write set this), the ValidDataToDisk is 0

What is this variable? why are you using it? can there be some bug with this variable, which is IIRC not used in FSD/Cc interaction at all?

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

OSR_Community_User · December 1, 2014, 3:37pm

FastFAT demonstrates this - it reflects the fact that the VDL on disk isn’t the same as it is in memory.

If you look at FAT’s definition of the File Control Block (FCB) in fatstruc.h:

//
// Valid data to disk
//

ULONG ValidDataToDisk;

It also tracks the VDL in memory via the common header. Both values are used in the Write path as well as the IRP_MJ_SET_INFORMATION path.

Tony
OSR

Gavin_Ye · December 2, 2014, 2:46am

Update:

The problem has been resolved.
The cause is very interesting — it is not related with VDL/FileSize/AllocSize or CcSetFileSize, it is caused by DesiredAccess in IRP_MJ_CREATE.

After a long time debugging, and by changing test code in many ways, I suddenly found our FSD code always append FILE_READ_DATA and FILE_READ_ATTRIBUTE in IRP_MJ_CREATE.
When test code use only GENERIC_WRITE to open target file and then use PAGE_READWRITE to create file mapping, the problem happens.

After remove those two lines code, the problem goes away.

I haven’t traced into it, but I guess it is because the access rights is not consistent and confuse CC/MM.

Will do more investigation when I have time.

Thank you, everyone!