ZwMapViewOfSection on compressed file?

I’m have a problem with a ZwMapViewOfSection call on a particular file. My
code has worked with thousands of other calls for other files. I noticed
that the file is compressed so I’m wondering if there is something I should
know about ZwMapViewOfSection with compressed files.

The error returned is STATUS_MAPPED_ALIGNMENT, which says my base address or
section offset is not aligned to an allocation size boundary.

For the base address I am passing a null ptr to let the system assign it. I
get the AllocationSize value from a FileStandardInformation class query
using FltQueryInformationFile call. I then adjust my section offset to be a
multiple of the allocation size.

The weird thing is that the AllocationSize I am getting from
FileStandardInformation, and using for the alignment, is 0x0001c000, which
seems like a really strange size. In any case the offset I want is
0x00021000 so aligning it amounts to using the section offset value of
0x0001c000, which is certainly aligned with 0x0001c000.

Can anyone shed any light on this problem or suggest how I should proceed?

I just figured out that the AllocationSize value from the
FileStandardInformation struct is not the granularity, like I thought, but
rather the entire allocation for the file. I’m surprised my code worked as
well as it did. I guess the total allocation would usually (for
non-compressed files) be a multiple of the granularity so my code worked,
even though it was mapping much bigger views of sections than it needed.

This still leaves the question of what I should use for the allocation
granularity. How do I query that? The docs implied I could pass any value
to ZwMapViewOfSection and it would adjust for the granularity, but I found
out right away that I needed to adjust the section offset myself before
calling it.

I just tried 65K and it seems to be working, but it bothers me that I don’t
really know the correct value. I got the 65K number from several places on
the web that talked like that was the correct value.

“Mark Hahn” wrote in message news:xxxxx@ntfsd…
> I’m have a problem with a ZwMapViewOfSection call on a particular file.
> My code has worked with thousands of other calls for other files. I
> noticed that the file is compressed so I’m wondering if there is something
> I should know about ZwMapViewOfSection with compressed files.
>
> The error returned is STATUS_MAPPED_ALIGNMENT, which says my base address
> or section offset is not aligned to an allocation size boundary.
>
> For the base address I am passing a null ptr to let the system assign it.
> I get the AllocationSize value from a FileStandardInformation class query
> using FltQueryInformationFile call. I then adjust my section offset to be
> a multiple of the allocation size.
>
> The weird thing is that the AllocationSize I am getting from
> FileStandardInformation, and using for the alignment, is 0x0001c000, which
> seems like a really strange size. In any case the offset I want is
> 0x00021000 so aligning it amounts to using the section offset value of
> 0x0001c000, which is certainly aligned with 0x0001c000.
>
> Can anyone shed any light on this problem or suggest how I should proceed?
>
>

Am I wrong that AllocationSize parameter is optional?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Mark Hahn”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Tuesday, July 12, 2005 11:03 AM
Subject: [ntfsd] ZwMapViewOfSection on compressed file?

> I’m have a problem with a ZwMapViewOfSection call on a particular file. My
> code has worked with thousands of other calls for other files. I noticed
> that the file is compressed so I’m wondering if there is something I should
> know about ZwMapViewOfSection with compressed files.
>
> The error returned is STATUS_MAPPED_ALIGNMENT, which says my base address or
> section offset is not aligned to an allocation size boundary.
>
> For the base address I am passing a null ptr to let the system assign it. I
> get the AllocationSize value from a FileStandardInformation class query
> using FltQueryInformationFile call. I then adjust my section offset to be a
> multiple of the allocation size.
>
> The weird thing is that the AllocationSize I am getting from
> FileStandardInformation, and using for the alignment, is 0x0001c000, which
> seems like a really strange size. In any case the offset I want is
> 0x00021000 so aligning it amounts to using the section offset value of
> 0x0001c000, which is certainly aligned with 0x0001c000.
>
> Can anyone shed any light on this problem or suggest how I should proceed?
>
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

Mark,

You could ask for as little as a page, but Mm will consume an entire 64K
address range for that one page (that’s an implementation detail, of
course) which is one reason why the 64K number is frequently used -
that’s the smallest contiguous chunk Mm will use.

The cache manager uses 256KB views (on x86, I haven’t verified this is
the same on x64) because *it* has a limited addressable range (x86 -
2048 simultaneous views * 256KB/view = 512MB). Were I coding this
today, I wouldn’t choose less than 64KB. If you are reviewing the
entire file, it might make sense to choose a larger size - like 1MB.
Much would depend upon the target platform and the number of
simultaneous files being accessed plus the limitations on the size of
the address space. You might even want to use a fall-back algorithm
(try a big size and if there’s no address space, fall down to a smaller
size)

So, choose a value that fits for your implementation and you’ll be fine.
If you are happy with 64KB, go for it (I am curious where your address
space is being consumed - mapped view space is not plentiful in kernel
mode, but being unable to do < 256KB seems rather unusual. Are you
mapping a lot of files?)

One trick to extend the address space size is to map them into user
space for a particular process. For example, there’s usually a 2GB
address space between 0-2GB range that people don’t use in the system
process. I also know of products that spawn their own separate system
process precisely to get an empty 2GB address space. The disadvantage
of course is that you must be IN the correct process context to access
those addresses.

This won’t be a big problem in x64 though, because all of these older
limits have been dramatically relaxed.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Hahn
Sent: Tuesday, July 12, 2005 4:29 AM
To: ntfsd redirect
Subject: Re:[ntfsd] ZwMapViewOfSection on compressed file?

I just figured out that the AllocationSize value from the
FileStandardInformation struct is not the granularity, like I thought,
but
rather the entire allocation for the file. I’m surprised my code worked
as
well as it did. I guess the total allocation would usually (for
non-compressed files) be a multiple of the granularity so my code
worked,
even though it was mapping much bigger views of sections than it needed.

This still leaves the question of what I should use for the allocation
granularity. How do I query that? The docs implied I could pass any
value
to ZwMapViewOfSection and it would adjust for the granularity, but I
found
out right away that I needed to adjust the section offset myself before
calling it.

I just tried 65K and it seems to be working, but it bothers me that I
don’t
really know the correct value. I got the 65K number from several places
on
the web that talked like that was the correct value.

“Mark Hahn” wrote in message news:xxxxx@ntfsd…
> I’m have a problem with a ZwMapViewOfSection call on a particular
file.
> My code has worked with thousands of other calls for other files. I
> noticed that the file is compressed so I’m wondering if there is
something
> I should know about ZwMapViewOfSection with compressed files.
>
> The error returned is STATUS_MAPPED_ALIGNMENT, which says my base
address
> or section offset is not aligned to an allocation size boundary.
>
> For the base address I am passing a null ptr to let the system assign
it.
> I get the AllocationSize value from a FileStandardInformation class
query
> using FltQueryInformationFile call. I then adjust my section offset
to be
> a multiple of the allocation size.
>
> The weird thing is that the AllocationSize I am getting from
> FileStandardInformation, and using for the alignment, is 0x0001c000,
which
> seems like a really strange size. In any case the offset I want is
> 0x00021000 so aligning it amounts to using the section offset value of
> 0x0001c000, which is certainly aligned with 0x0001c000.
>
> Can anyone shed any light on this problem or suggest how I should
proceed?
>
>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

> I just tried 65K and it seems to be working, but it bothers me that I

don’t really know the correct value. I got the 65K number from several
places on the web that talked like that was the correct value.

The allocation granularity can be retrieved using NtQuerySystemInformation
with SystemBasicInformation. The result structure (SYSTEM_BASIC_INFORMATION)
looks like this:

typedef struct _SYSTEM_BASIC_INFORMATION
{
ULONG Unknown;
ULONG MaximumIncrement;
ULONG PhysicalPageSize;
ULONG NumberOfPhysicalPages;
ULONG LowestPhysicalPage;
ULONG HighestPhysicalPage;
ULONG AllocationGranularity;
ULONG LowestUserAddress;
ULONG HighestUserAddress;
ULONG ActiveProcessors;
UCHAR NumberProcessors;
} SYSTEM_BASIC_INFORMATION, *PSYSTEM_BASIC_INFORMATION;

L.

> Am I wrong that AllocationSize parameter is optional?

There is no AllocationSize parameter. The call just requires that the
section offset parameter be aligned. The docs say that if you don’t align
it, it will be aligned internally, but when I tried this it did not work and
I got alignment errors.

Have you successfully used ZwMapViewOfSection without adjusting the section
offset yourself? Sometimes I have a problem early in my project but then
later I find the problem wasn’t real. I assume in these cases I did
something wrong in my early debugging, but I never know for sure.

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
> Am I wrong that AllocationSize parameter is optional?
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
> ----- Original Message -----
> From: “Mark Hahn”
> Newsgroups: ntfsd
> To: “Windows File Systems Devs Interest List”
> Sent: Tuesday, July 12, 2005 11:03 AM
> Subject: [ntfsd] ZwMapViewOfSection on compressed file?
>
>
>> I’m have a problem with a ZwMapViewOfSection call on a particular file.
>> My
>> code has worked with thousands of other calls for other files. I noticed
>> that the file is compressed so I’m wondering if there is something I
>> should
>> know about ZwMapViewOfSection with compressed files.
>>
>> The error returned is STATUS_MAPPED_ALIGNMENT, which says my base address
>> or
>> section offset is not aligned to an allocation size boundary.
>>
>> For the base address I am passing a null ptr to let the system assign it.
>> I
>> get the AllocationSize value from a FileStandardInformation class query
>> using FltQueryInformationFile call. I then adjust my section offset to
>> be a
>> multiple of the allocation size.
>>
>> The weird thing is that the AllocationSize I am getting from
>> FileStandardInformation, and using for the alignment, is 0x0001c000,
>> which
>> seems like a really strange size. In any case the offset I want is
>> 0x00021000 so aligning it amounts to using the section offset value of
>> 0x0001c000, which is certainly aligned with 0x0001c000.
>>
>> Can anyone shed any light on this problem or suggest how I should
>> proceed?
>>
>>
>>
>> —
>> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>>
>> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>

Thanks for the useful info.

I am just doing small reads and performance is not a big issue, so I am
mapping just the section and view I need for each file access.

Are you saying that the internal alignment requirement is just a 4K page?
That has not been my experience.

“Tony Mason” wrote in message news:xxxxx@ntfsd…
Mark,

You could ask for as little as a page, but Mm will consume an entire 64K
address range for that one page (that’s an implementation detail, of
course) which is one reason why the 64K number is frequently used -
that’s the smallest contiguous chunk Mm will use.

The cache manager uses 256KB views (on x86, I haven’t verified this is
the same on x64) because it has a limited addressable range (x86 -
2048 simultaneous views * 256KB/view = 512MB). Were I coding this
today, I wouldn’t choose less than 64KB. If you are reviewing the
entire file, it might make sense to choose a larger size - like 1MB.
Much would depend upon the target platform and the number of
simultaneous files being accessed plus the limitations on the size of
the address space. You might even want to use a fall-back algorithm
(try a big size and if there’s no address space, fall down to a smaller
size)

So, choose a value that fits for your implementation and you’ll be fine.
If you are happy with 64KB, go for it (I am curious where your address
space is being consumed - mapped view space is not plentiful in kernel
mode, but being unable to do < 256KB seems rather unusual. Are you
mapping a lot of files?)

One trick to extend the address space size is to map them into user
space for a particular process. For example, there’s usually a 2GB
address space between 0-2GB range that people don’t use in the system
process. I also know of products that spawn their own separate system
process precisely to get an empty 2GB address space. The disadvantage
of course is that you must be IN the correct process context to access
those addresses.

This won’t be a big problem in x64 though, because all of these older
limits have been dramatically relaxed.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Hahn
Sent: Tuesday, July 12, 2005 4:29 AM
To: ntfsd redirect
Subject: Re:[ntfsd] ZwMapViewOfSection on compressed file?

I just figured out that the AllocationSize value from the
FileStandardInformation struct is not the granularity, like I thought,
but
rather the entire allocation for the file. I’m surprised my code worked
as
well as it did. I guess the total allocation would usually (for
non-compressed files) be a multiple of the granularity so my code
worked,
even though it was mapping much bigger views of sections than it needed.

This still leaves the question of what I should use for the allocation
granularity. How do I query that? The docs implied I could pass any
value
to ZwMapViewOfSection and it would adjust for the granularity, but I
found
out right away that I needed to adjust the section offset myself before
calling it.

I just tried 65K and it seems to be working, but it bothers me that I
don’t
really know the correct value. I got the 65K number from several places
on
the web that talked like that was the correct value.

“Mark Hahn” wrote in message news:xxxxx@ntfsd…
> I’m have a problem with a ZwMapViewOfSection call on a particular
file.
> My code has worked with thousands of other calls for other files. I
> noticed that the file is compressed so I’m wondering if there is
something
> I should know about ZwMapViewOfSection with compressed files.
>
> The error returned is STATUS_MAPPED_ALIGNMENT, which says my base
address
> or section offset is not aligned to an allocation size boundary.
>
> For the base address I am passing a null ptr to let the system assign
it.
> I get the AllocationSize value from a FileStandardInformation class
query
> using FltQueryInformationFile call. I then adjust my section offset
to be
> a multiple of the allocation size.
>
> The weird thing is that the AllocationSize I am getting from
> FileStandardInformation, and using for the alignment, is 0x0001c000,
which
> seems like a really strange size. In any case the offset I want is
> 0x00021000 so aligning it amounts to using the section offset value of
> 0x0001c000, which is certainly aligned with 0x0001c000.
>
> Can anyone shed any light on this problem or suggest how I should
proceed?
>
>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Thank you very much. Some day I should just sit down and read all the docs
and memorize them. :slight_smile:

“Ladislav Zezula” wrote in message news:xxxxx@ntfsd…
>> I just tried 65K and it seems to be working, but it bothers me that I
>> don’t really know the correct value. I got the 65K number from several
>> places on the web that talked like that was the correct value.
>
> The allocation granularity can be retrieved using NtQuerySystemInformation
> with SystemBasicInformation. The result structure
> (SYSTEM_BASIC_INFORMATION)
> looks like this:
>
> typedef struct _SYSTEM_BASIC_INFORMATION
> {
> ULONG Unknown;
> ULONG MaximumIncrement;
> ULONG PhysicalPageSize;
> ULONG NumberOfPhysicalPages;
> ULONG LowestPhysicalPage;
> ULONG HighestPhysicalPage;
> ULONG AllocationGranularity;
> ULONG LowestUserAddress;
> ULONG HighestUserAddress;
> ULONG ActiveProcessors;
> UCHAR NumberProcessors;
> } SYSTEM_BASIC_INFORMATION, *PSYSTEM_BASIC_INFORMATION;
>
> L.
>
>