OSR's ntfsd List: memory management question

Message 1 of 5

27 Apr 00 10:12

ntfsd member 1612

xxxxxx@ntpsoftware.com

Join Date:

Posts To This List: 180

memory management question

Hello, Say you have a driver that needs to manage many small structures of fixed size such that the driver may have several megabytes worth of these structures allocated at any given time from paged pool. Since the structures are allocated from paged pool, I can imagine that it would be beneficial to allocate the structures from a contiguous heap of memory to preserve 'locality of reference': one page fault will bring in tens or hundreds of these structures reducing overall paging of the system as these structures are accessed. Is it worth the effort of doing some of your own memory management or will the VMM do an adequate job of preserving this locality of reference (or does it even try)? Looking at the DDK documentation, it seems that 'zones' were designed to address this issue, however, they have been declared obsolete. The documentation suggests 'lookaside lists', although lookaside lists seem to address a different need than did zones; lookaside lists seem not to be designed to reduce fragmentation and preserve locality of reference as I've described. So, Will lookaside lists help here? Should I use zones to reduce fragmentation? Or should I just assume the VMM will do some work to help out with fragmentation? Any suggestions? BTW are there any guidelines as to how much address space a well mannered driver should consume? Given that I'm sharing my 2GB with everyon else now (and just a fraction of that for the pools), I'm wondering how much is too much. Is several megabytes of of paged pool too much? I figure if its there, I may as well use it - but that's coming from the perspective of a user mode developer! Thanks, Joel --

Message 2 of 5

27 Apr 00 10:29

ntfsd member 2225

xxxxxx@OTG.com

Join Date:

Posts To This List: 9

RE: memory management question

You might want to look into using the zone memory routines. ExInitializeZone(); ExExtendZone(); ExAllocateFromZone(); ExFreeToZone(); ExIsFullZone(); -----Original Message----- From: Smith, Joel [mailto:xxxxx@ntpsoftware.com] Sent: Thursday, April 27, 2000 10:08 AM To: File Systems Developers Subject: [ntfsd] memory management question Hello, Say you have a driver that needs to manage many small structures of fixed size such that the driver may have several megabytes worth of these structures allocated at any given time from paged pool. Since the structures are allocated from paged pool, I can imagine that it would be beneficial to allocate the structures from a contiguous heap of memory to preserve 'locality of reference': one page fault will bring in tens or hundreds of these structures reducing overall paging of the system as these structures are accessed. Is it worth the effort of doing some of your own memory management or will the VMM do an adequate job of preserving this locality of reference (or does it even try)? Looking at the DDK documentation, it seems that 'zones' were designed to address this issue, however, they have been declared obsolete. The documentation suggests 'lookaside lists', although lookaside lists seem to address a different need than did zones; lookaside lists seem not to be designed to reduce fragmentation and preserve locality of reference as I've described. So, Will lookaside lists help here? Should I use zones to reduce fragmentation? Or should I just assume the VMM will do some work to help out with fragmentation? Any suggestions? BTW are there any guidelines as to how much address space a well mannered driver should consume? Given that I'm sharing my 2GB with everyon else now (and just a fraction of that for the pools), I'm wondering how much is too much. Is several megabytes of of paged pool too much? I figure if its there, I may as well use it - but that's coming from the perspective of a user mode developer! Thanks, Joel --

Message 3 of 5

27 Apr 00 10:46

ntfsd member 1845

xxxxxx@NSISW.COM

Join Date:

Posts To This List: 96

RE: memory management question

Why does the W2K DDK say the zone routines are obsolete and suggest we use look-aside lists? As I see it, these two sub-allocators serve two separate purposes. Zone routines are useful for allocating a large number of fixed size small blocks. Look-aside lists are useful for allocating larger blocks where there is a substantial difference between the high and low watermarks. I don't see that either sub-allocator is always a suitable replacement for the other. -----Original Message----- From: Bob Depelteau [mailto:xxxxx@OTG.com] Sent: Thursday, April 27, 2000 9:25 AM To: File Systems Developers Subject: [ntfsd] RE: memory management question You might want to look into using the zone memory routines. ExInitializeZone(); ExExtendZone(); ExAllocateFromZone(); ExFreeToZone(); ExIsFullZone(); -----Original Message----- From: Smith, Joel [mailto:xxxxx@ntpsoftware.com] Sent: Thursday, April 27, 2000 10:08 AM To: File Systems Developers Subject: [ntfsd] memory management question Hello, Say you have a driver that needs to manage many small structures of fixed size such that the driver may have several megabytes worth of these structures allocated at any given time from paged pool. Since the structures are allocated from paged pool, I can imagine that it would be beneficial to allocate the structures from a contiguous heap of memory to preserve 'locality of reference': one page fault will bring in tens or hundreds of these structures reducing overall paging of the system as these structures are accessed. Is it worth the effort of doing some of your own memory management or will the VMM do an adequate job of preserving this locality of reference (or does it even try)? Looking at the DDK documentation, it seems that 'zones' were designed to address this issue, however, they have been declared obsolete. The documentation suggests 'lookaside lists', although lookaside lists seem to address a different need than did zones; lookaside lists seem not to be designed to reduce fragmentation and preserve locality of reference as I've described. So, Will lookaside lists help here? Should I use zones to reduce fragmentation? Or should I just assume the VMM will do some work to help out with fragmentation? Any suggestions? BTW are there any guidelines as to how much address space a well mannered driver should consume? Given that I'm sharing my 2GB with everyon else now (and just a fraction of that for the pools), I'm wondering how much is too much. Is several megabytes of of paged pool too much? I figure if its there, I may as well use it - but that's coming from the perspective of a user mode developer! Thanks, Joel --

Message 4 of 5

27 Apr 00 11:37

ntfsd member 2253

xxxxxx@Exchange.Microsoft.com

Join Date:

Posts To This List: 64

RE: memory management question

Zones are not interlocked - they require external synchronization - and can't shrink. Build your own out of SLISTs. You should also have a development-cycle switch to turn it off and use regular pool. This will let you get the most out of using the verifier against the driver. -----Original Message----- From: Rob Fuller [mailto:xxxxx@NSISW.COM] Sent: Thursday, April 27, 2000 7:43 AM To: File Systems Developers Subject: [ntfsd] RE: memory management question Why does the W2K DDK say the zone routines are obsolete and suggest we use look-aside lists? As I see it, these two sub-allocators serve two separate purposes. Zone routines are useful for allocating a large number of fixed size small blocks. Look-aside lists are useful for allocating larger blocks where there is a substantial difference between the high and low watermarks. I don't see that either sub-allocator is always a suitable replacement for the other. -----Original Message----- From: Bob Depelteau [mailto:xxxxx@OTG.com] Sent: Thursday, April 27, 2000 9:25 AM To: File Systems Developers Subject: [ntfsd] RE: memory management question You might want to look into using the zone memory routines. ExInitializeZone(); ExExtendZone(); ExAllocateFromZone(); ExFreeToZone(); ExIsFullZone(); -----Original Message----- From: Smith, Joel [mailto:xxxxx@ntpsoftware.com] Sent: Thursday, April 27, 2000 10:08 AM To: File Systems Developers Subject: [ntfsd] memory management question Hello, Say you have a driver that needs to manage many small structures of fixed size such that the driver may have several megabytes worth of these structures allocated at any given time from paged pool. Since the structures are allocated from paged pool, I can imagine that it would be beneficial to allocate the structures from a contiguous heap of memory to preserve 'locality of reference': one page fault will bring in tens or hundreds of these structures reducing overall paging of the system as these structures are accessed. Is it worth the effort of doing some of your own memory management or will the VMM do an adequate job of preserving this locality of reference (or does it even try)? Looking at the DDK documentation, it seems that 'zones' were designed to address this issue, however, they have been declared obsolete. The documentation suggests 'lookaside lists', although lookaside lists seem to address a different need than did zones; lookaside lists seem not to be designed to reduce fragmentation and preserve locality of reference as I've described. So, Will lookaside lists help here? Should I use zones to reduce fragmentation? Or should I just assume the VMM will do some work to help out with fragmentation? Any suggestions? BTW are there any guidelines as to how much address space a well mannered driver should consume? Given that I'm sharing my 2GB with everyon else now (and just a fraction of that for the pools), I'm wondering how much is too much. Is several megabytes of of paged pool too much? I figure if its there, I may as well use it - but that's coming from the perspective of a user mode developer! Thanks, Joel --

Message 5 of 5

27 Apr 00 12:06

ntfsd member 2575

xxxxxx@sodatsw.cz

Join Date:

Posts To This List: 186

RE: memory management question

I think you're right. A good example of this is Lpc susbsystem which still uses zones for his tasks. > -----P?vodn? zpr?va----- > Od: Rob Fuller [SMTP:xxxxx@NSISW.COM] > Odesl?no: 27. dubna 2000 16:43 > Komu: File Systems Developers > P?edm?t: [ntfsd] RE: memory management question > > Why does the W2K DDK say the zone routines are obsolete and suggest we use > look-aside lists? As I see it, these two sub-allocators serve two > separate purposes. Zone routines are useful for allocating a large number > of fixed size small blocks. Look-aside lists are useful for allocating > larger blocks where there is a substantial difference between the high and > low watermarks. I don't see that either sub-allocator is always a > suitable replacement for the other. > > -----Original Message----- > From: Bob Depelteau [mailto:xxxxx@OTG.com] > Sent: Thursday, April 27, 2000 9:25 AM > To: File Systems Developers > Subject: [ntfsd] RE: memory management question > > <...excess quoted lines suppressed...> paged > pool. Since the structures are allocated from paged pool, I can imagine > that it would be beneficial to allocate the structures from a contiguous > heap of memory to preserve 'locality of reference': one page fault will > bring in tens or hundreds of these structures reducing overall paging of > the system as these structures are accessed. > > Is it worth the effort of doing some of your own > memory management or will the VMM do an adequate job of preserving this > locality of reference (or does it even try)? Looking at the DDK > documentation, it seems that 'zones' were designed to address this issue, > however, they have been declared obsolete. The documentation suggests > 'lookaside lists', although lookaside lists seem to address a different > need than did zones; lookaside lists seem not to be designed to reduce > fragmentation and preserve locality of reference as I've described. > > So, Will lookaside lists help here? Should I use > zones to reduce fragmentation? Or should I just assume the VMM will do > some work to help out with fragmentation? Any suggestions? > > BTW are there any guidelines as to how much address > space a well mannered driver should consume? Given that I'm sharing my > 2GB with everyon else now (and just a fraction of that for the pools), I'm > wondering how much is too much. Is several megabytes of of paged pool too > much? I figure if its there, I may as well use it - but that's coming > from the perspective of a user mode developer! > > Thanks, > Joel >

Posting Rules
You may not post new threads You may not post replies You may not post attachments	You must login to OSR Online AND be a member of the ntfsd list to be able to post.

All times are GMT -5. The time now is 13:56.

Contact Us - Osr Online Homepage - Top