RtlCompressBuffer in NDIS

sagi_zar · October 1, 2015, 9:47am

Hi,

Can I use RtlCompressBuffer APi from within a miniport driver?
for some reason I get unresolved external errors.

Thanks,
Sagi

harald_brown · October 1, 2015, 9:59am

yes, can. it in NtosKrnl.lib

sagi_zar · October 1, 2015, 10:15am

Thanks but I have two issues:

if I include Ntifs.h it have redefinitions errors with wdm.h (for example on PEPROCESS)
if I copy the relevant definitions from Ntifs.h I get unresolved external on RtlCompressBuffer and I do link with NtosKrnl.lib

Don_Burn · October 1, 2015, 10:18am

Are you including ntifs.h in your NDIS driver?

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@tandemg.com
Sent: Thursday, October 01, 2015 9:47 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] RtlCompressBuffer in NDIS

Hi,

Can I use RtlCompressBuffer APi from within a miniport driver?
for some reason I get unresolved external errors.

Thanks,
Sagi

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

sagi_zar · October 1, 2015, 10:22am

Hi,
I tried to add #include <ntifs.h> but I got re-definitions errors on PEPROCESS and PETHREAD.
I tried to copy the structures and API headers from ntifs.h and than I got the unresolved external error.</ntifs.h>

Don_Burn · October 1, 2015, 10:31am

Are you putting ntifs.h as the first include?

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@tandemg.com
Sent: Thursday, October 01, 2015 10:15 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] RtlCompressBuffer in NDIS

Thanks but I have two issues:

if I include Ntifs.h it have redefinitions errors with wdm.h (for example
on PEPROCESS) 2. if I copy the relevant definitions from Ntifs.h I get
unresolved external on RtlCompressBuffer and I do link with NtosKrnl.lib

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

sagi_zar · October 1, 2015, 10:47am

thanks a lot
I change the order (put the ntifs.h include first) and it solved all my problems.

sagi_zar · October 11, 2015, 9:27am

Hi,
I used the RtlCompressBuffer to compress buffers before writing them to a file.
I hopped that after I will close the file I will be able to open it using a zip application.
was I wrong?
Do I need to decompress the buffers using the RtlDecompressBuffer on the original output buffers from the RtlCompressBuffer?

Can I somehow send an IOCL FSCTL_SET_COMPRESSION from my NDIS driver to file I created so the file will be compress automatically?

Thanks,
Sagi

Maxim_S_Shatskih · October 11, 2015, 5:47pm

> I used the RtlCompressBuffer to compress buffers before writing them to a file.

Ancient totally inferior algorithm.

Better to implement your own, looking around on existing open source stuff.

I hopped that after I will close the file I will be able to open it using a zip application.
was I wrong?

No for sure, this algorithm is more primitive then ZIP uses and is not compatible with ZIP.

Can I somehow send an IOCL FSCTL_SET_COMPRESSION from my NDIS driver to file I created

Why not? yes you can.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Tim_Roberts · October 12, 2015, 2:56pm

xxxxx@tandemg.com wrote:

I used the RtlCompressBuffer to compress buffers before writing them to a file.
I hopped that after I will close the file I will be able to open it using a zip application.
was I wrong?

Yes. What on earth led you to that conclusion? Even if you used LZNT1,
all you end up with is a single compressed buffer. It doesn’t have any
of the file system or directory overhead that is required to make a zip
file.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

anton_bassov · October 13, 2015, 12:00am

> Better to implement your own, looking around on existing open source stuff.

What a pity our “Dr.Flounder” is gone - just imagine a diatribe that the above suggesion would have provoked. Hopefully, the bloke with CapLocks key on most of the time will partly “compensate the loss” and voice his opinion on the topic…

Anton Bassov

MBond · October 13, 2015, 5:26pm

There is no need for open source IMHO. An effective implementation of an LZ style algorithm with minimal memory requirements can be done in 500 or so lines based on published standards and academic research

Sent from Mail for Windows 10

From: xxxxx@hotmail.com
Sent: October 13, 2015 12:10 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] RtlCompressBuffer in NDIS

Better to implement your own, looking around on existing open source stuff.

What a pity our “Dr.Flounder” is gone - just imagine a diatribe that the above suggesion would have provoked. Hopefully, the bloke with CapLocks key on most of the time will partly “compensate the loss” and voice his opinion on the topic…

Anton Bassov

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

sagi_zar · October 14, 2015, 11:33am

Hi,

I tried to send the FSCTL_SET_COMPRESSION IOCTL on the file handle I created so it will be compressed.
For some reason I keep on getting error 0xc000000d on the call to ZwDeviceIoControlFile.
I used the same parameters when using a call to DeviceIoControl from an application and it worked.
can this work?

Is there another better way to save data from an NDIS driver to a file in a zip format?

Thanks,
Sagi

Bo_Branten · October 14, 2015, 12:23pm

On Wed, 14 Oct 2015, xxxxx@tandemg.com wrote:

I tried to send the FSCTL_SET_COMPRESSION IOCTL on the file handle I created so it will be compressed.
For some reason I keep on getting error 0xc000000d on the call to ZwDeviceIoControlFile.
I used the same parameters when using a call to DeviceIoControl from an application and it worked.
can this work?

Did you use ZwFsControlFile?

Bo

Alex_Ionescu-2 · October 17, 2015, 10:05pm

A few thoughts…

As of Windows 8, the Rtl compression functions can also do Xpress and Huffman. A bit less “ancient” than LZNT1.
Given the complexity and risks of getting things wrong, I’d still recommend, in the kernel, sticking with the Rtl functions. I’ve seen an LZMA engine ported to the kernel, and it wasn’t pretty…

If you really want to do better compression, talk to a user-mode service, imo.

Maxim_S_Shatskih · October 18, 2015, 12:46pm

> 1) As of Windows 8, the Rtl compression functions can also do Xpress and Huffman. A bit less

“ancient” than LZNT1.

Huffman is also ancient, but gives much lesser “window” for slow or pathetic quality implementation. The main complexity of Huffman is analyze pass, with the compression pass being very, very simple (not much more complex then things like CRC32).

I’m not saying LZ is obsolete. I’m saying that LZNT1 is a very inferior and primitive implementation of LZ.

BTW, ZIP also uses inferior and primitive implementation of LZ, though lesser inferior or primitive then LZNT1.

That’s why ZIP in speed/minimal compression mode loses hands up in speed to modern stuff based on LZO, QuickLZ etc, though providing nearly the same ratio.

The same ZIP in maximum compression mode provides the same speed as BZ2, a bit worse ratio, but very small memory requirements (kinda 128KB vs kinda 4MB). So, in this segment, it is at least viable.

High-ratio algorithms (higher then Huffman) are very slow and thus usually unsuitable for data streaming.

XPress is just LZNT1+Huffman:

https://msdn.microsoft.com/en-us/library/hh536252.aspx

Given the complexity and risks of getting things wrong,

Since compression is a single-threaded “thing in itself” which does not rely on any APIs, porting it to kmode is trivial (provided you have good reliable umode code).

sticking >with the Rtl functions.

…and with their pathetic speed and not so good ratio. Your competitor will have it better

I’ve seen an LZMA engine ported to the kernel, and it wasn’t pretty…

So what? a problem of one particular implementation.

All serious backup software vendors provide kernel-mode mounter of their disk images. Images are compressed, so they need at least a kmode decompressor.

If you really want to do better compression, talk to a user-mode service, imo.

Your competitor will have it better

Compression comes to play where the amounts of data are large. This means it is nearly always perf important.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Alex_Ionescu-2 · October 18, 2015, 12:54pm

Rtl functions are actually extremely fast (unless you put them in “maximum” mode)… this is how the kernel actually compresses the hibernation file. Last time I bench-marked Xpress, I was getting 500MB/s+.

True, their compression ratios are not as good as something like LZMA. But LZMA is very slow. So if we’re talking perf, I still don’t think you can do much better than the kernel’s algorithms, without significantly more code/complexity. If we’re talking compression ratio, then yes, there’s way better out there, but it doesn’t come cheap.

I continue to disagree that putting LZMA in the kernel is a good idea. That type of put-it-all-in-kernel mentality is why Microsoft now has to rely on the hypervisor to provide security. It’s hopeless to do it in kernel nowadays.

BTW, “The implementation” of LZMA I was talking about is the reference source. Getting compression right is about as hard as getting encryption right.

Maxim_S_Shatskih · October 18, 2015, 1:10pm

> Rtl functions are actually extremely fast (unless you put them in “maximum” mode)…

It is trivial to find the algo on the web which is 3 times faster then them, with the same ratio.

Especially in the maximum mode, where “3” can become “5”.

It’s hopeless to do it in kernel nowadays.

All major vendors do this without losing speed on inverted calls.

BTW, “The implementation” of LZMA I was talking about is the reference source. Getting
compression right is about as hard as getting encryption right.

Yes, but porting it from umode to kmode is trivial.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

MBond · October 18, 2015, 8:02pm

well, compression is a form of encryption right? you replace one data representation with another that is harder to use - and is smaller you hope.

but seriously, 500 lines of code can yield a constant memory (pre-allocated) algorithm that exceeds the rtl routines in both speed and compression ratio. the one I use, I wrote while undergoing cancer treatments - which is the cause I ascribe to needing a revision build on account of an off by one error with specific input. the bottom line is it isn’t that hard to code this for any programmer competent enough to do KM programming

Date: Sun, 18 Oct 2015 12:53:19 -0400
From: xxxxx@videotron.ca
To: xxxxx@lists.osr.com
Subject: RE:[ntdev] RtlCompressBuffer in NDIS

Rtl functions are actually extremely fast (unless you put them in “maximum” mode)… this is how the kernel actually compresses the hibernation file. Last time I bench-marked Xpress, I was getting 500MB/s+.

True, their compression ratios are not as good as something like LZMA. But LZMA is very slow. So if we’re talking perf, I still don’t think you can do much better than the kernel’s algorithms, without significantly more code/complexity. If we’re talking compression ratio, then yes, there’s way better out there, but it doesn’t come cheap.

I continue to disagree that putting LZMA in the kernel is a good idea. That type of put-it-all-in-kernel mentality is why Microsoft now has to rely on the hypervisor to provide security. It’s hopeless to do it in kernel nowadays.

BTW, “The implementation” of LZMA I was talking about is the reference source. Getting compression right is about as hard as getting encryption right.

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

David_R_Cattley · October 21, 2015, 8:28am

> well, compression is a form of encryption right?

Yeah, not so much. One might call it obfuscation. Clearly compression and encryption are both “Transformations”.

Compression has no ‘secret’ required to decode a message as an overall architectural feature. Something has to be (supposedly) secret for it to be encryption. Algorithms are very weak secrets. Parameters to algorithms independent of the data to be encrypted/decrypted are much better secrets.

I suppose it is semantics but I would hate for someone to think that they have achieved any sort of implied encryption benefit by compression alone.

Cheers,
Dave Cattley

Sent from Mail for Windows 10