How does the Windows kernel know which process or application does a single tcp packet being received belongs to ? Secondly then how does it copy the tcp data to the application user’s space ? does it have anything to do with the kernel code that operates within the 2GB of virtual space which is allocated to it ?
https://technet.microsoft.com/en-us/library/ms189334(v=sql.105).aspx
On Feb 21, 2017, at 7:15 PM, xxxxx@hotmail.com wrote:
How does the Windows kernel know which process or application does a single tcp packet being received belongs to ?
TCP packets all have a port number (HTTP is port 80, for example). If an app is interested in receiving packets for port 80, it creates a socket and opens a connection to port 80. When a TCP packet arrives, the data gets queued up in a buffer for that port. Sooner or later, the application will do a “read” on the socket, and the kernel returns the next chunk from the buffer.
Secondly then how does it copy the tcp data to the application user’s space ?
The socket “read” call becomes a ReadFile call, which crosses over to kernel space. The user’s buffer gets mapped into kernel space, so when the kernel network handler copies data in, it’s the same pages that the application will be reading.
does it have anything to do with the kernel code that operates within the 2GB of virtual space which is allocated to it ?
You can’t be serious with this question, can you?. And the 2GB virtual space only applies to 32-bit systems. It’s vastly larger on the ubiquitous 64-bit systems of today.
—
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
@Tim thank you for the response and for the correction on the fact that article speaks of 32 bit systems. OK then just for the sake of the argument and in keeping with 64 bit systems where the virtual space is much larger. Would there be a kernel component that resides in the virtual 8-terabyte address space that an application on a 64 bit resides ? My question as above if there is such a kernel component within this virtual address space does it play any role in communicating IO data such as TCP to the user application residing within the same virtual address space ? Such as the mapping you talked about above ? if no then what is the purpose of the kernel component which resides in the same virtual address space as the application ?
Thanks again
xxxxx@hotmail.com wrote:
thank you for the response and for the correction on the fact that article speaks of 32 bit systems. OK then just for the sake of the argument and in keeping with 64 bit systems where the virtual space is much larger. Would there be a kernel component that resides in the virtual 8-terabyte address space that an application on a 64 bit resides ? My question as above if there is such a kernel component within this virtual address space does it play any role in communicating IO data such as TCP to the user application residing within the same virtual address space ? Such as the mapping you talked about above ? if no then what is the purpose of the kernel component which resides in the same virtual address space as the application ?
I don’t think your mental model of the system is quite correct yet.
Let’s say we freeze the whole system at a given point in time. In the
lower parts of the virtual address space, we have the currently running
application, its DLLs, and the user-mode operating system DLLS like
gdi32.dll and user32.dll. In the upper parts of the virtual address
space, we have the operating system kernel, add-in kernel modules,
kernel drivers, page tables, and so on. EVERYTHING you need to run the
entire computer is in that address space. Let’s say the application
calls “recv” on a socket. The CPU starts executing in the user-mode
Windows sockets DLL, ws2_32.dll. That function will eventually call
ReadFile, in the user-mode DLL kernel32.dll.
ReadFile then makes a syscall which shifts the CPU into kernel mode, and
ends up calling into ZwReadFile in ntoskrnl.exe. Now, that’s just a
regular DLL, except that it lives in the kernel part of the process
space. It eventually will create an I/O request and call IoCallDriver,
which eventually calls into the network driver stack. The network
drivers all live in that kernel part of the address space, so these are
just function calls that happen to be done while the CPU is in ring 0
(kernel mode). While that’s going on, the rest of the address space
(including the user-mode portion) is all still there. These kernel
drivers can copy to user-mode addresses just by writing to a local pointer.
Eventually, the I/O request is completed, the functions return, we mode
the CPU back to ring 3 (user mode), and the application continues on.
When the system wants to start running a different application, it just
switches the page tables for the lower half of the virtual space. The
upper half (with the kernel) remains the same, always available.
So, it’s not like the operating system lives in some separate space.
The operating system is in memory at all times.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
@Thanks again Tim. I have a further question to ask how can I find out much more about the “kernel network handler” you described above ? Ideally I want to understand exactly in details as much as possible how TCP data is received and sent by an application but exactly what is happening on the kernel side of things. You given some excellent insights but I need more information or details on how the OS (kernel code) handles TCP data all the way up.down the tcp stack and the way the data gets mapped (via “kernel network handler”) to and from a user mode application but from the OS (kernel) perspective. I’d like learn what is happening behind the scenes so to speak.
Would there be any good online resources or chapters of any book or someone here on OSROnline who could explain more about this area. Windows Internals is a good book would it have sufficient details that I am looking for or going much further and deeper by writing a simple socket program and tracing the call under a debugger like Windbg or KD.
Any comments would be appreciated.
Thank you
Take a look at
https://msdn.microsoft.com/en-us/library/windows/desktop/bb451830(v=vs.85).a
spx
Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Thursday, March 02, 2017 6:39 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How does the Windows Kernel know which process is a
single tcp packet being received
@Thanks again Tim. I have a further question to ask how can I find out much
more about the “kernel network handler” you described above ? Ideally I want
to understand exactly in details as much as possible how TCP data is
received and sent by an application but exactly what is happening on the
kernel side of things. You given some excellent insights but I need more
information or details on how the OS (kernel code) handles TCP data all the
way up.down the tcp stack and the way the data gets mapped (via “kernel
network handler”) to and from a user mode application but from the OS
(kernel) perspective. I’d like learn what is happening behind the scenes so
to speak.
Would there be any good online resources or chapters of any book or someone
here on OSROnline who could explain more about this area. Windows Internals
is a good book would it have sufficient details that I am looking for or
going much further and deeper by writing a simple socket program and tracing
the call under a debugger like Windbg or KD.
Any comments would be appreciated.
Thank you
—
NTDEV is sponsored by OSR
Visit the list online at:
http:
MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:
To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>
xxxxx@hotmail.com wrote:
@Thanks again Tim. I have a further question to ask how can I find out much more about the “kernel network handler” you described above ? Ideally I want to understand exactly in details as much as possible how TCP data is received and sent by an application but exactly what is happening on the kernel side of things. You given some excellent insights but I need more information or details on how the OS (kernel code) handles TCP data all the way up.down the tcp stack and the way the data gets mapped (via “kernel network handler”) to and from a user mode application but from the OS (kernel) perspective. I’d like learn what is happening behind the scenes so to speak.
You have focused on the mapping, but that is such a small part of it.
Remember that every I/O request for every device in the system
(including disk files) ends up mapping (or copying) data from user mode
to kernel mode. That’s just one of the services provided by the I/O
Manager. By the time a request gets into the network stack, all of that
has already been done.
There are network driver samples in the WDK sample set that might show
you a bit about how drivers in the network stack are constructed.
The TCP/IP layers are not all that mysterious. You can pretty much
guess how they have to work, with each layer adding its own packet
headers and passing it down, until finally the packet gets to some
network card driver, where the packets gets queued up to get sent out to
a chip that will convert it to the wire.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
@Thank you Don. Don I’ll get back to on those return control codes as I call them. Tim thanks again. Looking at the WDK source samples on the network stack is a good idea which I’ll be pursuing. So basically IO Manager is made up of several if not many services that handle the data mapping that I was asking about and the same for how data flows up.down the network (tcp) stack basically yes ?
Suppose that process A sends an HTTP HEAD request to someserver.org and that process B sends the exact same request to the exact same server. Now TCP/IP connections have a source and destination address, and a source and destination port. As process A and B have the same source address, the same destination address and the same destination port, the only thing left to distinguish between them is the source port.
On Mar 3, 2017, at 7:16 PM, xxxxx@hotmail.com wrote:
So basically IO Manager is made up of several if not many services that handle the data mapping that I was asking about and the same for how data flows up.down the network (tcp) stack basically yes ?
Yes. Virtually all I/O operations in Windows happen by passing I/O Request Packets (IRPs) from driver to driver. The IRP includes a command code and a pointer to buffers, among other things. When a user-mode app makes an I/O request. the I/O Manager creates an IRP and hands it to the top driver for that device. The IRP is passed down, possibly being modified along the way, until some driver completes it.
—
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
@Thank you D.T. and thank you Tim and of course thank you Don Burns. You guys have been very helpful. I’ll dig a little further into the detail of the information each of you have provided above and if I have any questions come back and post again.
Victor
Just a note to anyone else interested in this topic but I believe that Don Burns might be on the right track in that WFP via the ALE layers. I’ve discovered from someone that using the WFP and ALE layers might be the way to go.
>Just a note to anyone else interested in this topic but I believe that Don Burns might be on the right track in that WFP via the ALE layers. I’ve discovered from someone that using the WFP and ALE layers might be the way to go.
TCPIP.SYS is one the few system components that register a PCREATE_PROCESS_NOTIFY_ROUTINE_EX or PCREATE_PROCESS_NOTIFY_ROUTINE notify routine. So TCPIP.SYS monitors processes creation and termination.