Hello,
Is it possible to get the linker that comes with the WDK to output a flat
executable (instead of PE_COFF)?
I want to develop a bootloader for my hobby operating system. Right now, I
am booting my (PE_COFF) kernel with grub, but I wish to develop my own
bootloader. Grub does not understand PE_COFF, but it is multiboot compliant.
By carefully crafting a PE file, I can tell grub, with the aid of the
multiboot support, where to load stuff from it. This is very fragile, plus
I’d want my own bootloader anyway. The problem is that the build tools in
the WDK only generate PE_COFF files, while the bootloader needs to be a flat
binary for it to load directly into memory.
If I can’t use the C compilers to get a flat file, I can write it in
assembly without any problem, but I still need linker support (which I think
it is lacking).
I could use gas (GNU) or some other tools, but what I am really trying to do
is to use the WDK build environment for everything. I don’t want to mix and
maintain 2 build environments. How does the Windows team build it’s
bootloader?
setenv.bat from the WDK has a HAL environment option. Is that of any use to
me?
Thanks in advance for any suggestions.
–
Aram Hăvărneanu
Aram Hăvărneanu wrote:
Is it possible to get the linker that comes with the WDK to output a
flat executable (instead of PE_COFF)?
No, the Microsoft 32-bit linker won’t do that. There are certainly
other 32-bit linkers available that do have that ability. Check the web
pages for nasm and fasm.
The Microsoft 16-bit segmented linker can produce a flat executable (a
.com file), but you can’t feed it 32-bit object files.
I want to develop a bootloader for my hobby operating system. Right
now, I am booting my (PE_COFF) kernel with grub, but I wish to develop
my own bootloader. …
If I can’t use the C compilers to get a flat file, I can write it in
assembly without any problem, but I still need linker support (which I
think it is lacking).
Right. The compiler is not the issue. It’s the linker.
I could use gas (GNU) or some other tools, but what I am really trying
to do is to use the WDK build environment for everything. I don’t want
to mix and maintain 2 build environments. How does the Windows team
build it’s bootloader?
Take a look at it in a hex editor. It’s a 16-bit .COM file stub (which
is flat), plus a 32-bit executable tacked on to the end. The 16-bit
stub has enough smarts to load the PE file that’s tacked on the end.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
I’ve always found it surprising that up until really pretty recently - beta versions of Vista/W2K8 - the DD/WDK still included a pretty complete 16 bit toolchain. In particular, you can still download the 2K3 DDK.
I’m not sure that this will necessarily help you, but you might want to take a look at the ‘\bin\bin16’ subdirectory. In addition to the regular (link16, cl16, masm) tools, there’s also exe2bin, which might be able to help out. If I recall correctly, it accepts a load address as an argument, but then again, I think I was about 14 the last and only time that I ran across it.
Now, even assuming these all work for you, exactly how one goes about using these as a part of a proper DDK BUILD environment, I haven’t a clue, and as Tim already mentioned, it looks like msft rolled their own anyway. My only thoughts on this subject would be to take a look at the samples that come with this DDK involving Win95/98/Me support or VDD drivers and then take a tour of the MSDN Subscriber Downloads ‘Tools and Resources’ section (and probably ‘Developer Tools’ as well) and see what you can find in the way of old DDK’s/Tools. There might be some old examples that show how to do something along these lines; not very likely, but it’s the best thing I can think of. In particular, that ‘Tools and Resources’ section contains some things like the NT 3.51 DDK, as well as host of things that I’ve never even heard of.
Personally, while I think the goal of having one build environment obviously makes a lot of sense, I’d just go with gnu tools for this. In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking) could really make most of this pretty easy for you.
Good luck,
mm
> Personally, while I think the goal of having one build environment obviously makes a lot of sense,
I’d just go with gnu tools for this.
Outputs of GNU tools always come with a header pre- pended . Therefore, you will have to write a helper app that strips your target of this header if you use binutils…
In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking)
could really make most of this pretty easy for you.
NASM is, indeed, just an excellent option for bootloader writer…
In any case, what surprises me here is the OP’s choice of WDK as of his build environment for his own OS - this option seems to be quite strange for someone who wants to build a custom OS (unless the target OS somehow relies upon illegitimately obtained Windows sources, of course)…
Anton Bassov
Outputs of GNU tools always come with a header pre- pended . Therefore, you will have to write a helper app that strips your target of this header if you use binutils…
objcopy --output-target binary
mm
On Wed, Mar 3, 2010 at 4:25 AM, wrote:
> I’m not sure that this will necessarily help you, but you might want to take a look at the ‘\bin\bin16’ subdirectory. In addition to the regular (link16, cl16, masm) tools, there’s also exe2bin, which might be able to help out. If I recall correctly, it accepts a load address as an argument, but then again, I think I was about 14 the last and only time that I ran across it.
Yes, I know about those tools. Things like the compiler are 32 bit
(although they produce 16 bit code) and work in a modern system.
Thinks like masm.exe and exe2bin.exe are 16 bit and do not work on
modern, 64 bit systems.
> Personally, while I think the goal of having one build environment obviously makes a lot of sense, I’d just go with gnu tools for this. In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking) could really make most of this pretty easy for you.
Yes, I used nasm a lot with some UNIX projects. At the moment, I think
nasm or yasm are the best options. I can put the yasm binaries in my
bootloader subdirectory and use a custom Makefile to be used by
build.exe that references yasm.exe as a build tool, so when someone
checkouts the tree and want to build the thing, won’t have to install
stuff separately etc. I only need this kind of stuff for the
bootloader.
–
Aram Hăvărneanu
On Wed, Mar 3, 2010 at 5:12 AM, wrote:
> objcopy --output-target binary
I can use objcopy with binaries generated by the WDK tools, but it’s
useless as the code is 32 bit, so I’d still need a 16 bit stub that
does the transition to 32 bit.
–
Aram Hăvărneanu
On Wed, Mar 3, 2010 at 5:06 AM, wrote:
> In any case, what surprises me here is the OP’s choice of WDK as of his build environment for his own OS - this option seems to be quite strange for someone who wants to build a custom OS (unless the target OS somehow relies upon illegitimately obtained Windows sources, of course)…
I think the opposite is true. I mostly use Windows as a desktop OS
this days (I’ve used UNIX exclusively many years). On Windows, the WDK
is great. You just install it and you don’t have to think about the
build tools too much. GCC on Windows is a nightmare. Simply put,
although GCC mostly works on Windows, Windows is not a first tier
platform for GNU. There isn’t even a single install. There are many
mingw, cygwin and djgpp versions. Considering djgpp obsolete (16 bit
DOS binaries), mingw and cygwin remain. mingw is never updated and the
quality is lacking. For instance, C++ exceptions don’t work across
dlls. This is irrelevant to kernel development, but it gives you an
idea about the quality of the product. Cygwin usually crashes on 64
bit and even when it works, the performance is abysmal.
I’ve done some work on Solaris and BSDs. Excepting Solaris, which uses
it’s own compiler, the BSDs use GCC. They do not use “vanilla” GNU GCC
but they maintain their own fork, which in some cases (OpenBSD) is
very old (OpenBSD used GCC 2.95 (some platforms) and GCC 3.3 (most
platforms)) and very, very patched. Basically they do not blindly
trust GNU GCC. I don’t blindly trust GCC either, but I do not have the
resources of studying GCC code to see what versions is “best” and what
do I have to patch, and even if had the resources, I wish to spend
time coding my own product and drinking beer and painting and writing
poetry instead of tinkering with the build tools. I trust Microsoft
release engineering much more then GNU releng.
Apple and FreeBSD want to replace GCC with LLVM+clang, OpenBSD and
NetBSD want to replace GCC with PCC. There is this tendency of
replacing GCC even in UNIX world because of the decreasing quality of
the product and because it is slower and slower and less maintainable.
Why would I want to use GCC on Windows where GCC works even worse then
on UNIX? I think WDK tools represent the *sane" choice. The drawback
is that the WDK tools target only IA-32(e) and IA-64, while GCC
targets… everything.
As for the other thing… my OS can have nothing in common with
Windows code because:
1) My OS is a pure third generation L4 architecture microkernel that
runs most of the stuff in userspace.
2) My OS is Open Source, so anybody can look at the code. I didn’t
release any code yet, but I plan to do so at every release. That is,
is “Open Source” but not “Open Development”.
–
Aram Hăvărneanu
Oh yes, adding to the GNU vs. WDK thing above, I must say that I consider
build.exe to be superiour to GNU Make in every aspect. No wonder *EVERY* OS
(including Linux) has developed it’s own build tools and it not using GNU
Make…
I’m happy with build.exe, I want to use it and drink beer, I don’t want to
develop an alternative build system. OS devel is hard enough ;-).
–
Aram Hăvărneanu
WEll, I don’t want to divert this thread into comparing GCC and binutils to MSFT development tools and I definitely don’t want to compare Linux to Windows (if you are interested in it we can move the discussion to NTTALK).
WDK seems to be (IMHO, of course) quite strange option for someone who wants to build a custom OS simply because I would not expect someone who wants to develop his own OS from the scratch to be willing to sacrifice his flexibility for whatever reason - he hardly needs anything, apart from compiler, linker , assembler and command-line utility that allows him to invoke above mentioned tools with the arguments of his choice so that he can get the maximum possible flexibility and control over the build process. Although I would not expect one to write his own binary-code-generating tools, developing an OS without developing its build system sounds…ugh, " a bit strange"…
However, if the target OS heavily relies upon sources and headers that come from elsewhere, the situation becomes totally different, for understandable reasons…
Anton Bassov
> No wonder *EVERY* OS (including Linux) has developed it’s own build tools and it not using GNU Make…
You must have made your post while I was typing mine - indeed, every OS tends to have its own build environment. Therefore, I am quite surprised that you want to take another route( unless you have a good reason for this, of course)…
Anton Bassov
Many moons ago, I tried to have a boot loader that I can image to a diskette, and yet when it comes with a shell prompt, I landed in 32bit mode. It may not be the requirement in your case. And perhaps you have a very good reason to follow just the build.exe path from WDK and on top of it you might be looking exclusively for x64bit…
But there was a book called 32bit Operating systems, that had basically all the procedure I followed, and was able to boot from the floppy to a 32bit mode. As Tim pointed out, the module was old com formatted. I don’t remember exactly but I tried Tasm ( as was referred) and Masm ( just to see …). Both were able to do the job, iirc. the loader was exactly 512 byte, stuffed into 0 sector, and the signature was ( I think) 5A5A or some such.
-pro
On Mar 3, 2010, at 3:39 AM, Aram Hăvărneanu wrote:
Oh yes, adding to the GNU vs. WDK thing above, I must say that I consider build.exe to be superiour to GNU Make in every aspect. No wonder *EVERY* OS (including Linux) has developed it’s own build tools and it not using GNU Make…
I’m happy with build.exe, I want to use it and drink beer, I don’t want to develop an alternative build system. OS devel is hard enough ;-).
–
Aram Hăvărneanu
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
> I want to develop a bootloader for my hobby operating system.
It is a good task, but I think WDK is even legally prohibited from being used in this area, and surely this is not technically supported.
Use GNU tools, or strip the PE file using your own tool to the format understandable by GRUB (if you really want to violate the WDK’s license).
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
“Maxim S. Shatskih” wrote in message
news:xxxxx@ntdev…
> It is a good task, but I think WDK is even legally prohibited from being
> used in this area, and surely this is not technically supported.
I have read license.rtf, and have not found anything that would suggest that
using the WDK in this mode is legally prohibited. IANAL, but the only
reference to other operating systems is related to using the samples on
other OSes, which seems to be illegal. I certainly would not want to break
the law.
> Use GNU tools, or strip the PE file using your own tool to the format
> understandable by GRUB (if you really want to violate the WDK’s license).
Well right now, I am able to boot by PE kernel with Grub’s multiboot
support. However, this is very fragile, requires the PE file to be build in
a specific way, with stuff at specific addresses, code size is limited etc.
–
Aram Hăvărneanu
You know, you could run the Windows POSIX subsystem, which I believe comes with gcc. Windows 7 Ultimate includes the POSIX subsystem as a standard component, you just have to turn it on from control panel. The POXIS subsystem has been an installable component for many years. The POSIX subsystem is not layered over the Win32 subsystem, but has a POSIX environment subsystem, so has a POSIX correct process model that I believe includes things like fork and signals. It also has case sensitive pathnames, everything under a single tree, and forward slash as a separator, just like you’d expect on a POSIX system.
Jan
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-403783-
xxxxx@lists.osr.com] On Behalf Of Aram Havarneanu
Sent: Wednesday, March 03, 2010 3:33 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Flat Executable Compiler Output
On Wed, Mar 3, 2010 at 5:06 AM, wrote:
> > In any case, what surprises me here is the OP’s choice of WDK as of
> his build environment for his own OS - this option seems to be quite
> strange for someone who wants to build a custom OS (unless the target
> OS somehow relies upon illegitimately obtained Windows sources, of
> course)…
>
> I think the opposite is true. I mostly use Windows as a desktop OS
> this days (I’ve used UNIX exclusively many years). On Windows, the WDK
> is great. You just install it and you don’t have to think about the
> build tools too much. GCC on Windows is a nightmare. Simply put,
> although GCC mostly works on Windows, Windows is not a first tier
> platform for GNU. There isn’t even a single install. There are many
> mingw, cygwin and djgpp versions. Considering djgpp obsolete (16 bit
> DOS binaries), mingw and cygwin remain. mingw is never updated and the
> quality is lacking. For instance, C++ exceptions don’t work across
> dlls. This is irrelevant to kernel development, but it gives you an
> idea about the quality of the product. Cygwin usually crashes on 64
> bit and even when it works, the performance is abysmal.
>
> I’ve done some work on Solaris and BSDs. Excepting Solaris, which uses
> it’s own compiler, the BSDs use GCC. They do not use “vanilla” GNU GCC
> but they maintain their own fork, which in some cases (OpenBSD) is
> very old (OpenBSD used GCC 2.95 (some platforms) and GCC 3.3 (most
> platforms)) and very, very patched. Basically they do not blindly
> trust GNU GCC. I don’t blindly trust GCC either, but I do not have the
> resources of studying GCC code to see what versions is “best” and what
> do I have to patch, and even if had the resources, I wish to spend
> time coding my own product and drinking beer and painting and writing
> poetry instead of tinkering with the build tools. I trust Microsoft
> release engineering much more then GNU releng.
>
> Apple and FreeBSD want to replace GCC with LLVM+clang, OpenBSD and
> NetBSD want to replace GCC with PCC. There is this tendency of
> replacing GCC even in UNIX world because of the decreasing quality of
> the product and because it is slower and slower and less maintainable.
> Why would I want to use GCC on Windows where GCC works even worse then
> on UNIX? I think WDK tools represent the *sane" choice. The drawback
> is that the WDK tools target only IA-32(e) and IA-64, while GCC
> targets… everything.
>
> As for the other thing… my OS can have nothing in common with
> Windows code because:
>
> 1) My OS is a pure third generation L4 architecture microkernel that
> runs most of the stuff in userspace.
> 2) My OS is Open Source, so anybody can look at the code. I didn’t
> release any code yet, but I plan to do so at every release. That is,
> is “Open Source” but not “Open Development”.
>
> –
> Aram Hăvărneanu
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
> Oh yes, adding to the GNU vs. WDK thing above, I must say that I consider
build.exe to be superiour to GNU Make in every aspect.
It is inferior in features, but is is “correct subset wrapper”.
Actually, BUILD is a “correct subset wrapper” for NMAKE, which in turn also inferior to GNU make in features.
No wonder *EVERY* OS
(including Linux) has developed it’s own build tools and it not using GNU
Make…
Using “correct subset wrappers” for GNU make I think.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
>You know, you could run the Windows POSIX subsystem, which I believe comes with gcc.
IIRC it is renamed Interix (if not even Services for UNIX), but the rest of your post is true.
Historically, the thing used its own PIDs which are not NT kernel/Win32 PIDs. This was not so good.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
>
Apple and FreeBSD want to replace GCC with LLVM+clang.
Yes, it is a very powerful feature - In another discussion on nttalk I said that IL(intermediate langs) has it its own benefit, and I recently encountered this idea pioneered by Latner, it is perhaps to have the Objective-C as front end to gcc ( but it is OT).
-pro
On Win 7 Ultimate, which I’m running right now, if I go into control panel->programs->turn features on/off it’s called “subsystem for Unix”, and sometimes goes by the name SFU. Wikipedia has a quick overview and useful links http://en.wikipedia.org/wiki/Subsystem_for_UNIX-based_Applications
I didn’t know that about process id’s, although am curious how a kernel process id could be anything other than a kernel process id.
Jan
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-403817-
xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Wednesday, March 03, 2010 7:52 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Flat Executable Compiler Output
>You know, you could run the Windows POSIX subsystem, which I believe
comes with gcc.
IIRC it is renamed Interix (if not even Services for UNIX), but the
rest of your post is true.
Historically, the thing used its own PIDs which are not NT
kernel/Win32 PIDs. This was not so good.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
> I think WDK is even legally prohibited from being used in this area,
I just wonder in which jurisdiction one can legally prohibit to use a certain compiler for something other than a certain purpose…
strip the PE file using your own tool to the format understandable by GRUB (if you really want
to violate the WDK’s license).
Gets more and more exciting…
Are you claiming that one can be legally prohibited from editing binary data if it happens to be a certain compiler’s output???
I just wonder what jurisdiction you are living in. I strongly suspect that this “jurisdiction” exists only in
imagination of those who take everything from " Politburo in Redmond" for granted…
Anton Bassov