Connsistent Binary Interface?

Connsistent Binary Interface?

Post by bd » Mon, 24 Jun 2002 03:49:34




> It's not that you have to limit yourself to one interface for eternity.
> I mean, why should the interface changes with every major release. So if
> there were an interface for 2.2, it makes sense to have it replaced for
> 2.4 and a 2.4 interface will not be the same as 2.6. But at least within
> a "kernel family" so to say, the interface would stay the same, so a
> company could offer binary kernel modules on their webpage, saying "This
> one is for 2.2.x, this one is for 2.4.x, this one is for 2.6.x and so
> on". Of course they can't take the work in the develop kernel's into
> account, like 2.5, as that's the one where the interface might change
> and every day (these changes will decide about the interface of 2.6 for
> example), so developers working with these kernels will have to
> recompile modules whenever they use a new developer kernel, but
> end-users shouldn't be bothered with that.

Turn on module versioning both in the kernel *and* in the module.

--
Whether you can hear it or not,
The Universe is laughing behind your back.
                -- National Lampoon, "Deteriorata"

 
 
 

Connsistent Binary Interface?

Post by Christopher Brown » Mon, 24 Jun 2002 03:51:49



Quote:> I'm sorry if I ask the a question that asked hundred times a week, feel
> free to redirect me a to webpage/FAQ, I haven't found one.

> Can someone of the Linux kernel gurus please explain me (and try to keep
> it simple, okay?) why exactly the Linux kernel can't have a consistent
> binary interface?

> All I know is that XFree86 4.x has one, but the kernel has not. The
> advantage of it is that you don't have to recompile the modules if you
> change the version of XFree86 4.x, not unlike the interface has changed
> in the new version.

> Isn't it time to give the kernel such an interface as well, so companies
> can write drivers for the Linux kernel and distribute them in binary
> format? Currently that's not possible because you'd had to compile them
> for every existing kernel, otherwise loading the modules doesn't seem to
> work.

The general attitude about this seems to be that there is no
significant public value to making it particularly easy for companies
to do this.  It would certainly restrict the ability of the kernel
developers to make changes that they might wish to make, which is a
distinct _disadvantage_.

And the merit of the idea seems pretty limited.  

Why would it be _useful_ to have this interface?  Basically to toady
to companies that want to take advantage of Linux, and who are
uninterested in contributing anything back.

And the kernel developers should toady to such people precisely why?

If a particular device driver proves _so_ interesting that it is of
great value, the tendancy is for vendors to contribute and ask for it
to enter the official kernel stream.  That has proven fairly common
for various vendors. 3ware, Adaptec, BusLogic all come to mind.  

That seems to be the preferrable approach, so why should any effort be
put into toadying to more proprietary approaches?  The fact that you'd
like it doesn't mean that it is in the interests of kernel developers
to provide it.

If you're really interested in keeping things proprietary, then go
grab a copy of NetBSD, and you can customize to your heart's content,
and sell it as a product without _any_ need for _any_ public
disclosure.  That is one of the advantages of NetBSD over Linux.
--

http://www.ntlug.org/~cbbrowne/
"Parentheses?  What parentheses? I haven't noticed any parentheses
since my first month of Lisp programming.  I like to ask people who
complain about parentheses in Lisp if they are bothered by all the
spaces between words in a newspaper..."  -- Kenny Tilton


 
 
 

Connsistent Binary Interface?

Post by Kasper Dupon » Mon, 24 Jun 2002 04:23:09



> I'm sorry if I ask the a question that asked hundred times a week, feel
> free to redirect me a to webpage/FAQ, I haven't found one.

Don't know if it is listed in any FAQ, but it surely has been asked before.

Quote:

> Can someone of the Linux kernel gurus please explain me (and try to keep
> it simple, okay?) why exactly the Linux kernel can't have a consistent
> binary interface?

Because the kernel is changing. As the kernel is improved and bugs are
corrected the interfaces will have to change from time to time. Sometimes
even the interface itself improves.

Quote:

> All I know is that XFree86 4.x has one, but the kernel has not. The
> advantage of it is that you don't have to recompile the modules if you
> change the version of XFree86 4.x, not unlike the interface has changed
> in the new version.

Aha, is it also compatible with 3.x?

Quote:

> Isn't it time to give the kernel such an interface as well, so companies
> can write drivers for the Linux kernel and distribute them in binary
> format? Currently that's not possible because you'd had to compile them
> for every existing kernel, otherwise loading the modules doesn't seem to
> work.

You can load it with "insmod -f". But it is of course not garanteed
to work in every situation.

Quote:

> This is annoying, it means if you just replace your current kernel with
> a slightly newer one, you must re-build all modules that are not part of
> the kernel sources and that you build for the old version.

It can be automated, and I don't think many people have so many
nonstandard modules that the additional time is any problem.

Quote:

> It's not that you have to limit yourself to one interface for eternity.
> I mean, why should the interface changes with every major release. So if
> there were an interface for 2.2, it makes sense to have it replaced for
> 2.4 and a 2.4 interface will not be the same as 2.6. But at least within
> a "kernel family" so to say, the interface would stay the same, so a
> company could offer binary kernel modules on their webpage, saying "This
> one is for 2.2.x, this one is for 2.4.x, this one is for 2.6.x and so
> on". Of course they can't take the work in the develop kernel's into
> account, like 2.5, as that's the one where the interface might change
> and every day (these changes will decide about the interface of 2.6 for
> example), so developers working with these kernels will have to
> recompile modules whenever they use a new developer kernel, but
> end-users shouldn't be bothered with that.

The interface is being kept unchanged within the same series
unless there is some very good reason for changing it like if
it is necesarry in order to fix some bug. If a company really
wants to develop Linux drivers it wouldn't take much to do this
automatically. They could have a computer check kernel.org
daily and download any new patches and rebuild their drivers
and put the new driver on their webpage. Of course they should
of course test the driver once in a while and clearly state
which versions has been tested. If a company only release one
driver for every stable series, how would people be able to
use their hardware together with development versions?

Quote:

> That would help the people creating distributions a lot and it would
> help users a lot as well, as they can always re-use their compiled
> modules, thus it would also help companies that don't write drivers for
> Linux out of a simple reason only: They don't want to distribute them as
> source code.

A lot of Linux users doesn't want binary only drivers. For
security reasons we don't want any closed source code in
kernel mode, we want to be able to debug problems, we want
to be able to take part in development of the kernel. If you
had a single piece of hardware with only binary drivers, you
would loose all those posibilities. We cannot rely on companies
that don't want to release drivers as source.

Quote:

> Or is this some ideology nonsense? "Either your driver is open source or
> keep it away of the kernel"?

Nobody said that, but if you want to release binary only
drivers you have to offer your customers a lot more support.
1) You must release modules for a lot of kernel versions.
2) You must debug all problems ever happening on a system
   that has any of your drivers loaded, because nobody else
   will be able to do that.
If you don't do that your driver is more or less useless.

Also notice that kernel developers cannot have closed source
modules in mind while developing the kernel. They cannot
design for interoperability with software they don't know
what is doing, and they cannot test and debug it.

Quote:> Why must every hardware support be open
> source, just because the Kernel itself is?

It doesn't have to be, but binary only drives is giving a
lot of trouble.

Quote:> How's that an advantage if it
> means that hardware doesn't work at all, doesn't work properly or is so
> limited in functionality that it was a waste of money to buy it?

I don't get it. Open source drivers is an advantage because
they can be used with a lot more kernel versions, they can
be debugged, and they can be reviewed for security. A closed
source driver is not going to make the hardware perform
better than an open source driver.

Quote:> There
> is commercial software for Linux and guess what, some people are using
> it! I don't see why commercial and open source shouldn't be able to
> peacefully cooperate?

They should be able to cooperate. If you want to cooperate
the most important to do is not to keep secrets about
interfaces. That is true for all kinds of interfaces hardware
as well as software. The Linux developers doesn't keep any
secrets about the interfaces used in Linux. If the hardware
developers doesn't keep any secrets about the interfaces used
in their hardware, I'm sure it is going to work fine together.

Quote:> Linux users use commercial software and Windows
> users use open source software, it's not a black or white decision. Of
> course you can say: "I don't trust modules of that I haven't seen the
> source, who know what these may do to my system", but that's a decisions
> everyone must make for him/herself IMHO.

I use a lot of closed source software in user mode. I don't
want any of it in my kernel.

--
Kasper Dupont -- der bruger for meget tid p? usenet.

 
 
 

Connsistent Binary Interface?

Post by Pete Zaitc » Mon, 24 Jun 2002 05:05:47



> Can someone of the Linux kernel gurus please explain me (and try to keep
> it simple, okay?) why exactly the Linux kernel can't have a consistent
> binary interface?

I guess you mean "immutable" or "unchanging" under "consistent".
Before we discuss why it is so, please observe that no kernel
does have a fixed ABI over a period of time. It is possible
to create an extremely bloated kernel which would support an
ABI that changes slower than ABI of other kernels, but it would
not be immutable. Something that does not change becomes obsolete.

Consider Solaris, for instance. STREAMS modules without D_MP set
worked from 2.0 to 2.3, were deprecated in 2.4 and removed in 2.5 IIRC.
Also, 2.5 saw introduction of new DMA routines; old routines were
marked for removal in Solaris 7.

Thousands of people have difficulties loading their Windows 95
drivers in Windows 98. There is enough ABI change to trigger
incompatibility. Millions do not see a problem because they do
not use old drivers, or those drivers happen to work sometimes.
If they corrupt something, users just reboot. Linux is different
because its userland actually checks versions. BTW, does anyone
remember grief that MS-DOS programs checking versions caused?

There is nothing unique to Linux in the ABI shift. But there is
much more publicity.

Quote:> Isn't it time to give the kernel such an interface as well, so companies
> can write drivers for the Linux kernel and distribute them in binary
> format? Currently that's not possible because you'd had to compile them
> for every existing kernel, otherwise loading the modules doesn't seem to
> work.

As you can see, it is simply not possible technically to support
immutable ABIs across releases. So, where does the intrarelease
ABI skew comes from? It comes from urgent bugfixes.

Quote:> It's not that you have to limit yourself to one interface for eternity.
> I mean, why should the interface changes with every major release. So if
> there were an interface for 2.2, it makes sense to have it replaced for
> 2.4 and a 2.4 interface will not be the same as 2.6. But at least within
> a "kernel family" so to say, the interface would stay the same, so a
> company could offer binary kernel modules on their webpage, saying "This
> one is for 2.2.x, this one is for 2.4.x, this one is for 2.6.x and so
> on". [...]

I see you generally understand the problem, but I suspect you
do not realize the scope and severity of it. The thing is, how
much of the ABI your driver uses. It is possible to write a
small, simplistic driver which would not use more bleeding
edge ABIs. Such driver can be loaded even across stable releases
in Linux. In fact, this is what people did, or used to do, with
Winmodem drivers during the early 2.4 time frame.

Larger, more complicated drivers press on with use of facilities
which are bound to change quickly. For instance, consider a
transition from DRM 4.0 to DRM 4.1 in the 3D world.

Quote:> That would help the people creating distributions a lot and it would
> help users a lot as well, as they can always re-use their compiled
> modules, thus it would also help companies that don't write drivers for
> Linux out of a simple reason only: They don't want to distribute them as
> source code.

You are not completely right with your "would" here. An additional
burden of supporting immutable ABIs across releases is very very
very costrly for distributors. This includes every framework vendor,
such as Microsoft, Sun, Red Hat.

For a long time, Red Hat did not have the resources to do what
Microsoft has to do to reduce the API drift. Red Hat is conducting
an experiment in this area now with RHL AS2.1. It is a fixed-ABI
release, maintained in Boston office. Doing that costs enormous
amount of money.

Linux kernel may exacerbate the ABI drift with over-reliance on
inlines, but I think that fundamentally it does not make that much
of a difference.

Quote:>  I don't see why commercial and open source shouldn't be able to
> peacefully cooperate?

You should get your terminology straight. Commercial and open
source software are not antonims; software can be both.
Closed source and open source are opposed.

-- Pete

 
 
 

Connsistent Binary Interface?

Post by Pete Zaitc » Mon, 24 Jun 2002 05:09:18


BTW. Christopher's response reminded me of something. If I was
betting, I would bet that TGOS was talking about Nvidia without
naming it explicitly. It is the _only_ supplier of binary
drivers which have no alternative for many people. Well, tough
to be TGOS. Buy a Radeon next time.

-- Pete

 
 
 

Connsistent Binary Interface?

Post by Kasper Dupon » Mon, 24 Jun 2002 06:36:52



> Buy a Radeon next time.

I'd like to hear a litle more about why you like Radeon.

--
Kasper Dupont -- der bruger for meget tid p? usenet.

 
 
 

Connsistent Binary Interface?

Post by bd » Mon, 24 Jun 2002 06:44:39




>> Turn on module versioning both in the kernel *and* in the module.

> 1) What will the effect be of turning both on?

Some modules will work on multiple kernel versions.

Quote:> 2) How do you turn it on in a module?

I haven't the faintest idea. If it's part of the kernel, it gets turned on
with the kernel.

--
Blessed is the man who, having nothing to say, abstains from giving
wordy evidence of the fact.
                -- George Eliot

 
 
 

Connsistent Binary Interface?

Post by M?ns Rullg? » Mon, 24 Jun 2002 06:44:59




> comp.os.linux.development.system:

> > BTW. Christopher's response reminded me of something. If I was
> > betting, I would bet that TGOS was talking about Nvidia without
> > naming it explicitly. It is the _only_ supplier of binary
> > drivers which have no alternative for many people.

> NVidia release their kernel driver as source code, you can download it
> from their webpage and compile it yourself.

That is only the source code a simple wrapper around the real code,
which is only released in binary form. The binary-only module will
only call functions in the wrapper, which will call the appropriate
kernel function. This way the same binary module can be used with
different kernel versions, only requiring the wrapper to be recompiled.
Nvidia supply the source code for the wrapper so their driver can be
used with kernels they have not (yet) built the module for. The real
driver is a very well-kept secret.

Quote:> And you can also download and compile the XFree86 4.x driver from
> their webpage as source code and compile it yourself.

True.

--
M?ns Rullg?rd

 
 
 

Connsistent Binary Interface?

Post by Colin Lero » Mon, 24 Jun 2002 06:42:10



>> Buy a Radeon next time.

> Last time I checked there was no Linux support whatsoever for this
> chip,

There is now, as far as I know - I see messages about it in DRI and
Gatos mailing lists.
--
Colin
http://www.colino.net/
 
 
 

Connsistent Binary Interface?

Post by Kasper Dupon » Mon, 24 Jun 2002 06:47:11



> BTW, does anyone
> remember grief that MS-DOS programs checking versions caused?

Sure, I remember that far too well. I remeber writing a small TSR
program to disable that check. I just hooked the get version call
where I looked at the next instruction in the caller. If the
caller was going to compare the return value against a constant,
I would return that constant, otherwise I would return the real
version.

--
Kasper Dupont -- der bruger for meget tid p? usenet.

 
 
 

Connsistent Binary Interface?

Post by David Schwart » Mon, 24 Jun 2002 07:25:29



> The merit is to have excellently tuned code for your platform, as the
> compiler the company uses may generate much better code than the one you
> have installed and they might tune some methods in assembler for CPU
> types not even distinguished by your compiler so far.

        They can already do this. They simply write an adaptation layer that
they provide the source code to, and that adaptation layer provides the
kernel interface. What you seem to be asking is "why doesn't somebody
else do this for them?" and the answer is that the effort doesn't seem
to be worth it to anyone else. The fact that few module providers do
this seems to indicate that they don't think the effort is worth it
either.

        Your argument contains too many hypotheticals ("may generate", "might
tune") for anyone to take it seriously enough to work on the 'problem'.

        DS

 
 
 

Connsistent Binary Interface?

Post by Robert Redelmei » Mon, 24 Jun 2002 07:49:52



: > NVidia release their kernel driver as source code, you can download it
: > from their webpage and compile it yourself.
:  
:  That is only the source code a simple wrapper around the real code,
:  which is only released in binary form. The binary-only module will
:  only call functions in the wrapper, which will call the appropriate
:  kernel function. This way the same binary module can be used with
:  different kernel versions, only requiring the wrapper to be recompiled.
:  Nvidia supply the source code for the wrapper so their driver can be
:  used with kernels they have not (yet) built the module for. The real
:  driver is a very well-kept secret.

Quite possibly for non-nefarious reasons.

I notice that the NVidia driver takes an awfully long time (2-3 sec)
to startup.  I presume that the driver is doing an IPL of the GPU
code somehwere into GPU RAM.

The NVidia and other modern vidcards have powerful graphics
processors (GPUs) that have their own machine code and programming
languages.  I suspect that most of the secret binary module is
actually GPU code that is uploaded via slow I/O into the GPU.
The remainder is simple stubs to call a GPU program.  If that isn't
in the open-source part.

I look upon the vidcard as an embedded device, and AFAIK the GPL
doesn't require code release of embedded devices.  If actually got
NVidia to release GPU source, you'd find it useless. It's not `c`
but in whatever machine/LLL that NVidia uses, and they're _not_
required under any part of the GPL to release a compiler, assembler
or any documentation.

-- Robert

 
 
 

Connsistent Binary Interface?

Post by Kevin Buh » Mon, 24 Jun 2002 08:46:18



> Can someone of the Linux kernel gurus please explain me (and try to keep
> it simple, okay?) why exactly the Linux kernel can't have a consistent
> binary interface?

It *can*.  It just doesn't.

(Actually, it does.  The application binary interface for user-mode
programs has been frozen for a long time, and the standard shared
libraries have a binary interface that changes very seldomly, but
that's not what you're interested in, of course.)

Quote:> Isn't it time to give the kernel such an interface as well, so companies
> can write drivers for the Linux kernel and distribute them in binary
> format?

Binary-only modules generally cause nothing but heartache for kernel
developers.  It's hardly surprising that they don't want to spend much
time and effort freezing a binary interface and then living with the
resulting inflexibility and potential inefficiency (when underlying
structures have to change and someone has to add a conversion layer)
when the tiny group that'll benefit only causes them grief.

Quote:> But at least within a "kernel family" so to say, the interface would
> stay the same, so a company could offer binary kernel modules on
> their webpage, saying "This one is for 2.2.x, this one is for 2.4.x,
> this one is for 2.6.x and so on".

Oh, within stable branches, the interfaces usually *are* frozen.  (Not
officially, but by convention.)  Besides which, once a stable branch
has actually stabilized, new minor versions are released so
infrequently that a vendor should have no trouble offering a module
for every possible stock kernel version.

Quote:> That would help the people creating distributions a lot and it would
> help users a lot as well, as they can always re-use their compiled
> modules

How would it help?

Distributions usually bundle kernels with the standard set of modules
compiled for that kernel.  If they're bundling proprietary kernel
modules, too, they already have a licensing agreement with the vendor,
so they can and should demand modules compiled for the kernels they're
planning to ship.  How does having a frozen binary interface help
them?

For those users that want to upgrade their kernel by hand, why would
they upgrade their base kernel and not the modules?  Given that the
kernel *is* highly modular, bug fixes and improvements are going to
appear all over the place, in modular and non-modular parts of the
kernel.  What benefit could there be to upgrading your "bzImage" and
not your modules?  How would you even *start* to file a bug report on
"linux-kernel"?

        Dear Alan Cox,
        I am running a bzImage from linux-2.5.22-ac16-pre3, network
        modules from linux-2.4.18 (except for my IPV6 support from
        linux-2.5.22 stock), filesystem modules from linux-2.4.19-rc1,
        and blah blah blah).  It crashed.  Please help me.

Quote:> thus it would also help companies that don't write drivers for Linux
> out of a simple reason only: They don't want to distribute them as
> source code.

Wah.  There are several approaches.  First, and most obviously, you
can make a sound business decision and commit to supporting a
collection of kernels that will make your customers happy.  Often,
this means tracking a popular distribution: whatever Red Hat's kernel
du jour is, commit to having a module ready for it, for example.

Second, for your kernel junkie customers, you can take the trouble to
put a little thought into your design.  You can usually do all your
3l33t, sekrit processing in user space supported by a simple, freely
distributable stub kernel module, and this is a better approach for
most applications anyway.  Alternatively, you can deliver a tarfile of
object code and a stub C interface with a Makefile---instant
compatible kernel module!

Let me emphasize that this is a two-prong approach: for users that
*don't* compile their own kernels (and may not even know what a
tarfile is), you track a distribution's kernel updates.  To make users
that *do* compile their own kernels happy, you provide them with
advanced tools, which they are perfectly capable of using, to compile
their own versions of your module.

Quote:> Or is this some ideology nonsense?

Oh, you're one of those, are you?

Quote:> "Either your driver is open source or keep it away of the kernel"?

Nope, either your driver is open source, or you get off your *y
ass and stop expecting kernel developers to do your support work for
you.  If you can charge for your module, you can make a sound business
decision about which kernels you're planning to support and which
customers you're willing to turn away.

Quote:> Why must every hardware support be open source, just because the
> Kernel itself is? How's that an advantage if it means that hardware
> doesn't work at all, doesn't work properly or is so limited in
> functionality that it was a waste of money to buy it?

My hardware works great, and it's all supported by open source
drivers.  You don't have to make careful purchasing decisions if you
don't want to, but don't start whining when your lack of due diligence
makes life difficult for you.

--

 
 
 

Connsistent Binary Interface?

Post by Kevin Buh » Mon, 24 Jun 2002 08:52:21



> Whose forcing them upon them?
> I'm not forcing anything upon anyone.
> It's the current kernel developers who force their ever changing
> interface upon me. So give the kernel a more permanent interface and
> people will have the free choice, right now they have not.

Actually, the kernel copyright gives *you* the right to develop and
distribute your own kernel branch with a frozen, binary interface.
(Hint: the first one's easy!)

I'm 95% sure that maintaining your own kernel development branch,
getting all major distributions to use it, and convincing the
developers of every dipshit piece of hardware you see in your
magazines to hop on board and write drivers to your binary interface
will be easier than convincing Linus to freeze Linux's binary
interface.  So, you can keep ineffectually debating things here, or
you can get cracking!

--

 
 
 

Connsistent Binary Interface?

Post by Juha Laih » Mon, 24 Jun 2002 16:12:01




>comp.os.linux.development.system:

>> And the merit of the idea seems pretty limited.  

>The merit is to have hardware support for every tiny device of that
>existence you may not even be aware.

Like some HP printers (scanners, too?) that are only supported with
Windows version this and that, and no plans to widen the range of
supported Windows variants. Like devices that you would like to keep,
but cannot because you'd like to upgrade to a more modern Windows
from '95 or 3.11 (ouch..), but the vendor doesn't provide an updated
driver.

So, yes; pretty much any device comes with drivers for todays Win*.
There's no guarantee whether it will be usable tomorrow. Pretty
much any source driver that comes with Linux will be maintained
over kernel versions, so you can keep using the old devices as
long as they're of any use to you - and still run the latest OS
version.
--
Wolf  a.k.a.  Juha Laiho     Espoo, Finland

         PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

 
 
 

1. Application Binary Interface

Having seen the recent announcement that a consortium of MIPS vendors have
decided to standardize on an Application Binary Interface (ABI) which would
allow portability of binaries across various MIPS-based workstations, it
occurs to me that perhaps the "vendors" of the various "free" operating
systems (Linux, 386BSD/BSD386, Mach?) should pursue a similar initiative.
Initially this would involve standardization on the 386/486 platform; when
there is enough work done on the 680x0, an ABI could be created for it as
well. This would have the benefit of making binaries portable across the
various 386-based free operating systems while also presenting a unified
front to mainstream industry, which might turn out to be much more valuable
than the portability issue.

I don't normally read this group (though I do follow the 680x0 channel on
the mailing list), but I thought I'd throw this out there...
--
     Keep your data shining like new with Doctor Ozone's BIT CLEANER!
             Just think! Software rot is a thing of the past!


2. Apache Start up Help

3. Win Binary Interface ?

4. Backup Utilities/recommendations

5. IABI & Linux? (Intel Application Binary Interface)

6. BASH Frequently-Asked Questions (FAQ version 3.9)

7. Request for comment: SQL interface to binary files

8. Mediaone RoadRunner

9. gcc binaries/expect binaries?

10. binary to binary

11. search for binary value in binary file

12. Apache served binaries files not recognised by browser as binary, but displayed

13. Failed to execute binary (UML); binary compatibility broken?