! memory resident programs

! memory resident programs

Post by Return Addres » Sat, 24 Aug 2002 00:48:36



I would like to set up a few programs
so the first time they are loaded into memory
and executed is the only time it has to be
loaded.
After that I want it to stay in memory
until it is either called again or I explicitly
kill it.

My question: is there a way in linux
to make a program resident and re-entrant
into memory (maybe re-entrant is the wrong word)
since I want it to work as if it is loaded
into memory each time from scratch without
really taking the time to. Ideally I would like
to do this with both C++ program and
the PERL interpreter and some Perlscripts.

I just want to save on the time it takes
to read the program into memory since this
program is run many times by many users

Steve

 
 
 

! memory resident programs

Post by Grant Edwar » Sat, 24 Aug 2002 01:01:15



> I just want to save on the time it takes
> to read the program into memory since this
> program is run many times by many users

Forget about it.  The VM system does all that for you.  It's
far better at deciding what needs to be in RAM than you are. If
a program gets used a lot, it will stay in RAM.

--
Grant Edwards                   grante             Yow!  .. my NOSE is NUMB!
                                  at              
                               visi.com            

 
 
 

! memory resident programs

Post by Todd Knar » Sat, 24 Aug 2002 01:47:16



Quote:> I would like to set up a few programs
> so the first time they are loaded into memory
> and executed is the only time it has to be
> loaded.
> After that I want it to stay in memory
> until it is either called again or I explicitly
> kill it.

If you mean a program that keeps running after you've returned to the
command line, that's a normal program. All you do is run it with '&'
on the end of the command line ( or more likely, something like
'</dev/null >&/dev/null &' to insure it's no longer connected to it's
input and output ). There's more elaborate things you can do if you're
writing code, any standard Unix programming text should cover the details
under "daemon", but this is the simple way to do it.

If you mean a program that stays resident after it exits and running
it a second time just reactivates the resident copy, that's pointless
on Unix. The VM system does a much better job of insuring that the pages
you want are resident when you want them. Just run the program and let
the VM and cache system do it's job.

TSRs such as you find in the DOS and Windows world aren't needed in Unix.

--
Don't worry about where to land -- by the time you get to it, it
_will_ be flat.
                                -- concering Orion landing procedures

 
 
 

! memory resident programs

Post by Robert Kais » Sat, 24 Aug 2002 01:23:57





>> I just want to save on the time it takes
>> to read the program into memory since this
>> program is run many times by many users

> Forget about it.  The VM system does all that for you.  It's
> far better at deciding what needs to be in RAM than you are. If
> a program gets used a lot, it will stay in RAM.

Hmmm, considering the VM does not have a clue what I'm up to,
how can it make a better descision than I can ? It seems the VM
is optimised to keep things in RAM based on how frequently they
are being used. In most cases that may be what is desired, but
for some applications (e.g. real-time stuff) it could be an
extremely poor strategy.

To the OP: have a look at mlock() amd mlockall().

Rob

----------------------------------------------------------------
Robert Kaiser                     email: rkaiser AT sysgo DOT de
SYSGO RTS GmbH                    http://www.elinos.com
Klein-Winternheim / Germany       http://www.sysgo.de

 
 
 

! memory resident programs

Post by Mario Klebs » Sat, 24 Aug 2002 04:32:20



>I would like to set up a few programs
>so the first time they are loaded into memory
>and executed is the only time it has to be
>loaded.

Allthopugh this usually is a bad idea (the system normally handles
this sufficiently well), you can set the sticky-bit. However, I must
admint, that I do not know, wether linux honors it.

Quote:>After that I want it to stay in memory
>until it is either called again or I explicitly
>kill it.

I have no idea, how to remove a stiky executable out of memory.

Quote:>My question: is there a way in linux
>to make a program resident and re-entrant
>into memory (maybe re-entrant is the wrong word)
>since I want it to work as if it is loaded
>into memory each time from scratch without
>really taking the time to. Ideally I would like
>to do this with both C++ program and
>the PERL interpreter and some Perlscripts.

Are you sure, your problem really is loading? Chances arfe good, you
are observig delays caused by dynamic linking and initialisation. Even
if the executables are kept in memory, these steps cannot be skipped.

Quote:>I just want to save on the time it takes
>to read the program into memory since this
>program is run many times by many users

The operating system will do this four you, you normally do not have
to care about.

73, Mario
--

PGP-Key available at http://www.klebsch.de/public.key
Fingerprint DSS: EE7C DBCC D9C8 5DC1 D4DB  1483 30CE 9FB2 A047 9CE0
 Diffie-Hellman: D447 4ED6 8A10 2C65 C5E5  8B98 9464 53FF 9382 F518

 
 
 

! memory resident programs

Post by M?ns Rullg? » Sat, 24 Aug 2002 05:59:08




> >I would like to set up a few programs
> >so the first time they are loaded into memory
> >and executed is the only time it has to be
> >loaded.

> Allthopugh this usually is a bad idea (the system normally handles
> this sufficiently well), you can set the sticky-bit. However, I must
> admint, that I do not know, wether linux honors it.

Do any recent OSes bother with it?

Quote:> >After that I want it to stay in memory
> >until it is either called again or I explicitly
> >kill it.

> I have no idea, how to remove a stiky executable out of memory.

AFAIK, those OSes that do use the sticky bit will drop it from memory
when the memory is needed, just like any modern VM system.

Quote:> Are you sure, your problem really is loading? Chances arfe good, you
> are observig delays caused by dynamic linking and initialisation. Even
> if the executables are kept in memory, these steps cannot be skipped.

Link with -static.  I don't know how much difference it will make.

--
M?ns Rullg?rd

 
 
 

! memory resident programs

Post by Eric P. McC » Sat, 24 Aug 2002 06:29:28



> > Allthopugh this usually is a bad idea (the system normally handles
> > this sufficiently well), you can set the sticky-bit. However, I must
> > admint, that I do not know, wether linux honors it.
> Do any recent OSes bother with it?

From chmod(1):

  (the sticky bit) When set on a directory, unprivileged
  users can delete and rename only those files in the direc-
  tory that are owned by them, regardless of the permissions
  on the directory.  Under FreeBSD, the sticky bit is ignored
  for executable files and may only be set for directories
  (see sticky(8)).

Anyone got a copy of Solaris lying around?

Quote:> > Are you sure, your problem really is loading? Chances arfe good, you
> > are observig delays caused by dynamic linking and initialisation. Even
> > if the executables are kept in memory, these steps cannot be skipped.
> Link with -static.  I don't know how much difference it will make.

I remember IBM did some benchmarks (on OS/2) about dynamic loading.  I
don't remember any of the numbers, but their results showed pretty
conclusively that dynamic linking and lazy name resolution are
_faster_ than static linking in most cases.  (It surprised me too.)
But it might also be an OS/2 thing rather than a dynamic-linking
thing, so YMMV.

--

"Last I checked, it wasn't the power cord for the Clue Generator that
was sticking up your ass." - John Novak, rasfwrj

 
 
 

! memory resident programs

Post by mic.. » Sat, 24 Aug 2002 17:06:46



Quote:> I remember IBM did some benchmarks (on OS/2) about dynamic loading.  I
> don't remember any of the numbers, but their results showed pretty
> conclusively that dynamic linking and lazy name resolution are
> _faster_ than static linking in most cases.  (It surprised me too.)
> But it might also be an OS/2 thing rather than a dynamic-linking
> thing, so YMMV.

There has been a long discussion of this question on the FreeBSD newsgroup
by J. Dyson, the author of the FreeBSD VM management, and he said very
strongly the opposite. In particular he argued that shells should always
be statically linked (since shells are typically used many times on a machine
the text segment will be shared anyways), and that this makes huge timing
differences on the execution of shell scripts like ./configure which
fork a lot.

--
Michel Talon

 
 
 

! memory resident programs

Post by M?ns Rullg? » Sat, 24 Aug 2002 20:08:33



Quote:> From chmod(1):

>   (the sticky bit) When set on a directory, unprivileged
>   users can delete and rename only those files in the direc-
>   tory that are owned by them, regardless of the permissions
>   on the directory.  Under FreeBSD, the sticky bit is ignored
>   for executable files and may only be set for directories
>   (see sticky(8)).

> Anyone got a copy of Solaris lying around?

Solaris 8 man page only mentions the sticky bit for directories.

Quote:> > > Are you sure, your problem really is loading? Chances arfe good, you
> > > are observig delays caused by dynamic linking and initialisation. Even
> > > if the executables are kept in memory, these steps cannot be skipped.

> > Link with -static.  I don't know how much difference it will make.

> I remember IBM did some benchmarks (on OS/2) about dynamic loading.  I
> don't remember any of the numbers, but their results showed pretty
> conclusively that dynamic linking and lazy name resolution are
> _faster_ than static linking in most cases.  (It surprised me too.)
> But it might also be an OS/2 thing rather than a dynamic-linking
> thing, so YMMV.

I would guess this is very dependant on the linker.  One thing that
could speed up dynaminc linking is the fact that many of the shared
libraries required will already be in ram, possibly also in cache.

--
M?ns Rullg?rd

 
 
 

! memory resident programs

Post by John S. Dyso » Sat, 24 Aug 2002 21:24:36



> > I remember IBM did some benchmarks (on OS/2) about dynamic loading.  I
> > don't remember any of the numbers, but their results showed pretty
> > conclusively that dynamic linking and lazy name resolution are
> > _faster_ than static linking in most cases.  (It surprised me too.)
> > But it might also be an OS/2 thing rather than a dynamic-linking
> > thing, so YMMV.

> I would guess this is very dependant on the linker.  One thing that
> could speed up dynaminc linking is the fact that many of the shared
> libraries required will already be in ram, possibly also in cache.

It is also dependent upon the application.  Some applications are
very transient, and are the only current instance running, and haven't
run for a long time in the past...   Shared libs (esp on X and other
large systems) can make a big positive difference.   On shell-like
applications, where multiple instances are already running, the shared
text (and tightly packed shared-data, and shared-bss to some extent)
are sufficient, and shared libs actually slow things down for such
processes.

One case where I don't remember my results, programs like 'xterm' which
are large, Xwin based, but have multiple instances, it would be interesting
to measure the difference.   My guess is that people build xterm with shared
libs for maintenance issues as much as for diskspace reasons.  Xterms
are likely not doing fork/execs themselves nearly as often as the underlying
command-line shells, so the system-wide speed improvement for static builds
is likely nil (perhaps negative.)

John

 
 
 

! memory resident programs

Post by Nils O. Sel?sda » Sun, 25 Aug 2002 06:20:58


In reply to Return Address:

Quote:> I would like to set up a few programs
> so the first time they are loaded into memory
> and executed is the only time it has to be
> loaded.
> After that I want it to stay in memory
> until it is either called again or I explicitly
> kill it.

> My question: is there a way in linux
> to make a program resident and re-entrant
> into memory (maybe re-entrant is the wrong word)
> since I want it to work as if it is loaded
> into memory each time from scratch without
> really taking the time to. Ideally I would like
> to do this with both C++ program and
> the PERL interpreter and some Perlscripts.

> I just want to save on the time it takes
> to read the program into memory since this
> program is run many times by many users

Just remember that if its loaded once, its already in memory
and wont load again when the second instance is run.
Secondly, when the last instance terminates, its still kept
in the memory untill something else really needs the memory.
That said, the closest you probably get is putting the app
on a ram disk and run it from there. It will still be "loaded",
but not from a disk.

--

System Developer, UtelSystems a/s
w w w . u t e l s y s t e m s . c o m

 
 
 

! memory resident programs

Post by Return Addres » Mon, 26 Aug 2002 00:24:32


Interesting,
so if I run the program at , say, 9AM
then copy a new version of the program
into the old program name & location on disk
and then rerun the program.
It will run the old version since it is still
in memory from the first run.
Or does it check the disk to make sure
the file it used originally to load the
program hasn't changed, and then reload
the program as the new version.



> In reply to Return Address:

> > I would like to set up a few programs
> > so the first time they are loaded into memory
> > and executed is the only time it has to be
> > loaded.
> > After that I want it to stay in memory
> > until it is either called again or I explicitly
> > kill it.

> > My question: is there a way in linux
> > to make a program resident and re-entrant
> > into memory (maybe re-entrant is the wrong word)
> > since I want it to work as if it is loaded
> > into memory each time from scratch without
> > really taking the time to. Ideally I would like
> > to do this with both C++ program and
> > the PERL interpreter and some Perlscripts.

> > I just want to save on the time it takes
> > to read the program into memory since this
> > program is run many times by many users
> Just remember that if its loaded once, its already in memory
> and wont load again when the second instance is run.
> Secondly, when the last instance terminates, its still kept
> in the memory untill something else really needs the memory.
> That said, the closest you probably get is putting the app
> on a ram disk and run it from there. It will still be "loaded",
> but not from a disk.

> --

> System Developer, UtelSystems a/s
> w w w . u t e l s y s t e m s . c o m

 
 
 

! memory resident programs

Post by John S. Dyso » Mon, 26 Aug 2002 00:40:52



> Interesting,
> so if I run the program at , say, 9AM
> then copy a new version of the program
> into the old program name & location on disk
> and then rerun the program.
> It will run the old version since it is still
> in memory from the first run.
> Or does it check the disk to make sure
> the file it used originally to load the
> program hasn't changed, and then reload
> the program as the new version.

Assuming NO OTHER SYSTEM ACTIVITY, a proper
merged VM/Buffer cache system will still have the copied
contents of the new file in memory.   The new file information
will also still likely be in memory, so the number of disk accesses
due to running a new program should be nil.    On a totally
standard POSIX system, with perfect caching, the only real
I/O will be to eventually update the access times, etc.   The old file
buffers will either be over-written by the file modification, or
the old file buffering will have been deleted by a file deletion
before creation of the new copy.   (This depends upon the
filesystem semantics and the user program behavior.)

If the disk drive is shared (through locking protocols), then there
might be a need to re-read the file (depending upon system network
activity.)   This doesn't appear to be the circumstance that you are
asking about, however.

One major key to maximum system efficiency is to make sure that
use of memory doesn't compete too much against the file caching.   Also,
it is important to make sure that I/O or copying isn't necessary to
make a newly created executable program into a runnable entity.   (*Of course,
there are times where semantics might require this -- e.g. networked
environments with robust program behavior.)

John

 
 
 

! memory resident programs

Post by Steve Kirkendal » Mon, 26 Aug 2002 02:12:58



> so if I run the program at , say, 9AM
> then copy a new version of the program
> into the old program name & location on disk
> and then rerun the program.
> It will run the old version since it is still
> in memory from the first run.
> Or does it check the disk to make sure
> the file it used originally to load the
> program hasn't changed, and then reload
> the program as the new version.

Actually it doesn't need to physically check the disk.  All disk I/O
goes through the block cache (that's the main difference between block
devices and character devices in Unix/Linux) so if you write a new
version of the program then the new blocks will be in the cache instead
of the old ones.

I'd also like to raise a point that nobody else has mentioned.  Although
Linux has very good memory management, earlier Unix variants weren't so
lucky.  They did need a way to lock a program into memory, and so a way
evolved to do this: The "sticky" bit in a file's permissions (also called
the "tacky" bit).  THIS BIT HAS NO EFFECT IN LINUX, but some old versions
of Unix would keep an executable file in memory if the sticky bit was
turned on.  You can turn it on by running "chmod +t myprogram".  After
that, running "ls -l myprogram" will show "t" as the last character in
the file's permissions string instead of "x", to denote this.

 
 
 

! memory resident programs

Post by Nils O. Sel?sda » Mon, 26 Aug 2002 03:05:16


In reply to Return Address:

Quote:> Interesting,
> so if I run the program at , say, 9AM
> then copy a new version of the program
> into the old program name & location on disk
> and then rerun the program.
> It will run the old version since it is still
> in memory from the first run.
> Or does it check the disk to make sure
> the file it used originally to load the
> program hasn't changed, and then reload
> the program as the new version.

Well, if you replaced the binary, the blocks of the file
would be in the cache because of the copy you did, so
when requesting the binary again the file will possibly
already be in the cache.

--

System Developer, UtelSystems a/s
w w w . u t e l s y s t e m s . c o m

 
 
 

1. Restarting crashed memory resident program?

We have a researcher here who was running a large Smalltalk program
for several days, when the program core dumped. This program is one
*Large* memory resident image with no File I/O, processes, etc.

He wants to try to recover the data, as was wondering if the core file
could be patched enough to let him restart the program and get a
listing out. SunOS 4. Anyone have any tips. I have briefly looked an
unexec in GNU emacs sources, but haven't comprehended it yet.

Send me mail, and I will summarize. Thanks!

--

2. named log question

3. ! memory resident programs

4. Solaris x386

5. Make Shared Memory Block Memory-Resident Possible?

6. "Booting" - then nothing...

7. Partiton table info is resident in memory?

8. X and SiS 6326 PCI video board ?

9. Making a binary executable memory resident

10. resident set size vs. real memory

11. how do I accessing memory resident BIOS info at F0000-FFFFF

12. lmitation of resident memory

13. Memory Resident Databases