Plot page size and data limits...

Plot page size and data limits...

Post by Val Schmid » Tue, 04 Mar 2003 05:53:28



I have two questions I was hoping someone could help me with. I've looked in
the faq and the manual and didn't find these addressed.

First, is there a way to change the default page/plot dimenstions in
gnuplot?  I'd like to generate a plot for legal or 11x17 paper, or better
yet, a plot to print on a large format plotter (several feet in width and
arbitrary length).

Second, I've been amazed at gnuplots ability to plot large data sets.
Unfortunately, I managed to kill it on a 3 month data file with 1 second
samples.  I assume this is largely due to an insufficient amount of ram on
my machine and not some other limit in gnuplot.    My question is simply,
will a memory addition do the trick, or are there some hard limits to the
number of data points gnuplot can plot?
Or is there perhaps a better way to make a plot of this magnitude?

Thanks in advance,

-Val

 
 
 

Plot page size and data limits...

Post by Ethan A Merri » Tue, 04 Mar 2003 06:50:55




>I have two questions I was hoping someone could help me with. I've looked in
>the faq and the manual and didn't find these addressed.

>First, is there a way to change the default page/plot dimenstions in
>gnuplot?  I'd like to generate a plot for legal or 11x17 paper, or better
>yet, a plot to print on a large format plotter (several feet in width and
>arbitrary length).

For such a large plot you probably want PostScript output.
The gnuplot command 'set term post' does not have a separate sub-option
for specifying the page size, so you have to use the generic command
'set size <xmult>,<ymult>'
Since the default PostScript page size is 8.5x11, to get 11x17 instead
you would want 'set size 11/8.5, 17/11'

Quote:>Second, I've been amazed at gnuplots ability to plot large data sets.
>Unfortunately, I managed to kill it on a 3 month data file with 1 second
>samples.  I assume this is largely due to an insufficient amount of ram on
>my machine and not some other limit in gnuplot.    My question is simply,
>will a memory addition do the trick, or are there some hard limits to the
>number of data points gnuplot can plot?

It is unlikely to be limited by the amount of ram, although conceivably
it is limited by the amount of virtual memory (= ram + swap space on disk).
Gnuplot will keep on requesting more virtual memory as more data points
are read in, up to the point where the operating system refuses to give
it any more.

You will have to provide more information about what operating system
you are running under.  On a unix-like system it is possible that you
can increase the amount of memory available to your process by using
a shell command such as 'unlimit'.

Quote:>Or is there perhaps a better way to make a plot of this magnitude?

Maybe. It depends on what type of plot you are making. For instance,
it turns out that PostScript is a bad choice for huge scatter-plots,
since it writes out a separate command to draw each point even if
the points large fall on top of each other. This can produce immense
output files, which may not be printable because they exhaust the
resources available to your printer or printer driver.
In such a case one of the raster-mode plots is more practical, since
the size of the output file is independent of the number of points
that are plotted.  This won't, however, change the fact the gnuplot itself
will need lots of memory to handle the points as they are read in.

 
 
 

Plot page size and data limits...

Post by Hans-Bernhard Broeke » Wed, 05 Mar 2003 19:51:05



> First, is there a way to change the default page/plot dimenstions in
> gnuplot?  I'd like to generate a plot for legal or 11x17 paper, or better
> yet, a plot to print on a large format plotter (several feet in width and
> arbitrary length).

Use 'set size', but *before* you start the plot, i.e. the sequence should
be
        set size
        set terminal
        set output

Quote:> samples.  I assume this is largely due to an insufficient amount of ram on
> my machine and not some other limit in gnuplot.    

Don't assume where you can observe.  Use a process monitoring program
('top' on Unix, the Task Manager on MS Windows) to see the memory
consumption of gnuplot as it's working.  If it dies of memory
starvation, there should be an error message telling you so (--> "out
of memory").  To give you a rough outline: the relevant data
structures "weighs" about 60 bytes per input point (yes, that is quite
a lot...), so you'ld need at least about 500 MB of (virtual) memory
for such a plot.

One sample per second for three months is really a lot of stuff ---
that's about 8 million datapoints.  Which of course raises a
conceptual question: what on earth could you possibly need a plot of
all those points in a single diagram for?  You won't be able to
discern the x step size of them unless you use something like

        8e6 / 1200 DPI * 0.0254 m/inch =~ 170 meters

of paper, which I'm quite certain you're not.

What this means is that you should think about how to preprocess these
data before you send them for plotting.  gnuplot may be able to bear
with this large sample --- but nobody you will present the final plot
to will.

Quote:> My question is simply, will a memory addition do the trick, or are
> there some hard limits to the number of data points gnuplot can
> plot?  

We try to remove any hard limits as we find them.  I'm quite sure that
absolute size of the datafile has no hard limit but "the sky"
(i.e. the amount of memory the process can acquire).

--

Even if all the snow were burnt, ashes would remain.

 
 
 

Plot page size and data limits...

Post by Val Schmid » Sat, 08 Mar 2003 02:03:43


Thanks for the pointers gentlemen.

With regard to data limits, and to answer your questions...

I'm on a Linux machine with about 120Mb of RAM and 500 MB of swap.

I did watch the process with top and it did appear to stop when all the
virtual memory was filled.  However gnuplot exited with no warning but a
simple "killed" rather than "out of memory".

You are quite right, it's hard to see the utility in plotting this number of
data points.  I just finished reading Edward Tufte's book, "The Visual
Display of Quantitative Information" and without thinking about it too much,
I was keen to make an annual plot from a 1Hz sampled weather station.
Clearly I need to decimate the data to a more reasonable sample rate for the
medium I'd like to display.  Can gnuplot do this on the fly?

Thanks again,

-Val



> > First, is there a way to change the default page/plot dimenstions in
> > gnuplot?  I'd like to generate a plot for legal or 11x17 paper, or
better
> > yet, a plot to print on a large format plotter (several feet in width
and
> > arbitrary length).

> Use 'set size', but *before* you start the plot, i.e. the sequence should
> be
> set size
> set terminal
> set output

> > samples.  I assume this is largely due to an insufficient amount of ram
on
> > my machine and not some other limit in gnuplot.

> Don't assume where you can observe.  Use a process monitoring program
> ('top' on Unix, the Task Manager on MS Windows) to see the memory
> consumption of gnuplot as it's working.  If it dies of memory
> starvation, there should be an error message telling you so (--> "out
> of memory").  To give you a rough outline: the relevant data
> structures "weighs" about 60 bytes per input point (yes, that is quite
> a lot...), so you'ld need at least about 500 MB of (virtual) memory
> for such a plot.

> One sample per second for three months is really a lot of stuff ---
> that's about 8 million datapoints.  Which of course raises a
> conceptual question: what on earth could you possibly need a plot of
> all those points in a single diagram for?  You won't be able to
> discern the x step size of them unless you use something like

> 8e6 / 1200 DPI * 0.0254 m/inch =~ 170 meters

> of paper, which I'm quite certain you're not.

> What this means is that you should think about how to preprocess these
> data before you send them for plotting.  gnuplot may be able to bear
> with this large sample --- but nobody you will present the final plot
> to will.

> > My question is simply, will a memory addition do the trick, or are
> > there some hard limits to the number of data points gnuplot can
> > plot?

> We try to remove any hard limits as we find them.  I'm quite sure that
> absolute size of the datafile has no hard limit but "the sky"
> (i.e. the amount of memory the process can acquire).

> --

> Even if all the snow were burnt, ashes would remain.

 
 
 

Plot page size and data limits...

Post by Hans-Bernhard Broeke » Sat, 08 Mar 2003 02:54:58



> I did watch the process with top and it did appear to stop when all the
> virtual memory was filled.  However gnuplot exited with no warning but a
> simple "killed" rather than "out of memory".

That can happen on a platform with "overcommitted" virtual memory,
i.e. where malloc() will return non-NULL pointers, but the memory they
point to can turn out not to be available when you actually use it.

Quote:> Clearly I need to decimate the data to a more reasonable sample rate
> for the medium I'd like to display.  Can gnuplot do this on the fly?

gnuplot isn't intended for heavy-duty data processing.  For that, you have
tools like 'awk', 'perl' or whatever you like best.

That said, you can indeed do something along these lines.  Try:

        quant=100.0
        plot 'datafile' u (int($1/quant)*quant):2 smooth uniq

This reduces the input to one point every `quant' units in x.  The
other 'smooth' options that sort the input will work similarly.  But
they all rely on the input to fit into memory before they start their
work, so they won't help for insanely large inputs.
--

Even if all the snow were burnt, ashes would remain.

 
 
 

Plot page size and data limits...

Post by Guy Maskal » Sat, 08 Mar 2003 17:15:30


Check out 'ulimit'. Raising the shell's limits **may** give you a little
extra.
Like Hans-Bernhard said, you really want to be looking at preprocessing
the data in some way to reduce the no. points.

> Thanks for the pointers gentlemen.

> With regard to data limits, and to answer your questions...

> I'm on a Linux machine with about 120Mb of RAM and 500 MB of swap.

> I did watch the process with top and it did appear to stop when all the
> virtual memory was filled.  However gnuplot exited with no warning but a
> simple "killed" rather than "out of memory".

> You are quite right, it's hard to see the utility in plotting this number of
> data points.  I just finished reading Edward Tufte's book, "The Visual
> Display of Quantitative Information" and without thinking about it too much,
> I was keen to make an annual plot from a 1Hz sampled weather station.
> Clearly I need to decimate the data to a more reasonable sample rate for the
> medium I'd like to display.  Can gnuplot do this on the fly?

> Thanks again,

> -Val




>>>First, is there a way to change the default page/plot dimenstions in
>>>gnuplot?  I'd like to generate a plot for legal or 11x17 paper, or

> better

>>>yet, a plot to print on a large format plotter (several feet in width

> and

>>>arbitrary length).

>>Use 'set size', but *before* you start the plot, i.e. the sequence should
>>be
>>set size
>>set terminal
>>set output

>>>samples.  I assume this is largely due to an insufficient amount of ram

> on

>>>my machine and not some other limit in gnuplot.

>>Don't assume where you can observe.  Use a process monitoring program
>>('top' on Unix, the Task Manager on MS Windows) to see the memory
>>consumption of gnuplot as it's working.  If it dies of memory
>>starvation, there should be an error message telling you so (--> "out
>>of memory").  To give you a rough outline: the relevant data
>>structures "weighs" about 60 bytes per input point (yes, that is quite
>>a lot...), so you'ld need at least about 500 MB of (virtual) memory
>>for such a plot.

>>One sample per second for three months is really a lot of stuff ---
>>that's about 8 million datapoints.  Which of course raises a
>>conceptual question: what on earth could you possibly need a plot of
>>all those points in a single diagram for?  You won't be able to
>>discern the x step size of them unless you use something like

>>8e6 / 1200 DPI * 0.0254 m/inch =~ 170 meters

>>of paper, which I'm quite certain you're not.

>>What this means is that you should think about how to preprocess these
>>data before you send them for plotting.  gnuplot may be able to bear
>>with this large sample --- but nobody you will present the final plot
>>to will.

>>>My question is simply, will a memory addition do the trick, or are
>>>there some hard limits to the number of data points gnuplot can
>>>plot?

>>We try to remove any hard limits as we find them.  I'm quite sure that
>>absolute size of the datafile has no hard limit but "the sky"
>>(i.e. the amount of memory the process can acquire).

>>--

>>Even if all the snow were burnt, ashes would remain.

 
 
 

Plot page size and data limits...

Post by Val Schmid » Sun, 09 Mar 2003 04:08:31


Thanks once again.

I have one final and unrelated question, then I'll stop being such a bother
(for now:).

Is there a way to set a range such that a certain number of points are
plotted, instead of the max and min?

-Val



> > I did watch the process with top and it did appear to stop when all the
> > virtual memory was filled.  However gnuplot exited with no warning but a
> > simple "killed" rather than "out of memory".

> That can happen on a platform with "overcommitted" virtual memory,
> i.e. where malloc() will return non-NULL pointers, but the memory they
> point to can turn out not to be available when you actually use it.

> > Clearly I need to decimate the data to a more reasonable sample rate
> > for the medium I'd like to display.  Can gnuplot do this on the fly?

> gnuplot isn't intended for heavy-duty data processing.  For that, you have
> tools like 'awk', 'perl' or whatever you like best.

> That said, you can indeed do something along these lines.  Try:

> quant=100.0
> plot 'datafile' u (int($1/quant)*quant):2 smooth uniq

> This reduces the input to one point every `quant' units in x.  The
> other 'smooth' options that sort the input will work similarly.  But
> they all rely on the input to fit into memory before they start their
> work, so they won't help for insanely large inputs.
> --

> Even if all the snow were burnt, ashes would remain.

 
 
 

Plot page size and data limits...

Post by Ethan Merri » Sun, 09 Mar 2003 08:12:40




>Thanks once again.

>I have one final and unrelated question, then I'll stop being such a bother
>(for now:).

>Is there a way to set a range such that a certain number of points are
>plotted, instead of the max and min?

Assuming standard unix-like tools:

plot '< head -1234 file.dat' using ....

That would grab the first 1234 points from the files.  Similar
games could be played with 'tail -1234', or with awk commands
to select every nth line, etc.

--
Ethan A Merritt

 
 
 

Plot page size and data limits...

Post by Hans-Bernhard Broeke » Tue, 11 Mar 2003 20:04:05



> Is there a way to set a range such that a certain number of points are
> plotted, instead of the max and min?

The "every" and "index" options do that.  They reduce the number of
points even before they occupy memory space, so they can be used to
reduce the memory footprint of a plot.
--

Even if all the snow were burnt, ashes would remain.
 
 
 

1. Limit Plotted Data by Unplotted Data

Is it possible given a data set such as:

2003-06-20 19:56:36 38.4853 76.3642  0.65   1.00   3.6   340 22.3
2003-06-21 00:51:04 38.4855 76.3642  0.44   0.74   2.9   345 21.8
2003-06-21 02:30:42 38.4853 76.3642  0.58   0.96   3.1   340 21.6

to plot data from columns 4 and 3 excluding those lines where columns 1
& 2 (collectively a date value) fall outside of a range?

I tried this:

plot 'file.dat' using 4:( ( $1 > '2003-07-20 21:21:00' && $1 <
'2003-07-23 21:21:00' ) ? $3 : 1/0)

but to no avail.  Any help would be greatly appreciated.

2. FS: BCD5000A Animation Controller Module

3. OpenGL Optimizer opviz: data size limits?

4. AVI's bomb Max 1.2

5. Plotting data sets of different sizes

6. Strange Request, but this if for real

7. Need an EGA page-to-page limited-area blit routine (C)

8. Should I upgrade?

9. Page size, image size & stuff

10. Don't plot missing data in a timeseries line plot

11. Plotting multiple plots on a single page.

12. plotting 2 timeseries data files on one plot with pre3.6/pl273

13. How to Plot multiple plot on same page..