parse values from ncurses output

parse values from ncurses output

Post by Stanley Jaddo » Tue, 27 Jun 2006 02:51:42



Hi,

I've stored the output of an ncurses program to a file using tee.
Example outputs are uploaded here:
http://student.science.uva.nl/~vjaddoe/gm/

Some fields of the ncurses program are updated every second.
I'd like to parse these values and store them in a separate file, so I can
plot the change of those values over time using Gnuplot.

Unfortunately, the output files stored by tee contain an extreme amount of
terminal control characters (used to update the old values). I'm afraid
simply filtering/removing these characters is not the solution, because
then I cannot see which updated value belongs to which location in the
ncurses program screen output.

If I cat the file, then the complete program output is shown again (but very
fast). It shows the values at the correct locations (see example below),
but I cannot parse the values.

Does anyone know how I can parse values from an ncurses output file (using
for example row and column numbers)? I've tried cat -v, col, strings, perl
regexes...

Thanks in advance,

Stanley

Here is the screen view of the program (output of "cat
rtorrent-output-data", and then pressed Scroll Lock to copy the screen)
-----------------------------------------------------------------------
                   *** rTorrent 0.5.3 - libTorrent 0.9.3 ***
[View: main]
   testfile
  Torrent: 6633.2 / 10240.0 MB Rate: 3991.1 / 2981.9 KB Uploaded:  4460.0 MB
[6

[Throttle off/off KB] [Rate 3991.1/2981.9 KB] [Port: 6998] [U 7/0] [S
0/11/768]
-----------------------------------------------------------------------
From this view I'd like to get the values 6633.2 and 4460.0

 
 
 

parse values from ncurses output

Post by Chris F.A. Johnso » Tue, 27 Jun 2006 03:22:46



Quote:> Hi,

> I've stored the output of an ncurses program to a file using tee.
> Example outputs are uploaded here:
> http://student.science.uva.nl/~vjaddoe/gm/

> Some fields of the ncurses program are updated every second.
> I'd like to parse these values and store them in a separate file, so I can
> plot the change of those values over time using Gnuplot.

> Unfortunately, the output files stored by tee contain an extreme amount of
> terminal control characters (used to update the old values). I'm afraid
> simply filtering/removing these characters is not the solution, because
> then I cannot see which updated value belongs to which location in the
> ncurses program screen output.

    Replace the ESCs with LFs, and search for the code that positions
    the cursor in the appropriate location, then remove the
    positioning code and print the rest. Adjust to taste:

tr '\033' '\012' < rtorrent-output-janeway | awk '
/^\[4;13H/ { sub(/[^H]*H/,"")
             printf "Val1: %s\n", $0 }
/^\[4;68H/ { sub(/[^H]*H/,"")
              printf "Val2: %s\n", $0 }
'

- Show quoted text -

Quote:

> If I cat the file, then the complete program output is shown again (but very
> fast). It shows the values at the correct locations (see example below),
> but I cannot parse the values.

> Does anyone know how I can parse values from an ncurses output file (using
> for example row and column numbers)? I've tried cat -v, col, strings, perl
> regexes...

> Thanks in advance,

> Stanley

> Here is the screen view of the program (output of "cat
> rtorrent-output-data", and then pressed Scroll Lock to copy the screen)
> -----------------------------------------------------------------------
>                    *** rTorrent 0.5.3 - libTorrent 0.9.3 ***
> [View: main]
>    testfile
>   Torrent: 6633.2 / 10240.0 MB Rate: 3991.1 / 2981.9 KB Uploaded:  4460.0 MB
> [6

> [Throttle off/off KB] [Rate 3991.1/2981.9 KB] [Port: 6998] [U 7/0] [S
> 0/11/768]
> -----------------------------------------------------------------------
> From this view I'd like to get the values 6633.2 and 4460.0

--
   Chris F.A. Johnson, author              <http://cfaj.freeshell.org>
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
   ===== My code in this post, if any, assumes the POSIX locale
   ===== and is released under the GNU General Public Licence

 
 
 

parse values from ncurses output

Post by Thomas Dicke » Tue, 27 Jun 2006 05:12:09



Quote:>> Unfortunately, the output files stored by tee contain an extreme amount of
>> terminal control characters (used to update the old values). I'm afraid
>> simply filtering/removing these characters is not the solution, because
>> then I cannot see which updated value belongs to which location in the
>> ncurses program screen output.
>     Replace the ESCs with LFs, and search for the code that positions
>     the cursor in the appropriate location, then remove the
>     positioning code and print the rest. Adjust to taste:

That would work if the application used only cursor-positioning,
but it's unlikely that would be the only method.  ncurses examines
a half-dozen schemes for moving the cursor in each direction,
choosing the one that costs least...

--
Thomas E.*ey
http://www.veryComputer.com/
ftp://invisible-island.net

 
 
 

parse values from ncurses output

Post by Stanley Jaddo » Tue, 27 Jun 2006 08:44:16



>>     Replace the ESCs with LFs, and search for the code that positions
>>     the cursor in the appropriate location, then remove the
>>     positioning code and print the rest. Adjust to taste:

> That would work if the application used only cursor-positioning,
> but it's unlikely that would be the only method.  ncurses examines
> a half-dozen schemes for moving the cursor in each direction,
> choosing the one that costs least...

Indeed, say, 3312.65 was printed at time step 0 and 3328.26 is printed at
time step 1, then only 28.26 is stored, so it's more difficult than I
thought.
 
 
 

parse values from ncurses output

Post by Martijn Lievaar » Tue, 27 Jun 2006 19:56:04



> Does anyone know how I can parse values from an ncurses output file (using
> for example row and column numbers)? I've tried cat -v, col, strings, perl
> regexes...

You could try to set your TERM variable to bumb before starting the
program. Alas, most programs refuse to run in that case. Even if it does
run, ncurses may still use some intelligence that gets in the way. Still
may be worth a try.

M4
--
Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
their time inventing and implementing new, exciting ways for software
to suck. -- Toni Lassila in the Monastry

 
 
 

parse values from ncurses output

Post by Thomas Dicke » Wed, 28 Jun 2006 00:01:58



> Indeed, say, 3312.65 was printed at time step 0 and 3328.26 is printed at
> time step 1, then only 28.26 is stored, so it's more difficult than I
> thought.

I think an effective screen-scraper would have to maintain (like curses)
a full screen's data and to detect when a new screen update occurs.  For
instance, erasing the screen is an easy point to detect.  Also screen
updates _usually_ work from top to bottom (though the optimizer may be
clever enough to start from the line on which the last update occurred).
Assuming the optimizer was not that clever, it would be enough to see
when an update occurs on the current or preceding line, using that as
the starting point.

(It's probably simpler to modify the application to log data - unless
you have no source for it).

--
Thomas E.*ey
http://www.veryComputer.com/
ftp://invisible-island.net

 
 
 

1. parse error with ncurses functions

Thanks for the suggestions on my earlier post. nodelay() does the trick.
But I guess I was too confident as I've not been able to incorporate
ncurses into my program as smoothly as I hoped. I've been making sample
programs to get a feel for ncurses with no problems at all. But now in my
actual program I'm getting "parse error before '='"

It happens with addch, bkgd and initcolor. Commenting those functions out
removes the problem. I thought at first it was because I was trying to use
ncurses in a c++ program, but even if I try to compile the file below as a
C object file, I still get the parse errors. I'm using gcc 2.96 on Linux.

#include "myncurse.h"
#include <curses.h>

void initncurses()
{
        if ((initscr()) == NULL)
        {
                perror("initscr");
                exit(EXIT_FAILURE);
        }
        start_color();

        initpair(PACCOLOR, COLOR_YELLOW, COLOR_BLACK); //1
        initpair(SPACECOLOR, COLOR_WHITE, COLOR_BLACK); //2

        bkgd(' ' | COLOR_PAIR(SPACECOLOR));  //4

        nodelay(stdscr, TRUE);
        noecho();

lines 1, 2 and 4 evoke the "parse error before '='" problem.

In searching around for an idea on this error, I found some info on ncurses
c++ bindings. Does this just enable ncurses to be used in a c++ program, or
is there an OO version of ncurses out there somewhere?

Thanks again.

--
Matt
replace "windycity" with "chicago" to mail me

2. Very New User

3. how to parse 'variable=value' format from file

4. gcc on 5.0.4: illegal instruction

5. Parse an input file using a Key value

6. Unix to PC Connect

7. lk-changelog.pl now to parse bk changes - output

8. Microsoft solicits game console opinions

9. parsing ps -ax output and killing processes

10. parse html output

11. parsing the output of a php-page with a xml-handler like cocoon

12. tcpdump output parsing

13. gdthd: Cannot parse output