lex/flex problem scanning a file many times

lex/flex problem scanning a file many times

Post by Nicolas Schmi » Fri, 04 Aug 1995 04:00:00



Does anyone have experience in using lex/flex to scan a file
more than one time? If so, what variables have to be reset
or which functions have to be called to make the scanner
work??

With flex I tried to use yyrestart() after opening the same
file a second time, but the lexer broke, i.e. it scanned
where it shouldn't scan (don't know - strange things happend).

With lex the following happend:
While scanning a very large file, after some 100K, suddenly
the message "Input string too long, limit 200" appeared.
But the file definitely was alright. I used dbx to see the
string yytext grow larger and larger, but it contained only
tokens that _can_ be matched...

If you ever did have an similar problem or anyway have some
kind of remark to this problem please let me know!!

Thanks
Nico Schmidt


 
 
 

lex/flex problem scanning a file many times

Post by Peter Blem » Wed, 09 Aug 1995 04:00:00


: Does anyone have experience in using lex/flex to scan a file
: more than one time? If so, what variables have to be reset
: or which functions have to be called to make the scanner
: work??

As much as I'd like to, I'm not going to get into the details
of why it would be more efficient to simply save your tokens
in a temp file ...

I use the following with success on lots of lexers that are
expected to scan multiple files.  The trick is to make sure
that your pushback buffer is reset when you start into a new
file.

%{
   /* I have had some problems with the input macro on some
   ** platforms, so I also redfine it to something consistant
   ** with this implimentation.
   */

# undef input
# define input() ((((yytchar = yysptr > yysbuf ? U(*--yysptr) : \
        getc(yyin)) == 10) ? (yylineno++,yytchar):yytchar)==EOF ? \
        0 : yytchar)
%}

When I open a new file, I set yyin to point to the new file, and
clear the pushback buffer.

        *(yysptr = yysbuf) = '\0';

Hope that helps.

Peter
--
Peter A. Blemel                      Management Sciences, Inc.

                                     Albuquerque, NM 87110
                                     (505) 255 - 8611

 
 
 

1. lex/flex-problem scanning file many times

Does anyone have experience in using lex/flex to scan a file
more than one time? If so, what variables have to be reset
or which functions have to be called to make the scanner
work??

With flex I tried to use yyrestart() after opening the same
file a second time, but the lexer broke, i.e. it scanned
where it shouldn't scan (don't know - strange things happend).

With lex the following happend:
While scanning a very large file, after some 100K, suddenly
the message "Input string too long, limit 200" appeared.
But the file definitely was alright. I used dbx to see the
string yytext grow larger and larger, but it contained only
tokens that _can_ be matched...

If you ever did have an simliar problem or anyway have some
kind of remark to this problem please let me know!!

Thanks
Nico Schmidt


2. Pcmcia LAN and M4

3. Lex problem (How do i scan a binary file?)

4. Xconfig for Cirrus5428 & Samsung

5. No X programs work / Need flex or lex

6. Boot as standalone

7. Object-oriented lex(flex) and yacc(bison) for c++?

8. WANTED: BINARY DIFFS for large files

9. searching for lex, not flex...

10. lookinf for Flex.Lex and Bison/Yacc ...

11. Need lex not flex for linux

12. flex/lex/bison/yacc parser for C ?

13. Yacc/Lex vs Bison/Flex