tag:blogger.com,1999:blog-7176062489626496619.post2972676144212484311..comments2023-05-03T11:51:28.484-04:00Comments on Lazy Pythonista: Using PLY for Parsing Without Using it for LexingAlexhttp://www.blogger.com/profile/14054821112394577330noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-7176062489626496619.post-84033743134700657492009-12-06T08:00:22.692-05:002009-12-06T08:00:22.692-05:00@David Beazley--yes, generators, or more generally...@David Beazley--yes, generators, or more generally, iterators. Sounds like a good idea. The lexer interface described (method token() which returns a token or None) sounds very similar to an iterator (method next() which returns a value or raises StopIteration).<br /><br />It should be fairly straightforward to write an adapter to convert an iterator's API to the API that the lexer currently expects. Then we could write our lexers as iterators/generators, and use the adapter to plug them into PLY.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7176062489626496619.post-63081220480113631512009-11-23T08:14:53.304-05:002009-11-23T08:14:53.304-05:00Being able to specify alternate lexers has always ...Being able to specify alternate lexers has always been an important part of PLY since the beginning. Prior to creating PLY, I was doing a lot of work on SWIG--which uses a yacc parser and a hand-written lexer. Also, writing a custom lexer would be about the only way to use PLY with streaming I/O sources such as pipes or sockets. I don't know how much people have explored that angle of PLY, but LALR(1) parsers can actually work pretty well on huge data sources or streams because they never do any kind of backtracking (e.g., once a token has been passed into to yacc, it can simply be discarded).<br /><br />My only wish for PLY is that I've sometimes thought about changing the lex/yacc interface to rely on generators (i.e., having lex define a generator). The only reason it doesn't use generators now is that PLY predates generators by a few years and I haven't gotten around to changing it.David Beazleyhttp://www.dabeaz.comnoreply@blogger.com