|Expected Token Density in Random Stream email@example.com (Andrew Tomazos) (2011-12-07)|
|Re: Expected Token Density in Random Stream firstname.lastname@example.org (Kaz Kylheku) (2011-12-11)|
|Re: Expected Token Density in Random Stream email@example.com (Andrew Tomazos) (2011-12-13)|
|Re: Expected Token Density in Random Stream firstname.lastname@example.org (Gene) (2011-12-19)|
|From:||Andrew Tomazos <email@example.com>|
|Date:||Tue, 13 Dec 2011 06:00:40 -0800 (PST)|
|Posted-Date:||14 Dec 2011 13:28:26 EST|
On Dec 11, 6:56 pm, Kaz Kylheku <k...@kylheku.com> wrote:
> On 2011-12-07, Andrew Tomazos <and...@tomazos.com> wrote:
> > Summary: We want to find out how often a given token appears in a
> > random stream formed by concatenating randomly chosen strings from a
> > given set of strings.
> > (Note hits can overlap each other)
> But tokens do not overlap, so you're not actually extracting tokens. Using
> C tokens as an example, the C token >>= is one hit, not four. The longest
> match calls for extracting three characters and moving on.
Substitute occurrences of the word "token" in my post for "key
string" (or just "string") and reinterpret.
[I suppose, but finding tokens would be more interesting. -John]
Return to the
Search the comp.compilers archives again.