Reverse Engineering Regular Expressions

I recently published a powerful ruby gem on Github: regexp-examples. This library allows you to generate all (or one random) strings that match any regular expression! (With just a few limitations on what's possible.) To install it yourself and have a quick play is dead easy:

In this post, I will try to explain some of the core logic and techniques that I used to build this library. What is a regular expression, really? How/why is it always possible de-construct a regex, to list all possible strings that match it? How on Earth did I get back-references to work?!

All shall be revealed...

What Is A "Regular" Expression?

Perhaps the most confusing aspect of regular expressions comes from their formal definition, and the fact that several features in the regex language are not really "regular" at all! These "irregular" pieces of syntax are, in short (and by no coincidence!), the "illegal syntax" in my regexp-examples gem.

However, rather than mysteriously telling you what a regular expression isn't, let's try to explain what it is:

There are only really four (yes, that's right, four!) pieces of syntax allowed in a "true" regular expression:

1. The "empty string", usually denoted by: ε
2. Literal characters, e.g. /abc123/
3. The * repeater, e.g. /a*b*c*/
4. The | ("Or") operator, e.g. /a|b|c/

Oh, and there's also brackets - so maybe five pieces of syntax, if you want to count those as well!

Every other piece of syntax is really just a nice way to simplify writing out horrendously long, complicated combinations of the above. Let's try a few examples:

...Hopefully, you get the idea. Or, to put it another way, any regex that can't be expressed in this way is not really regular!
An easy way to see whether or not this is the case is: Does (part of) the regex need to know its surrounding context, in order to determine a match? For example:

These all need to know "what came before", or "what comes next", and are therefore not True Regular Expressions™. Hopefully this makes the common claim that "back-references are not regular" a little more obvious to understand: You need to know what the capture group matches before you can know what the back-reference matches (i.e. knowledge of context). So of course you cannot express such patterns using only those four symbols!

One final point to make, before we move on: There is only really one type of repeater in regex; the others are all nothing more than shorthand:

the fundamental structure of all regex patterns

Understanding this structure is at the very heart of my ruby gem; the whole library architecture depends on (and, for some occasional edge cases, is restricted by!) it.

All True Regular Expressions™ are built using this structure:

/Group-Repeater-Group-Repeater-Group-Repeater-.../

Where every group can, itself, be built using that same structure.

What?! Show me some examples!

I'm glad you asked. Consider the following:

(Yuck! Thankfully we don't normally need to write them out like this!...)
But this lays the foundations for the main purpose of this blog post:

How To Parse A Regular Expression

Without getting bogged down in the nitty-gritty implementation details of parsing, let's dive straight in and look at the internal objects generated by RegexpExamples::Parser:

This may look complicated, but it's essentially not much different to what I described above. There is only one key additional thing to consider: In order to avoid problems with infinity, we must restrict repeaters like * and + to have an upper limit. Taken straight from the gem's README:

Or, in other words, the above regex has been interpreted as equivalent to the following:

/(a{0,2}|b{1,3}){1}/

Like I said above: /Group-Repeater-Group-Repeater-Group-Repeater-.../

How To Generate Examples From A Regular Expression

So, we have our parsed regex. All that remains is to transform this into its possible strings. The trick to this is that all groups and repeaters are given a special method: #result. These results are then built up, piece by piece, to form the full strings that match the regex. Let's take the above example, one step at a time:

• The SingleCharGroup ("a") has one possible result: ["a"]
• Therefore the StarRepeater has three possible results: ["", "a", "aa"]
• Similarly, SingleCharGroup ("b") has one possible result: ["b"]
• Therefore, PlusRepeater has three possible results: ["b", "bb", "bbb"]
• Next, the OrGroup simply concatenates these arrays of possible results, i.e. it has six possible results: ["", "a", "aa", "b", "bb", "bbb"]
• And finally, the top level OneTimeRepeater just returns these same values.

And there you have it, for a fairly simple example! Let's look at one more, to demonstrate perhaps the most important method in the whole gem:

Once again, we make use of PlusRepeater#result and SingleCharGroup#result to build up the final answer from each "partial result".
However, in this case we end up with the following:

[["a", "aa", "aaa"], ["b", "bb", "bbb"], ["c", "cc", "ccc"]]

Where each of those inner arrays is the result of each PlusRepeater. We need to make one more step: Find all possible results, from joining one element from each array, to form a "final result" string. Enter the magic glue that holds this whole thing together:

*I've actually simplified this method slightly, to avoid confusion. The real deal can be found here.
And so, after applying this method to the above array, we end up with:

["abc", "abcc", "abccc", "abbc", "abbcc", .....]

This method gets used a lot, when dealing with more complicated regexes! It is the magic function that allows patterns to be made arbitrarily complicated, with unlimited nesting of groups and so on.

So, there you have it! Now you understand all about how to generate examples from regular expressions, right?...

A subtle metaphor

---

Oh, but...
How do you deal with escaped characters, like \d, \W, etc?
What about regexp options (ignorecase, multiline, extended form)?
What about unicode characters, control codes, named properties, and so on?
How on earth do you correctly parse all of the possible syntax in character sets, such as:

• /[abc]/.examples
• /[a-z]/.examples
• /[^\d\ba-c]/.examples
• /[[:alpha:]&&[a-c]]/.examples

...And I'm barely getting started here! There is a huge range of syntax to consider!


To cut a long story short: parsing is complicated!! However, all the basic principles discussed above still apply.

There is just one final piece of the puzzle that I have mostly avoided up until this point: back-references.
As discussed earlier, back-references are not regular, as they require knowledge of context. They are not strictly possible to fully support with this gem! (And indeed, there are some rare edge cases where my solution does not work.)

But, as promised, all shall be revealed...

How to generate examples with back-references

The important thing to recognise here is that you cannot know what the back-references need to match, until after the rest of the regex example has been generated. For example, consider the following:

/(a|b)\1/.random_example

You cannot possibly know whether the "\1" is meant to be an "a" or a "b", until after the capture group's "partial example" is chosen!
The solution? We cheat, and use a place-holder - then substitute the correct pattern back in later!

The pattern I chose is: __X__, where X is the name of your back-reference (in this case, "1").
There is a lot of intricate logic involved in actually keeping track of the results of these capture groups (perhaps the topic for a follow-up blog post?), so let's gloss over this detail for now. So in summary, examples for the above regex are calculated as follows:

• The SingleCharGroup ("a") has one possible result: ["a"]

• The SingleCharGroup ("b") has one possible result: ["b"]
• The OrGroup has two possible results: ["a", "b"]
• The MultiGroup with group_id=1 has two possible results: ["a", "b"]
• The BackReferenceGroup has one possible result: ["__1__"]
• This gives us a final array of possible results: [["a", "b"], ["__1__"]]
• After applying the permutations_of_strings method, this gives us two "final results": ["a__1__", "b__1__"]
• We now do one final step: Apply a #substitute_backreferences method on each string, to reveal the true strings that match the original regex: ["aa", "bb"]

And now finally, young Padawan, you are ready to see the actual implementation of Regexp#examples:

*Once again, I've been naughty and shown you a slightly simplified version, to avoid confusion. See the real thing over here.

I leave you with one final example, showing the true power of this gem:

Question: What the hell does this ridiculous regex match?! (Note: Don't ever use a regex like that to validate an email address!!)

Answer: (On my average machine, it takes ~0.01 seconds to generate an example string!!)