Introduction to Python – Regular Expressions

If you are interested to learn about the Python Inheritance

A regular expression is a special sequence of characters that helps you match or find other strings or sets of strings, using a specialized syntax held in a pattern. Regular expressions are widely used in UNIX world. The Python module re provides full support for Perl-like regular expressions in Python. The re module raises the exception re.error if an error occurs while compiling or using a regular expression. We would cover two important functions, which would be used to handle regular expressions. But a small thing first: There are various characters, which would have special meaning when they are used in regular expression. To avoid any confusion while dealing with regular expressions, we would use Raw Strings as r’expression’.

The match Function

This function attempts to match RE pattern to string with optional flags.

Here is the syntax for this function −

re.match(pattern, string, flags=0)

Here is the description of the parameters −

The re.match function returns a match object on success, None on failure. We usegroup(num) or groups() function of match object to get matched expression.

Example

#!/usr/bin/python
import re

line = "Cats are smarter than dogs"

matchObj = re.match( r'(.*) are (.*?) .*', line, re.M|re.I)

if matchObj:
   print "matchObj.group() : ", matchObj.group()
   print "matchObj.group(1) : ", matchObj.group(1)
   print "matchObj.group(2) : ", matchObj.group(2)
else:
   print "No match!!"

When the above code is executed, it produces following result −

matchObj.group() :  Cats are smarter than dogs
matchObj.group(1) :  Cats
matchObj.group(2) :  smarter

The search Function

This function searches for first occurrence of RE pattern within string with optional flags.

Here is the syntax for this function −

re.search(pattern, string, flags=0)

Here is the description of the parameters −

The re.search function returns a match object on success, none on failure. We use group(num) or groups() function of match object to get matched expression.

Example

#!/usr/bin/python
import re

line = "Cats are smarter than dogs";

searchObj = re.search( r'(.*) are (.*?) .*', line, re.M|re.I)

if searchObj:
   print "searchObj.group() : ", searchObj.group()
   print "searchObj.group(1) : ", searchObj.group(1)
   print "searchObj.group(2) : ", searchObj.group(2)
else:
   print "Nothing found!!"

When the above code is executed, it produces following result −

searchObj.group() :  Cats are smarter than dogs
searchObj.group(1) :  Cats
searchObj.group(2) :  smarter

Matching Versus Searching

Python offers two different primitive operations based on regular expressions: match checks for a match only at the beginning of the string, while search checks for a match anywhere in the string (this is what Perl does by default).

Example

#!/usr/bin/python
import re

line = "Cats are smarter than dogs";

matchObj = re.match( r'dogs', line, re.M|re.I)
if matchObj:
   print "match --> matchObj.group() : ", matchObj.group()
else:
   print "No match!!"

searchObj = re.search( r'dogs', line, re.M|re.I)
if searchObj:
   print "search --> searchObj.group() : ", searchObj.group()
else:
   print "Nothing found!!"

When the above code is executed, it produces the following result −

No match!!
search --> searchObj.group() :  dogs

Search and Replace

One of the most important re methods that use regular expressions is sub.

Syntax

re.sub(pattern, repl, string, max=0)

This method replaces all occurrences of the RE pattern in string with repl, substituting all occurrences unless max provided. This method returns modified string.

Example

#!/usr/bin/python
import re

phone = "2004-959-559 # This is Phone Number"

# Delete Python-style comments
num = re.sub(r'#.*$', "", phone)
print "Phone Num : ", num

# Remove anything other than digits
num = re.sub(r'\D', "", phone)    
print "Phone Num : ", num

When the above code is executed, it produces the following result −

Phone Num :  2004-959-559
Phone Num :  2004959559

Regular Expression Modifiers: Option Flags

Regular expression literals may include an optional modifier to control various aspects of matching. The modifiers are specified as an optional flag. You can provide multiple modifiers using exclusive OR (|), as shown previously and may be represented by one of these −

Regular Expression Patterns

Except for control characters, (+ ? . * ^ $ ( ) [ ] { } | \), all characters match themselves. You can escape a control character by preceding it with a backslash. Following table lists the regular expression syntax that is available in Python −

Regular Expression Examples

Literal characters

Character classes

Special Character Classes

Repetition Cases

Nongreedy repetition

This matches the smallest number of repetitions −

Grouping with Parentheses

Backreferences

This matches a previously matched group again −

Alternatives

Anchors

This needs to specify match position.

Special Syntax with Parentheses

Python Regular Expression Functions

We have a few functions to help us use Python regex.

1. match()

match() takes two arguments- a pattern and a string. If they match, it returns the string. Else, it returns None. Let’s take a few Python regular expression match examples.

>>> print(re.match('center','centre'))

Output

None>>> print(re.match('...\w\we','centre'))

Output

<_sre.SRE_Match object; span=(0, 6), match=’centre’>

2. search()

search(), like match(), takes two arguments- the pattern and the string to be searched. Let’s take a few examples.

>>> match=re.search('aa?yushi','ayushi')>>> match.group()

Output

‘ayushi’>>> match=re.search('aa?yushi?','ayush ayushi')>>> match.group()

Output

‘ayush’>>> match=re.search('\w*end','Hey! What are your plans for the weekend?')>>> match.group()

Output

‘weekend’>>> match=re.search('^\w*end','Hey! What are your plans for the weekend?')>>> match.group()

Output

Traceback (most recent call last):File “<pyshell#337>”, line 1, in <module>

Here, an AttributeError raised because it found no match. This is because we specified that this pattern should be at the beginning of the string. Let’s try searching for space.

>>> match=re.search('i\sS','Ayushi Sharma')>>> match.group()

Output

‘i S’>>> match=re.search('\w+c{2}\w*','Occam\'s Razor')>>> match.group()

Output

‘Occam’

It really will take some practice to get it into habit what the metacharacters mean. But since we don’t have so many, this will hardly take an hour.

Python Regex Options

The functions we discussed may take an optional argument as well. These options are:

1. Python Regular Expression IGNORECASE

This Python Regex ignore case ignores the case while matching. Take this example of Python Regex IGNORECASE:

>>> match=re.findall(r'hi','Hi, did you ship it, Hillary?',re.IGNORECASE)>>> <strong>for</strong> i <strong>in</strong> match: print(i)

Output

Hihi
Hi

2. Python MULTILINE

Working with a string of multiple lines, this allows ^ and $ to match the start and end of each line, not just the whole string.

>>> match=re.findall(r'^Hi','Hi, did you ship it, Hillary?\nNo, I didn\'t, but Hi',re.MULTILINE)>>> <strong>for</strong> i <strong>in</strong> match: print(i)

Output

Hi

3. Python DOTALL

.* does not scan everything in a multiline string; it only matches the first line. This is because . does not match a newline. To allow this, we use DOTALL.

>>> match=re.findall(r'.*','Hi, did you ship it, Hillary?\nNo, I didn\'t, but Hi',re.DOTALL)>>> <strong>for</strong> i <strong>in</strong> match: print(i)

Output

Hi, did you ship it, Hillary?No, I didn’t, but Hi

Greedy vs Non-Greedy

The metacharacters *, +, and ? are greedy. This means that they keep searching. Let’s take an example.

>>> match=re.findall(r'(&lt;.*>)','&lt;em>Strong&lt;/em> &lt;i>Italic&lt;/i>')>>> <strong>for</strong> i <strong>in</strong> match: print(i)

Output

<em></em> <i> </i>

This gave us the whole string, because it greedily keeps searching. What if we just want the opening and closing tags? Look:

print(i)&gt;&gt;&gt; match=re.findall(r'(&lt;.*?&gt;)','&lt;em&gt;Strong&lt;/em&gt; &lt;i&gt;Italic&lt;/i&gt;')&gt;&gt;&gt; <strong>for</strong> i <strong>in</strong> match: print(i)

Output

<em></em> <i> </i>

The .* is greedy, and the ? makes it non-greedy.

Alternatively, we could also do this:>>> match=re.findall(r'</?\w+>’,'<em>Strong</em> <i>Italic</i>’)>>> for i in match: print(i)

Output

<em></em> <i> </i>

Here’s another example:

>>> match=re.findall('(a*?)b','aaabbc')>>> <strong>for</strong> i <strong>in</strong> match: print(i)

Output

aaa

Here, the ? makes * non-greedy. Also, if we would have skipped the b after the ?, it would have returned an empty string. The ? here needs a character after it to stop at. This works for all three- *?, +?, and ??. Similarly, {m,n}? makes it non-greedy, and matches as few occurrences as possible.

Substitution

We can use the sub() function to substitute the part of a string with another. sub() takes three arguments- pattern, substring, and string.>>> re.sub(‘^a’,’an’,’a apple’)

Output‘

an apple’

Here, we used ^ so it won’t change apple to anpple. The grammar police approve.

Python Regex Applications

So, we learned so much about Python regular expressions, but where do we use them? They find use in these places:

• Search engines
• Find and Replace dialogues of word processor and text editors
• Text processing utilities like sed and AWK
• Lexical analysis