Regular Expressions, Your Job Requires Them

imageIn preparing my talk on Regular Expressions that I'll be giving this upcoming weekend at the Chicago Code Camp I decided to repost this article which I originally posted in February of last year.  Rereading the information contained within my position or thoughts on developers and regular expression have not changed.

I've thought about writing this post several times over the past two years.  Having had regular expressions come up three times last week, I thought it time to address the lack of programmers out there who understand regular expressions.  The sheer amount of fear surrounding regular expressions and the work that goes into avoiding them is astonishing.

Last year I used to troll around the asp.net forums and quite frequently I would answer the regex questions.  One question was posted which illustrates the problem with regexes among developers.

"...and i also i need to add a validator for the password textbox where the user is required to fill atleast [sic] 6 characters"

I suggested a solution to the problem using a regular expression validator. Making sure there are at least 6 characters, is a simple regex (example: \w{6,}), and yet my solution was met with skepticism.  The following was said, in the event a change was requested,

"Putting a new version of a web site can take a surprising amount of time than can go into man-weeks". 

Man-weeks?!?!?  To change a regular expression?!?!?  I see two problems, first the original developer who didn't know that regex would easily solve their problem.  The second problem is the other developer who doesn't know regex advocating his way as "the way", in effect, spreading his ignorance.  The first developer is easily forgiven, the second is not.

It's been said programmers can't program when faced with a simple FizzBuzz test, Imran states:

"Want to know something scary? - the majority of comp sci graduates can’t. I’ve also seen self-proclaimed senior programmers take more than 10-15 minutes to write a solution."

I'll pile on.  You want to know something scary?  The majority of professional programmers can't write regular expressions, even simple ones.  I'm not the first to say this.  Last year, at the ALT.NET conference, Scott Guthrie made the following statement when talking about routes in the new MVC framework:

"It's pluggable, so you can use Regexes...<some incoherent stuff>...if you wanna use regexes you can.  What we found is, regexes are super powerful, but only about 10% of people actually understand 'em."

Are regular expressions easy to understand? Well, let me ask you, was HTML easy when you started?  Were you born understanding the following HTML?

<fieldset class="CheckRadio">
    <div id="OngoingEventContainer">
        <input type="checkbox" id="OngoingEvent" name="OngoingEvent" value="1" />
        <label for="OngoingEvent">
            This is an ongoing event (no dates and times)
        </label>
    </div>
</fieldset>

If you understand the above, you didn't always.  My guess is that at some point you buckled down and learned HTML because you're job requires it.  Well, if you're a programmer, web or windows, you need to know regular expressions, your job requires it, it's that simple. 

imageRegular expressions have been around so long that they're deeply ingrained in many of the tools we use.  Christopher Bennage illustrates how regular expressions solved a recent problems in Visual Studio.  In a recent post he posts:

"Then I realized that I was missing the simple solution. Ctrl+F and a regular expression!"

I don't know that many people would be able to come to the conclusion that Christoper did.  It's my belief that regular expressions are fundamental, yet the average developer doesn't treat them as such.  They're ultimately doing themselves a disservice.

Regular Expressions are a tool that should be in every programmers bag.  If you don't understand regular expressions and do a google search every time you need a regular expression, shame on you!  It's time to bite the bullet and learn regular expressions.


Posted 05-25-2009 9:18 PM by Tim Barcz

[Advertisement]

Comments

Jay wrote re: Regular Expressions, Your Job Requires Them
on 05-25-2009 11:07 PM

Don't do a google search every time you need a regular expression, but if google search accepted regular expressions...

I can't agree with you more about the centrality of regular expressions to our work and the paradoxical lack of emphasis they are given.  I'm a self-taught developer who wasted so much time messing with string manipulation to accomplish relatively complex tasks in VB .NET 1.0 and 1.1 that would have been trivial if I'd known about RegEx.

I'd seen the term in the forums occasionally, and finally years later looked it up.  Once I realized what it was and what it could be used for, I got hold of Mastering Regular Expressions from O'Reilly and went to town fixing all my C# string spaghetti.

These days my bulletin board has 4 items on it,

RegEx metacharacter cheet sheets for:

.NET

Visual Studio

Office VBA

Vim, ViEmu

The last really permeates RegEx into your work and you get the benefits in productivity.

Thanks for bringing the issue up, and good luck with the Chicago Code Camp presentation.

Jeremy Wiebe wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 12:43 AM

I agree that Regex's are very useful.  I've seen "overusage" of them where their usage complicated the solution.

One place that you mention (via Christopher Brennage) is the usage of regex's in VS's Search and Replace dialog.  I've come to rely on that for all kinds of text manipulation.  From simple changes to my code to generating SQL statements via columnar data from Excel.  

I don't see why programmers can't learn regex's if they can learn multithreaded programming or other complex concepts.  

Anne Epstein wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 1:30 AM

I've seen developers write pages of code that could be done with a single, simple regex.  I can understand that at first they seem a bit strange and even initially hard to understand, but if your task is fundamentally string matching, writing those pages of code is a lot more chances for error than a simple expression that directly intent.  

In fact, I think once you get over the syntax, regexes express what they do in a way that MUCH better reflects how people think about parsing than the alternatives most people use.  For a simplified example, think of an email address: you might think about a valid address as "ok, some amount of letter or numbers, maybe underscores or periods or dashes(say, at least one character) , an @ sign, a few more characters like the ones before, and then a .com, .net or something else valid." using regex, you can say pretty much exactly that.  It's abstracting the string matching expression to something closer to the thought-Otherwise, you might have loops, subloops, substring picked out and compared, arrays, etc ... which is really not a natural way to think about parsing-as programmers, many of us have just trained ourselves to think in unnnatural ways.  

Artem wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 6:41 AM

I'm using regular expressions constantly, but it is a pain for me every time I have to switch to Visual Studio syntax in Search/Replace. Why not make it compatible to standard .NET syntax??

KG2V wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 7:26 AM

The huge problem with RegEx - which flavor?

Visual Studio uses one flavor, the Editor I use uses a slightly different flavor, or a 3rd flavor depending (posix/Unix Extended RE)

I use em, but to be honest, I usually have to grab a cheat sheet for "non trivial" regex expressions, and often end up breaking the RegEx into multiple parts.  The Greedy Inclusion often messes up the way I think about the problem

My most common problem is

"Give me all the string segments that begin with a (, and with a ), where ST=?? (where ?? changes) and OFC=?? and often one or two other conditions

I usually find it easier to first do a search replace on )( and replace with )/n(  then select the right ST, then select the right OFC

Eduardo Miranda wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 9:08 AM

I consider regex to be cumberstone of unreadable code. And yes, whenever I have the option of not using I do so, even if takes 10 lines of readable and maintainable code.

Regarding tools that use regex they give a lot of power, you should take advantage of that, but the "flavor" thing get in the way big time.

Tim Barcz wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 9:18 AM

@KG2V

I understand the "flavor" issue but that is a hurdle not a roadblock.  As a developer every day you move in and out of languages with ease.  Switching between SQL, CSS, HTML, C#, and JavaScript.  I can't imagine small differences in dialect matter THAT  much.

DotNetShoutout wrote Regular Expressions, Your Job Requires Them - Tim Barcz - Devlicio.us
on 05-26-2009 10:05 AM

Thank you for submitting this cool story - Trackback from DotNetShoutout

Jeremy Wiebe wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 10:36 AM

@Tim @KG2V regarding flavours, one of the annoying things I've found is that .NET regexes differ from the ones used in the VS search/replace dialog and quite significantly, at that (":Wh" vs "/s" for whitespace, for example).  I'll never understand that design decision because you can control whether you are searching by Regular Expression or not.

Stacy Spear wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 12:02 PM

Great article! I fell in love with regular expressions when having to search humongous log files for needed information. I can't imagine working without them.

I agree that dialect differences retard their use by most people. Why there are differences is as simple as everyone thinks there's is the best!

BTW, I'm not really a programmer, but was able to code a successful fizzbuzz in 5 minutes. Makes we wonder if students today are being taught anything about logic.

Stevi Deter wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 12:57 PM

I was happy to see that Microsoft is now including RegEx on the Application Development Foundation certification exam.

But I dismayed when I looked at the RegEx section in the MCTS Self-Paced Training Kit for the exam, where you can practically hear the whine "RegEx is HAAAARD."

John Safrit wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 2:42 PM

Maybe it's just me, but in the article when you talk of a regex to check a minimum password length, why not use string length?  Sometimes if regex is your "hammer" of choice, all your problems look to be nails.  Regex has it's place, but it is often overused.

Tim Barcz wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 2:55 PM

@John,

Your example of string length is fine, but the question in the post above was in the context of a validation control.  A regex validation control easily validates length.

"Regex has it's place, but it is often overused."  I would strongly disagree as would others who have commented (see Anne's comments above).  I would say it's the opposite, as far as text processing goes, they are far under-used.

Jeff Perrin wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 4:12 PM

"Regular Expressions, Your Job Requires Them"

I agree that regular expressions are nice to know and can simplify a bunch of things. But I strongly disagree with the assertion that you have to know them to be a successful developer. Can't we just have a nice discussion about their uses without throwing absolute statements around in order to generate a reaction?

The fact of the matter is, not a single programming job I've had (mostly web, with a couple desktop apps thrown in) has required knowledge of regex. I consider myself to be a good developer, and I can neither read nor write a regex without consulting a cheat sheet. I'm not proud of this, but I'm not apologetic about it either.

I think that as nerds, we tend to put an inordinate amount of emphasis on nerdy things that don't matter in the grand scheme of things. Yes, regex's, unit tests and other stuff we do are seen as important to us, but in the end the only thing that really matters is that the app we're building works for the client within acceptable parameters (and keeps working over time). There are many ways to accomplish this goal, and there are no absolutes.

Marius wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 4:52 PM

I agree with you completely. It amazes me how some people won't even look at a regular expression. The worst part is that for simple matches, you don't really have to know much either. You can get very far with a limited subset of regexp

Marius wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 4:58 PM

"The fact of the matter is, not a single programming job I've had (mostly web, with a couple desktop apps thrown in) has required knowledge of regex. "

You say yourself that you have to resort to a cheat sheat to write regular expressions. That says to me that you have never really put any effort into learning them.

You are right that none of your jobs requires regular expressions, but there are so many ways they can make your job simpler and get things done faster.

I'm not kidding. At my work I see people who don't want to learn regular expressions, and they waste incredible amounts of time. It's not just simple tasks like validation, but it should also be a part of your general tool set. Most of my regexp use is done outside the source code, but is used for making my work day more efficient.

Iain wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 5:42 PM

I have used RegEx like a lot of your other posters in the past and became quite proficent and parsing the text out of log file, but last week I had to go back to basics to remeber it, the syntax is the problem, I maybe time to use a DSL to write them to compile down to a RegEx but it a mater of making as consise

Tim Barcz wrote re: Regular Expressions, Your Job Requires Them
on 05-26-2009 5:56 PM

@Iain,

I would consider RegEx a DSL.  It gives you immense power in a short concise way.

What I think you mean is something readable...if that's the case check out. weblogs.asp.net/.../introducing-linq-to-regex.aspx

It's very verbose, and I personally would not use it, but generally think the idea is interesting.

BlindWanderer wrote re: Regular Expressions, Your Job Requires Them
on 06-02-2009 6:54 PM

I own the pictured book... be sure if you order the book you order a second edition, the one picture is a first edition.

There is just some types of programming you can't do without without RegEx. For the stuff I do, it would be impossible without RegEx. I pretty much forced myself to learn RegEx so I could write better GreaseMonkey scripts. XPath was much the same, using DOM was painful. With XPath and RegEx you can write code that has a certain amount of future proofing built in.

Kyralessa wrote re: Regular Expressions, Your Job Requires Them
on 06-11-2009 11:00 PM

I suspect that if experienced programmers really can't write FizzBuzz quickly, it's probably because they're assuming it can't be that simple, and they're trying to figure out what they're missing.

I've read of a similar phenomenon of teachers flunking a "basic history exam".  Some of them flunk because they've learned advanced history, and the answers to a lot of questions become "it depends"; they have to ascertain whether the exam writer's answer takes this or that obscure fact into account.

Robz wrote re: Regular Expressions, Your Job Requires Them
on 06-24-2009 11:15 PM

+1 for Regular Expressions! I love using them for search replace when they are warranted.

Sometimes I have to ask which is going to take more time to do, find and replace each, or write a regular expression to do it?!

That's just me though... I could definitely do much more with RegEx.

Brian wrote re: Regular Expressions, Your Job Requires Them
on 07-23-2009 11:50 AM

For all it's complexity and power it lacks the simple ability to validate zero length strings.

Tim Barcz wrote re: Regular Expressions, Your Job Requires Them
on 07-23-2009 12:04 PM

@Brian,

Not sure what you mean.  If you're talking about the ability to measure that a string is NOT zero length that is very easy: ".+" (meaning "any character one or more repetitions".

If you're talking about the ability to see if the string IS zero length, regex isn't really the tool, just do string.lenght.  Otherwise if you must use regex, use the pattern above and if you have a match you have a > 0 length string.

kenali dan kunjungi objek wisata di pandeglang wrote re: Regular Expressions, Your Job Requires Them
on 10-26-2009 7:01 AM

wow, this is really what I am looking for. thanks for that.

top bookmarking sites wrote re: Regular Expressions, Your Job Requires Them
on 01-18-2013 9:32 AM

7FbaEJ Thanks-a-mundo for the blog post.Really thank you!

discount imitrex wrote re: Regular Expressions, Your Job Requires Them
on 01-27-2013 8:26 AM

DM93I3 Muchos Gracias for your blog article.Really looking forward to read more. Fantastic.

pills for weight loss wrote re: Regular Expressions, Your Job Requires Them
on 02-01-2013 11:05 PM

tvQXzM Major thanks for the blog post.Thanks Again.

buy viagra online wrote re: Regular Expressions, Your Job Requires Them
on 02-03-2013 5:30 AM

XmIn88 Thanks so much for the blog post. Really Great.

buy stendra wrote re: Regular Expressions, Your Job Requires Them
on 02-15-2013 3:24 PM

79awpx Appreciate you sharing, great blog post. Really Cool.

buy viagra discount wrote re: Regular Expressions, Your Job Requires Them
on 02-23-2013 10:41 PM

BW8y1D Very good article post.Much thanks again. Will read on...

clomid no prescription wrote re: Regular Expressions, Your Job Requires Them
on 02-27-2013 6:52 AM

OEE0GS This is one awesome blog post.Really looking forward to read more. Great.

http://1buyviagrahere.com/ wrote re: Regular Expressions, Your Job Requires Them
on 03-02-2013 6:20 PM

YwrtHF Really informative post.Much thanks again. Much obliged.

bookmaring service wrote re: Regular Expressions, Your Job Requires Them
on 03-13-2013 3:06 PM

fGE52N I truly appreciate this post.Really looking forward to read more. Much obliged.

bookmaring service wrote re: Regular Expressions, Your Job Requires Them
on 03-15-2013 12:08 AM

LI5pna Im thankful for the blog article.Really thank you! Keep writing.

social bookmarking service wrote re: Regular Expressions, Your Job Requires Them
on 03-24-2013 7:43 PM

51XJMZ Thanks-a-mundo for the article.Much thanks again. Will read on...

Social bookmarks wrote re: Regular Expressions, Your Job Requires Them
on 04-04-2013 12:35 AM

lxukyX I really liked your blog.Much thanks again.

comedy shirts wrote re: Regular Expressions, Your Job Requires Them
on 04-06-2013 1:22 AM

Say, you got a nice blog article. Will read on...

social bookmarking service wrote re: Regular Expressions, Your Job Requires Them
on 04-12-2013 8:27 AM

lPoJCC Thanks so much for the blog.Much thanks again. Really Great.

Social bookmarks wrote re: Regular Expressions, Your Job Requires Them
on 04-15-2013 10:00 PM

ocoy7V I truly appreciate this article.Thanks Again. Fantastic.

digital camera guide wrote re: Regular Expressions, Your Job Requires Them
on 05-14-2013 5:09 AM

wkpBta Fantastic post. Want more.

social bookmarking service wrote re: Regular Expressions, Your Job Requires Them
on 06-05-2013 9:12 PM

lONAYB Thanks-a-mundo for the post.Really looking forward to read more. Fantastic.

best social bookmarks wrote re: Regular Expressions, Your Job Requires Them
on 06-19-2013 9:15 AM

iAMCFC This is one awesome blog.Really looking forward to read more. Keep writing.

good social bookmarks wrote re: Regular Expressions, Your Job Requires Them
on 06-21-2013 9:07 AM

Cj2wuL Im thankful for the blog article. Great.

great service wrote re: Regular Expressions, Your Job Requires Them
on 07-04-2013 11:41 AM

aILnDT I really like and appreciate your article post.Really thank you! Much obliged.

cool news wrote re: Regular Expressions, Your Job Requires Them
on 07-09-2013 9:40 PM

QWwLrf Awesome blog article.Thanks Again. Great.

super news wrote re: Regular Expressions, Your Job Requires Them
on 07-11-2013 10:02 PM

AueO6b I loved your article.Much thanks again.

buy viagra online cheap wrote re: Regular Expressions, Your Job Requires Them
on 07-24-2013 1:39 AM

Very informative blog.Much thanks again. Fantastic.

buy cialis online cheap wrote re: Regular Expressions, Your Job Requires Them
on 07-25-2013 3:42 AM

Very neat article.Thanks Again. Really Great.

hot news wrote re: Regular Expressions, Your Job Requires Them
on 07-26-2013 6:27 AM

pJLiWs Thanks so much for the blog article.Thanks Again. Much obliged.

social bookmarks wrote re: Regular Expressions, Your Job Requires Them
on 07-28-2013 9:42 PM

IkkLD9 I really liked your article.Really looking forward to read more.

super news wrote re: Regular Expressions, Your Job Requires Them
on 08-02-2013 5:07 PM

q7Zc0E Say, you got a nice article.Much thanks again. Keep writing.

awesome moldavian news wrote re: Regular Expressions, Your Job Requires Them
on 08-04-2013 4:33 PM

VUr9YE I really like and appreciate your blog.Really looking forward to read more. Much obliged.

great link buildng wrote re: Regular Expressions, Your Job Requires Them
on 08-19-2013 8:41 AM

o4ZOWe Thanks again for the blog article. Awesome.

awesome links for you wrote re: Regular Expressions, Your Job Requires Them
on 08-19-2013 7:58 PM

qd37Q5 I cannot thank you enough for the post.Really thank you! Want more.

awesome links for you wrote re: Regular Expressions, Your Job Requires Them
on 08-21-2013 9:43 AM

VBl5xE Very informative blog article.Really thank you! Fantastic.

buy viagra online cheap wrote re: Regular Expressions, Your Job Requires Them
on 09-02-2013 6:59 AM

I cannot thank you enough for the blog post.Really thank you! Awesome.

best link building wrote re: Regular Expressions, Your Job Requires Them
on 09-03-2013 9:17 PM

JO7g16 Thanks for sharing, this is a fantastic post. Keep writing.

seo service wrote re: Regular Expressions, Your Job Requires Them
on 09-05-2013 7:27 PM

3cAfSU Great, thanks for sharing this post.Really thank you! Much obliged.

website wrote re: Regular Expressions, Your Job Requires Them
on 09-08-2013 6:24 AM

Thank you ever so for you blog post. Will read on...

make money online wrote re: Regular Expressions, Your Job Requires Them
on 09-13-2013 7:15 AM

PwKhF1 Awesome blog article.Really thank you! Really Cool.

awesome link building wrote re: Regular Expressions, Your Job Requires Them
on 09-24-2013 4:02 PM

IeY4eq Really appreciate you sharing this blog post.Much thanks again. Want more.

cheap seo work wrote re: Regular Expressions, Your Job Requires Them
on 09-28-2013 11:30 PM

CsZTgS Great, thanks for sharing this blog article. Fantastic.

best link build wrote re: Regular Expressions, Your Job Requires Them
on 10-16-2013 11:49 AM

UhLgYX Looking forward to reading more. Great article.Much thanks again. Will read on...

top seo guys wrote re: Regular Expressions, Your Job Requires Them
on 10-25-2013 5:54 AM

HoIEun Thank you ever so for you article.Much thanks again. Much obliged.

link building wrote re: Regular Expressions, Your Job Requires Them
on 10-31-2013 9:19 AM

MF6vvj Hey, thanks for the blog.Thanks Again. Really Great.

awesome site wrote re: Regular Expressions, Your Job Requires Them
on 11-20-2013 8:06 PM

98hSl0 I really liked your article post.Really thank you!

you want this now wrote re: Regular Expressions, Your Job Requires Them
on 12-14-2013 3:22 PM

TcArKT Thanks so much for the blog post.

nice seo guys wrote re: Regular Expressions, Your Job Requires Them
on 03-25-2014 8:04 PM

hU1mmX A round of applause for your blog post.Thanks Again. Much obliged.

check this out now wrote re: Regular Expressions, Your Job Requires Them
on 04-05-2014 12:33 AM

CUg62U A round of applause for your blog article.Thanks Again. Want more.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)