Perl can't do all the work for you. If you want it to decide according to context whether "4.5" is a string or a number, you have to provide a context. I don't know about you, but I'd rather juggle two sets of operators than have to convert my numberish strings to numbers explicitly.

I say forget grep, awk and whatnot. sed has it all, if you can be sure the input is in a fixed format.

This is what you need:
wget -O - http://www.mjtsai.com/blog/files/2002-11-25-sample-input.txt 2>/dev/null | sed -ne '/\("[^"]*"\t\)\{26\}"1"/s#\("[^"]*"\t\)\{17\}"\([^"]*\)".*$#\2#p'

and I guess this is also as fast as you can get...

I have read Eric Raymond's discussion on why he prefers Python. I think he has good points, but he wrote those comments in 2000. The Perl community has come a long way. The best rejoinder is probably Perl Best Practices, by Damian Conway. But always remember: You can write fortran in any language. :)

As for scalability of perl scripts, I have heard that Perl programs get quite difficult to maintain after they get much larger than about 100 lines. I agree with that - maintaining a sense of context becomes mind-bending. My rule of thumb is that if my script gets much longer than 200 lines, it probably has some ideas that I can (and should) abstract into modules.

I recently ran into this very problem with a really simple simulator for my research. The program reached maybe 500 or 600 lines before I finally refactored it into many modules. Now I am back to writing 15 line scripts using my newly created libraries. It was painful but very enlightening to refactor and now I have an extensible (extendable) library that I can use for my research that should carry me through my thesis and quite possibly beyond.

By the way, you might think that my collection of modules should have the same 200-line mental limitations, but I find that I can work with a module with many thousands of lines of code and it doesn't bother me. This is because the modules tends to focus on a much more clearly defined target, which makes the context much narrower. Also, I tend to be much stricter with my libraries than with my scripts, especially when it comes to documentation.

Is this 200+ line mental limit on Perl scripts a feature or a bug? Obviously it depends on what you're trying to do, but for me, it has turned out to be a great feature.

P.S. Just to be sure everybody is clear: Perl can do serious computational stuff, at the computational level of C/Fortran. It's called PDL. I do all sorts of numerical stuff with PDL as grad student in physics, doing (single-core) simulations at the moment. I linked to PDL's web page, and while it looks old, the modules are still in active maintenance and development. And mailing list is awfully nice, too!

I used REXX for a long time, but use Python almost exclusively now. I never liked Perl. Why in the WORLD did someone come up with the idea in the late 1980s that there had to be TWO sets of comparison operators, one for numbers and one for strings? That looks like SUCH a "lazy compiler/interpreter writer" thing that one has to ask WTF? I mean even lowly BASIC didn't need that kludge. The first time I encountered that, I pretty much figured that Perl was a poorly designed language hack, even if it did make some things "easy to do".

Read Eric Raymond's 2000 article on Why Python? to get from someone who loved Perl and is well respected in the industry as a user of programming languages why Python is definitively a "better language".

BTW, I write/have written in MANY languages over the years, including like John, in ALC for the 370, Cobol, PL/I, C, C++, Objective-C (before it was used on the Mac :-), Java, REXX, Python, Ruby, SQL, etc. There are many languages, and there is no one "perfect" language. The good languages each have a problem domain (or several) that they are best suited for.

However, there are "poor" languages, languages that make things difficult or obscure or are just poorly designed. At the risk of LOTS of flames, I consider Perl to be one of these, ditto for PHP. This obviously ignores the fact that both are used by MANY MANY people, but that doesn't make them "better" by any means. Acceptance is as much an accident of history as it is a matter of goodness. Python, Perl and indeed PHP have many similar problem domains, and there is no question, that if I am in one of those domains, I prefer Python every time.

Jon
-------------------------------
"The difference between theory and reality is that in theory, there is no difference, but in reality, there is."

The reason I found this is that I was asked to help with something in my company which may require file/text manipulation for a conversion of data from one system to another.

I was very proficient with REXX (if anyone knows that that is) which I loved. Seeing that it's no longer relevant I did a search on interpretive languages for file/text manipulation and eventually found this thread.

Many thanks to the person who started this topic and to EVERYONE for all the replies. As a result, I will go with Perl (I dabbled in it a LONG time ago) and will play with the others for fun.

Here is my preferred solution:
http://paste.lisp.org/display/90009

Also, read pep8, in python we eschew the use of camelCase. Leave that to the Java peeps since they can't get enough widgetWhichProducesBSFactory. I like that with mine it is more clear which field we are sorting and filtering on.

> Even though I only knew it for days, it feels, weirdly comforting and nice (ref. to the warm blanket above).

Yes. That right there I think is one of the keys to Perl's success. Larry and Company have gone through great pains -- years of polishing and fine tuning -- to get Perl to have that well-worn comfortable feel.

My hunch/hope is that they're doing/have-done the same thing with Perl 6, but I haven't learned Perl 6 yet.