Skip Menu |
Queue is disabled
This queue is disabled and you may not create new tickets in it. Disabled queues are usually because the distribution was merged with another or changed names. Sometimes they are the end result of a bad autocreate from PAUSE data before anyone noticed.
Report information
Id: 19333
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: eperez <eperez [at] it.uc3m.es>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: (no value)



Date: Sat, 21 Dec 2002 18:31:33 +0000
From: Eduardo P
Download (untitled) / with headers
text/plain 699b
The operation: perl -pi -e 's/foo/bar/' file should be atomic. If something while perl is working hangs the system like a crash or a power failure all the data is lost. This is what perl currently does (simplified Linux strace): open("file", O_RDONLY|O_LARGEFILE) = original unlink("file") open("file", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = new read(original,... write(new,... close(new) close(original) Thus is the atomic approach: open("file", O_RDONLY|O_LARGEFILE) = original open("file.new", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = new read(original,... write(new,... close(new) close(original) rename("file.new", "file") This way you always keep a good copy if something fails.
Date: Sat, 21 Dec 2002 18:43:20 +0000
From: Eduardo P
Download (untitled) / with headers
text/plain 284b
This is the patch I'm working. It's segfaulting. As I'm not a perl core coder I don't know where's the problem. Be aware it removes some features, so some tests don't pass. But this is what a working patch should look like. Could some perl core coder tell me how to fix this code?

Message body is not shown because sender requested not to inline it.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 961b
On Sat Dec 21 10:32:12 2002, eperez wrote: Show quoted text
> The operation: > perl -pi -e 's/foo/bar/' file > should be atomic. > > If something while perl is working hangs the system like a crash or a > power failure all the data is lost. >
As of Perl 5.14.2, 'pods/perlrun.pod' says this about the '-i' option ### -i[*extension*] specifies that files processed by the "<>" construct are to be edited in-place. It does this by renaming the input file, opening the output file by the original name, and selecting that output file as the default for print() statements. The extension, if supplied, is used to modify the name of the old file to make a backup copy, following these rules: If no extension is supplied, no backup is made and the current file is overwritten. ### This seems to me to guarantee the atomicity the original poster was seeking. Am I correct in this conclusion? If so, then the ticket may be closed. Thank you very much. Jim Keenan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #19333] edit <> files in place is not atomic
Date: Sun, 27 Nov 2011 02:24:43 -0500
To: perlbug-followup [...] perl.org
From: Peter Martini <petercmartini [...] gmail.com>
Download (untitled) / with headers
text/plain 2.3k
On Thu, Nov 24, 2011 at 2:30 PM, James E Keenan via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Sat Dec 21 10:32:12 2002, eperez wrote:
> The operation:
> perl -pi -e 's/foo/bar/' file
> should be atomic.
>
> If something while perl is working hangs the system like a crash or a
> power failure all the data is lost.
>

As of Perl 5.14.2, 'pods/perlrun.pod' says this about the '-i' option
###
-i[*extension*]
 specifies that files processed by the "<>" construct are to be
 edited in-place. It does this by renaming the input file, opening
 the output file by the original name, and selecting that output
 file as the default for print() statements. The extension, if
 supplied, is used to modify the name of the old file to make a
 backup copy, following these rules:

 If no extension is supplied, no backup is made and the current file
 is overwritten.
###

This seems to me to guarantee the atomicity the original poster was seeking.

Am I correct in this conclusion?  If so, then the ticket may be closed.

Thank you very much.
Jim Keenan
 
The issue is what happens if the program crashes in the middle of the operation.  In most OSes* If no extension is supplied and perl dies unexpectedly, you're out of luck - the original file has been replaced by whatever perl had managed to write out so far.  The request in this ticket is to change that to silently using a default extension.  I'd propose a different solution: update perlrun.pod to point out the following two caveats if no extension is supplied:
 
1. If the script dies any time after opening the file, the file will now only have what had been processed up to that point.
2. Even if no extension is supplied and the file appears to be written in place, the filesystem doesn't count the space used by the original file as free until perl exits (or more technically, until the filehandle closes).  I vaguely recall this being brought up as a separate bug on this list, since it can lead to data loss if the filesystem is close to full.
 
I'm not sure how much detail to go into in perlrun, otherwise I'd have a patch attached!
 
* See http://perldoc.perl.org/perlcygwin.html; Windows does what the requester wants, with the .bak extension, because it does not support doing it any other way.  I don't see a note about that in perlwin32.pod, but I believe it would hold there as well?
Download (untitled) / with headers
text/plain 469b
On Sat Nov 26 23:25:16 2011, pcm wrote: Show quoted text
> ... > different solution: update perlrun.pod to point out the following two > caveats if no extension is supplied: > ...
We can add this, but they're not really "perl facts" so much as "how file I/O generally works." I'm not excited about the idea of adding this kind of warning everywhere it might be relevant. Is this case really special, or is this just one user who didn't think through the ramifications of his program?
RT-Send-CC: perl5-porters [...] perl.org
On Sun Nov 27 03:49:39 2011, rjbs wrote: Show quoted text
> On Sat Nov 26 23:25:16 2011, pcm wrote:
> > ... > > different solution: update perlrun.pod to point out the following two > > caveats if no extension is supplied: > > ...
> > We can add this, but they're not really "perl facts" so much as "how > file I/O generally works." I'm not excited about the idea of adding > this kind of warning everywhere it might be relevant. Is this case > really special, or is this just one user who didn't think through the > ramifications of his program?
I think adding the warning to perlrun.pod at least would be good so it's clear that if you don't want to lose your data, you should make backups. I realize that this really goes without saying, because if you mess up the search/replace pattern you can really screw up your data, but sometimes it's good to re-iterate or make obvious things we may consider common knowledge. I don't think we should make backing up the data default behavior though, but I suppose my only reasoning for that is because "This is how it's always been." -- Matthew Horsfall (alh)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.3k
On Sun Dec 11 15:23:08 2011, alh wrote: Show quoted text
> On Sun Nov 27 03:49:39 2011, rjbs wrote:
> > On Sat Nov 26 23:25:16 2011, pcm wrote:
> > > ... > > > different solution: update perlrun.pod to point out the following two > > > caveats if no extension is supplied: > > > ...
> > > > We can add this, but they're not really "perl facts" so much as "how > > file I/O generally works." I'm not excited about the idea of adding > > this kind of warning everywhere it might be relevant. Is this case > > really special, or is this just one user who didn't think through the > > ramifications of his program?
> > I think adding the warning to perlrun.pod at least would be good so it's > clear that if you don't want to lose your data, you should make backups. > > I realize that this really goes without saying, because if you mess up > the search/replace pattern you can really screw up your data, but > sometimes it's good to re-iterate or make obvious things we may consider > common knowledge. > > I don't think we should make backing up the data default behavior > though, but I suppose my only reasoning for that is because "This is how > it's always been." > > -- Matthew Horsfall (alh)
rjbs, alh: Can you come to some resolution of these issues so that we can patch if needed and then close this RT? Thank you very much. Jim Keenan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #19333] edit <> files in place is not atomic
Date: Thu, 27 Sep 2012 21:59:49 -0400
To: James E Keenan via RT <perlbug-followup [...] perl.org>
From: Ricardo Signes <perl.p5p [...] rjbs.manxome.org>
Download (untitled) / with headers
text/plain 232b
* James E Keenan via RT <perlbug-followup@perl.org> [2012-09-21T22:48:07] Show quoted text
> Can you come to some resolution of these issues so that we can patch if > needed and then close this RT? >
I relent. A warning is appropriate. -- rjbs
Download signature.asc
application/pgp-signature 490b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 386b
On Thu Sep 27 19:00:19 2012, perl.p5p@rjbs.manxome.org wrote: Show quoted text
> * James E Keenan via RT <perlbug-followup@perl.org> [2012-09-21T22:48:07]
> > Can you come to some resolution of these issues so that we can patch if > > needed and then close this RT? > >
> > I relent. A warning is appropriate. >
alh, pcm: Can either of you submit a *brief* patch? Thank you very much. Jim Keenan
CC: perl5-porters [...] perl.org
Subject: Re: [perl #19333] edit <> files in place is not atomic
Date: Mon, 8 Oct 2012 22:47:20 -0400
To: perlbug-followup [...] perl.org
From: Peter Martini <petercmartini [...] gmail.com>
Download (untitled) / with headers
text/plain 1.9k
On Thu, Sep 27, 2012 at 10:27 PM, James E Keenan via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Thu Sep 27 19:00:19 2012, perl.p5p@rjbs.manxome.org wrote:
> * James E Keenan via RT <perlbug-followup@perl.org> [2012-09-21T22:48:07]
> > Can you come to some resolution of these issues so that we can patch if
> > needed and then close this RT?
> >
>
> I relent.  A warning is appropriate.
>


alh, pcm:  Can either of you submit a *brief* patch?

Something like this?

From a09ace7d638055c4a5ecf3cce1d41de3347bfcda Mon Sep 17 00:00:00 2001
From: Peter Martini <PeterCMartini@GMail.com>
Date: Mon, 8 Oct 2012 22:31:37 -0400
Subject: [PATCH] Clarify that in-place editing actually creates a new file.
 If the in-place editing dies, the original is gone.
 Another implication of this is that hard links on UNIX
 won't work properly, since a new inode will be generated -
 I think that's a little too specific to spell out in the docs
 though.

---
 pod/perlrun.pod |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/pod/perlrun.pod b/pod/perlrun.pod
index 9ed678c..1f32cd5 100644
--- a/pod/perlrun.pod
+++ b/pod/perlrun.pod
@@ -506,8 +506,10 @@ default for print() statements.  The extension, if supplied, is used to
 modify the name of the old file to make a backup copy, following these
 rules:

-If no extension is supplied, no backup is made and the current file is
-overwritten.
+If no extension is supplied, and your system supports it, the original
+I<file> is kept open without a name while the output is redirected to
+a new file with the original I<filename>.  When perl exits, cleanly or not,
+the original I<file> is unlinked.

 If the extension doesn't contain a C<*>, then it is appended to the
 end of the current filename as a suffix.  If the extension does
--
1.7.1
 
Show quoted text

Thank you very much.
Jim Keenan

---
via perlbug:  queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=19333

RT-Send-CC: perl5-porters [...] perl.org, petercmartini [...] gmail.com
On Mon Oct 08 19:47:55 2012, pcm wrote: Show quoted text
> On Thu, Sep 27, 2012 at 10:27 PM, James E Keenan via RT < > perlbug-followup@perl.org> wrote: >
> > On Thu Sep 27 19:00:19 2012, perl.p5p@rjbs.manxome.org wrote:
> > > * James E Keenan via RT <perlbug-followup@perl.org>
> > [2012-09-21T22:48:07]
> > > > Can you come to some resolution of these issues so that we can
> patch if
> > > > needed and then close this RT? > > > >
> > > > > > I relent. A warning is appropriate. > > >
> > > > > > alh, pcm: Can either of you submit a *brief* patch? > >
> > Something like this? > > From a09ace7d638055c4a5ecf3cce1d41de3347bfcda Mon Sep 17 00:00:00 2001 > From: Peter Martini <PeterCMartini@GMail.com> > Date: Mon, 8 Oct 2012 22:31:37 -0400 > Subject: [PATCH] Clarify that in-place editing actually creates a new > file. > If the in-place editing dies, the original is gone. > Another implication of this is that hard links on UNIX > won't work properly, since a new inode will be generated - > I think that's a little too specific to spell out in the docs > though. > > --- > pod/perlrun.pod | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) > > diff --git a/pod/perlrun.pod b/pod/perlrun.pod > index 9ed678c..1f32cd5 100644 > --- a/pod/perlrun.pod > +++ b/pod/perlrun.pod > @@ -506,8 +506,10 @@ default for print() statements. The extension, > if > supplied, is used to > modify the name of the old file to make a backup copy, following > these > rules: > > -If no extension is supplied, no backup is made and the current file > is > -overwritten. > +If no extension is supplied, and your system supports it, the > original > +I<file> is kept open without a name while the output is redirected to > +a new file with the original I<filename>. When perl exits, cleanly > or not, > +the original I<file> is unlinked. > > If the extension doesn't contain a C<*>, then it is appended to the > end of the current filename as a suffix. If the extension does > -- > 1.7.1
Good enough for me. Thank you. Applied as 479e5f8787. -- Father Chrysostomos


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org