New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sv arenas are never freeded #13610
Comments
From @bulk88Created by @bulk88From an IRC discussion today about why perl doesn't free any mem after ------------------------------------------------------------ I wrote up the following test xsub. ran as "perl -MLocal::XS2 I will be using Process Explorer "private bytes" for ram usage statistic At "Check mem usage now, this is before minimum mem usage", 920 KB. About the Newxz'd array, array was 50 000 000 * 4 = 200 000 000 bytes I have a suspect there is no code path to ever free() SV arena block Perl Info
|
From @bulk88 |
From zefram@fysh.orgbulk88 wrote:
This is a very unusual pattern of memory usage, and it's not worth -zefram |
The RT System itself - Status changed from 'new' to 'open' |
From @bulk88On Wed Feb 19 02:20:46 2014, zefram@fysh.org wrote:
I disagree. If you allocate an AV with atleast (2^16)/16 = 4096 slices/SV*s, then free the AV. The memory won't be released. There are numerous public complaints through out Perl 5's history about "perl never releases memory" to the OS. Loading large data files (>100 KB) like CSVs, JSON, and the like, into perl data structures and transforming or searching them, with the operation taking atleast 1 second of full throttle I/O and/or CPU usage, is a common operation. To address what I think will be the response. Processing each data file/task in a fork is not a solution. It is a bandaid to a design bug in Perl. Perl should be able to goto 1 or 2 GB of memory to process a huge data set, then scale back down to, below, 150% of the memory usage before the task/data set job began. Not stay at 1 GB or 400% even though all variables/data belonging to the task were freed on a PP level. -- |
From @iabynOn Wed, Feb 26, 2014 at 11:35:09PM -0800, bulk88 via RT wrote:
But that is likely to be almost entirely down to the system's malloc() Yes, perl will keep SV head and bodies around for future use, but it -- |
From @bulk88On Thu Feb 27 09:09:47 2014, davem wrote:
No, AFAIK perl's SV arena usage never calls free on any arena blocks until interp exit. So this memory usage has nothing to do with malloc implementation since free() was never called on it. Calling free() is perl's domain, not libc's.
Forever? That is a leak.
This bug isn't about PVX buffers. None were allocated in the sample XS code in this ticket. -- |
From @iabynOn Thu, Feb 27, 2014 at 09:36:46AM -0800, bulk88 via RT wrote:
I'm referring to the fact that in the real world, most code that uses
No, that's efficiency. Its a chosen trade-off. Like PADTMPs;
And I feel that the sample XS code isn't representative of typical large Look, if you can come up with a scheme that can detect completely -- |
From @bulk88On Fri Feb 28 03:30:36 2014, davem wrote:
Should I write a GC for http://perl5.git.perl.org/perl.git/blob/HEAD:/lib/less.pm ? eval "use less 'memory'"; will actually do something. GCing arenas is 1 thing for less 'memory' to do. Another idea I've had was deallocating unused C stack. The perl compiler has horrible C stack recursion during the optimizer phase, and a dozen pages can be shaved off the C stack (see commented out code at https://rt.perl.org/Ticket/Attachment/1235621/643133/perlmain.c ) and returned to the OS after the compiler returns, one example of C stack https://rt.perl.org/Ticket/Attachment/1064560/528860/plstk1.txt which was made changing VM settings so C stack can't grow beyond whatever was allocated at main(). Whole bug https://rt-archive.perl.org/perl5/Ticket/Display.html?id=108276 . If use less 'memory' runs GC code, then its upto the user to decide when to run the GC and the performance implication with it, not P5P's decision. -- |
From @iabynOn Fri, Feb 28, 2014 at 09:12:33PM -0800, bulk88 via RT wrote:
I'm really not convinced of its utility.
Why the 'eval'? Are you proposing that each 'use' this module triggers a If the former, then this seems to be a misuse of the semantics of a pragma.
The best thing to do here is to avoid the recursion in the first place. If
No one's stopping anyone from putting up a CPAN module that does this. -- |
@iabyn - Status changed from 'open' to 'rejected' |
Migrated from rt.perl.org#121274 (status was 'rejected')
Searchable as RT121274$
The text was updated successfully, but these errors were encountered: