From rwilkinson@star.net.uk

This is a bug report for perl from rwilkinson@​star.net.uk,
generated with the help of perlbug 1.34 running under perl v5.8.0.

Hello

  I am trying to retrieve a directory listing from an FTP server using LWP.
I think that the site may well be sending me non-ascii characters? However,
Net​::FTP can handle it, and from reading LWP​::Protocol​::ftp.pm, there is an
implication that Net​::FTP is used by LWP?

  LWP does act as I expect if I use it to get a directory listing of the
parent directory of the one I need to list. I could bypass the problem by
using Net​::FTP for this one ftp server, however, since the rest of my code
uses LWP, I'd sooner get to the bottom of the problem.

  I have listed the programs here, followed by their output collected via
script. I have tried to reduce the programs to their essential statements. I
have changed the ftp address, username and password because I do not know if I
have permission to distribute them. However, the outputs are those garnered
before this change.

  The code is ordered​:

  netftp.pl
  netftp.op
  lwpbad.pl
  lwpbad.op
  lwpgood.pl
  lwpgood.op

I suspect that the fact that 'content-type' => 'application/octet-stream' is
set in the HTTP​::Header of the lwpbad output is wrong. It is set to
'content-type' => 'text/ftp-dir-listing' in the lwpgood output. I think that
this is probably due to the fact that LWP​::MediaTypes is interpreting the
non-ascii character?

Is it possible for me to override the assignation given. I do set the
content_type of my request object to be "text/ftp-dir-listing".

Bob

#!/usr/bin/perl -w

use strict;
use Net​::FTP;
use Data​::Dumper;

my $ftp = Net​::FTP->new("partner.abcde.ru", Debug => 0)
  or die "Cannot connect to partner.abcde.ru​: $@​";

$ftp->login("xyz",'123')
  or die "Cannot login ", $ftp->message;

$ftp->cwd("/bases")
  or die "Cannot change working directory ", $ftp->message;

my $dir = $ftp->dir
  or die "dir failed ", $ftp->message;
warn "DIR is ",Dumper($dir);

$ftp->quit;

Script started on Wed Jun 2 12​:25​:13 2004
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$ perl netftp.pl
DIR is $VAR1 = [
  'total 3870',
  '-rw-r--r-- 1 2133 4000 15297 May 20 12​:39 advware.avc',
  '-rw-r--r-- 1 2133 4000 962750 May 20 12​:39 avp0404.avc',
  '-rw-r--r-- 1 2133 4000 286326 May 20 12​:39 backdoor.avc',
  '-rw-r--r-- 1 2133 4000 53719 May 20 12​:39 ca.avc',
  '-rw-r--r-- 1 2133 4000 32075 Jun 2 14​:54 daily.avc',
  '-rw-r--r-- 1 2133 4000 1843 May 20 12​:39 eicar.avc',
  '-rw-r--r-- 1 2133 4000 19046 May 20 12​:39 extr-cab.avc',
  '-rw-r--r-- 1 2133 4000 211150 May 20 12​:39 extract.avc',
  '-rw-r--r-- 1 2133 4000 8401 May 20 12​:39 kernel.avc',
  '-rw-r--r-- 1 2133 4000 5275 May 20 12​:39 krndos.avc',
  '-rw-r--r-- 1 2133 4000 23515 May 20 12​:39 krnengn.avc',
  '-rw-r--r-- 1 2133 4000 64482 May 20 12​:39 krnexe.avc',
  '-rw-r--r-- 1 2133 4000 37999 May 20 12​:39 krnjava.avc',
  '-rw-r--r-- 1 2133 4000 76929 May 20 12​:39 krnmacro.avc',
  '-rw-r--r-- 1 2133 4000 55548 May 20 12​:39 krnunp.avc',
  '-rw-r--r-- 1 2133 4000 141049 May 20 12​:39 macro.avc',
  '-rw-r--r-- 1 2133 4000 14114 May 20 12​:39 mail.avc',
  '-rw-r--r-- 1 2133 4000 115922 May 20 12​:39 malware.avc',
  '-rw-r--r-- 1 2133 4000 375390 May 20 12​:39 newexe.avc',
  '-rw-r--r-- 1 2133 4000 14920 May 20 12​:39 newexeg.avc',
  '-rw-r--r-- 1 2133 4000 3162 May 20 12​:39 ocr.avc',
  '-rw-r--r-- 1 2133 4000 27541 May 20 12​:39 pornware.avc',
  '-rw-r--r-- 1 2133 4000 28492 May 20 12​:39 riskware.avc',
  '-rw-r--r-- 1 2133 4000 99749 May 20 12​:39 script.avc',
  '-rw-r--r-- 1 2133 4000 5366 May 20 12​:39 smart.avc',
  '-rw-r--r-- 1 2133 4000 356774 May 20 12​:39 trojan.avc',
  '-rw-r--r-- 1 2133 4000 569565 May 20 12​:39 unpack.avc',
  '-rw-r--r-- 1 2133 4000 15450 May 20 12​:39 up040402.avc',
  '-rw-r--r-- 1 2133 4000 18563 May 20 12​:39 up040409.avc',
  '-rw-r--r-- 1 2133 4000 23479 May 20 12​:39 up040416.avc',
  '-rw-r--r-- 1 2133 4000 40902 May 20 12​:39 up040423.avc',
  '-rw-r--r-- 1 2133 4000 27378 May 20 12​:39 up040430.avc',
  '-rw-r--r-- 1 2133 4000 19431 May 20 12​:39 up040507.avc',
  '-rw-r--r-- 1 2133 4000 35446 May 20 12​:39 up040514.avc',
  '-rw-r--r-- 1 2133 4000 23532 May 20 12​:39 x-files.avc',
  '-rw-r--r-- 1 2133 4000 2040 Jun 2 14​:54 xdaily.avc'
  ];
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$
Script done on Wed Jun 2 12​:25​:22 2004

#!/usr/bin/perl -w

use strict;

use LWP​::UserAgent;

my $url = "ftp​://partner.abcde.ru/bases";
my $user = "xyz";
my $pass = "123";
my $content_type = "text/ftp-dir-listing";

my $ua = LWP​::UserAgent->new;
my $req = HTTP​::Request->new('GET', $url);
$req->authorization_basic($user, $pass);
$req->content_type($content_type);
my $res = $ua->request($req);
use Data​::Dumper; warn "RES is ",Dumper($res);

Script started on Wed Jun 2 12​:34​:13 2004
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$ perl lwpbad.pl
RES is $VAR1 = bless( {
  '_content' => '��,����.����*� ���..����,�����servers.lst���,�����advware.avc���,�����avp0404.avc���,����backdoor.avc�³+À��,�����ca.avc�Ç��,���� daily.avc��ä��,���� eicar.avc��ä��,����extr-cab.avc�³+À �,�����extract.avc�!�,����
kernel.avc�ä.�,����
krndos.avc�ä/�,�����krnengn.avc�0�,����
krnexe.avc�ä1�,�����krnjava.avc�2�,����krnmacro.avc�³+À3�,����
krnunp.avc�ä4�,���� macro.avc��ä5�,�����mail.avc���ä6�,�����malware.avc�7�,����
newexe.avc�ä8�,�����newexeg.avc�9�,�����ocr.avc�​:�,����pornware.avc�³+À<�,����
script.avc�ä;�,����riskware.avc�³+À=�,���� smart.avc��ä>�,����
trojan.avc��,����
unpack.avc�ä@​�,����up040402.avc�³+ÀA�,����up040409.avc�³+ÀB�,����up040416.avc�³+ÀC�,����up040423.avc�³+ÀD�,����up040430.avc�³+ÀE�,����up040507.avc�³+ÀF�,����up040514.avc�³+ÀG�,�����x-files.avc�H�,�ð��
xdaily.avc����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������',
  '_rc' => 200,
  '_headers' => bless( {
  'client-date' => 'Wed, 02 Jun 2004 11​:34​:20 GMT',
  'content-type' => 'application/octet-stream',
  'server' => ' ###############################################################',
  'client-request-num' => 1
  }, 'HTTP​::Headers' ),
  '_msg' => 'OK',
  '_request' => bless( {
  '_content' => '',
  '_uri' => bless( do{\(my $o = 'ftp​://partner.abcde.ru/bases')}, 'URI​::ftp' ),
  '_headers' => bless( {
  'user-agent' => 'libwww-perl/5.65',
  'content-type' => 'text/ftp-dir-listing',
  'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ=='
  }, 'HTTP​::Headers' ),
  '_method' => 'GET'
  }, 'HTTP​::Request' )
  }, 'HTTP​::Response' );
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$
Script done on Wed Jun 2 12​:34​:29 2004

#!/usr/bin/perl -w

use strict;

use LWP​::UserAgent;

my $url = "ftp​://partner.abcde.ru/";
my $user = "xyz";
my $pass = "123";
my $content_type = "text/ftp-dir-listing";

my $ua = LWP​::UserAgent->new;
my $req = HTTP​::Request->new('GET', $url);
$req->authorization_basic($user, $pass);
$req->content_type($content_type);
my $res = $ua->request($req);
use Data​::Dumper; warn "RES is ",Dumper($res);

Script started on Wed Jun 2 12​:33​:32 2004
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$ perl lwpgood.pl
RES is $VAR1 = bless( {
  '_content' => 'total 2
drwxr-xr-x 2 2133 4000 1024 May 20 12​:39 bases
',
  '_rc' => 200,
  '_headers' => bless( {
  'client-date' => 'Wed, 02 Jun 2004 11​:33​:40 GMT',
  'content-type' => 'text/ftp-dir-listing',
  'content-length' => 58,
  'server' => ' ###############################################################',
  'client-request-num' => 1
  }, 'HTTP​::Headers' ),
  '_msg' => 'OK',
  '_request' => bless( {
  '_content' => '',
  '_uri' => bless( do{\(my $o = 'ftp​://partner.abcde.ru/')}, 'URI​::ftp' ),
  '_headers' => bless( {
  'user-agent' => 'libwww-perl/5.65',
  'content-type' => 'text/ftp-dir-listing',
  'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ=='
  }, 'HTTP​::Headers' ),
  '_method' => 'GET'
  }, 'HTTP​::Request' )
  }, 'HTTP​::Response' );
�]0;bob@​snoogen​:~/work/accurev/VirusSignatures�[bob@​snoogen VirusSignatures]$
Script done on Wed Jun 2 12​:33​:45 2004

Flags​:
  category=library
  severity=medium

Site configuration information for perl v5.8.0​:

Configured by bhcompile at Sun Sep 1 23​:55​:07 EDT 2002.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration​:
  Platform​:
  osname=linux, osvers=2.4.18-11smp, archname=i386-linux-thread-multi
  uname='linux daffy.perf.redhat.com 2.4.18-11smp #1 smp thu aug 15 06​:41​:59 edt 2002 i686 i686 i386 gnulinux '
  config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -Dmyhostname=localhost -Dperladmin=root@​localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
  optimize='-O2 -march=i386 -mcpu=i686',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/include/gdbm'
  ccversion='', gccversion='3.2 20020822 (Red Hat Linux Rawhide 3.2-5)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
  perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
  libc=/lib/libc-2.2.92.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.2.92'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches​:
 

@​INC for perl v5.8.0​:
  /home/bob/lib
  /usr/lib/perl5/5.8.0/i386-linux-thread-multi
  /usr/lib/perl5/5.8.0
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
  /usr/lib/perl5/site_perl/5.8.0
  /usr/lib/perl5/site_perl
  /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
  /usr/lib/perl5/vendor_perl/5.8.0
  /usr/lib/perl5/vendor_perl
  .

Environment for perl v5.8.0​:
  HOME=/home/bob
  LANG=en_GB
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/bob/bin​:/usr/bin​:/bin​:/usr/local/bin​:/usr/X11R6/bin​:/opt/accurev/bin
  PERL5LIB=/home/bob/lib
  PERL_BADLANG (unset)
  SHELL=/bin/bash

From @rgs

Bob Wilkinson (via RT) wrote​:

This is a bug report for perl from rwilkinson@​star.net.uk,
generated with the help of perlbug 1.34 running under perl v5.8.0.

Hello

 I am trying to retrieve a directory listing from an FTP server using LWP\.

Thanks for your report; however, the perlbug address is only used to
report bugs in perl itself, or in the modules that are distributed along
perl. For CPAN modules, you should use the CPAN bug interface at
<http​://rt.cpan.org/>, or write directly to the author.

The RT System itself - Status changed from 'new' to 'open'

@rgs - Status changed from 'open' to 'rejected'