From @jandubois

I changed the default read mode for scripts on Windows from binary
to text mode to avoid various bugs with the DATA filehandle​:

  http​://perl5.git.perl.org/perl.git/commitdiff/270ca148c

It turns out that the tokenizer does no longer recognize
byte order marks when the script contains CRs, but is read
in text mode​:

  Unrecognized character \xEF; marked by <-- HERE after <-- HERE near column 1 at bom.pl line 1.

I would consider this a regression for "Perl compiled with default
options on Windows". The attached patch fixes the problem for me
(but review welcome!).

Jesse, should this be a 5.14 blocker?

Cheers,
-Jan

PS​: Will provide a corresponding test tomorrow...

--- a/toke.c
+++ b/toke.c
@​@​ -4792,7 +4792,13 @​@​ Perl_yylex(pTHX)
  *(U8*)s == 0xEF ||
  *(U8*)s >= 0xFE ||
  s[1] == 0)) {
- bof = PerlIO_tell(PL_rsfp) == (Off_t)SvCUR(PL_linestr);
+ IV offset = (IV)PerlIO_tell(PL_rsfp);
+ bof = (offset == SvCUR(PL_linestr));
+#if defined(PERLIO_USING_CRLF) && defined(PERL_TEXTMODE_SCRIPTS)
+ /* offset may include swallowed CR */
+ if (!bof)
+ bof = (offset == SvCUR(PL_linestr)+1);
+#endif
  if (bof) {
  PL_bufend = SvPVX(PL_linestr) + SvCUR(PL_linestr);
  s = swallow_bom((U8*)s);

From @jandubois

http​://perl5.git.perl.org/perl.git/commitdiff/6d51015587940c2032a6533d886163f69ca028f9

@jandubois - Status changed from 'new' to 'resolved'