Re: [Templates] Patch: Full unicode support for TT under 5.8

[prev] [thread] [next] [Date index for 2004/06/28]

On Sun, 27 Jun 2004, Mark Fowler wrote:

> Attached is a patch (and a test which tests the patch) that allows
> the Template Toolkit to work properly with Unicode in perl 5.8.

Or so I thought.

First up, the '6.007' in the test should obviously read  '5.007' if you
want the test to run. D'oh.

Secondly, Tom Insam found a bug when using non-ascii keys to the stash
with the XS stash - it's looking up the key whose charecters are made up
of the utf8 encoding of the variable name rather than the key whose
charecters match the actual string that's meant to be represented (if you
see what I mean).  Here, let me demonstrate with a failing test case:

#!/usr/bin/perl

use strict;
use warnings;

use Test::More tests => 2;

use File::Temp qw(tempfile);
my ($fh, $filename) = tempfile();
print $fh "\x{ef}\x{bb}\x{bf}var = [% m\x{c3}\x{b8}\x{c3}\x{b8}se %]";
close $fh;

use Template;
use Template::Stash;
use Template::Stash::XS;

my $tt;
my $output;

#############################################################

$tt = Template->new( ABSOLUTE => 1,
                     STASH => Template::Stash->new());

$output = '';
$tt->process( $filename, {
  "m\x{f8}\x{f8}se" => "right",
  "m\x{c3}\x{b8}\x{c3}\x{b8}se" => "wrong",
}, \$output)
  or $output = $tt->error;

is($output, "var = right","perl");

###################################

$tt = Template->new( ABSOLUTE => 1,
                     STASH => Template::Stash::XS->new());

$output = '';
$tt->process( $filename, {
  "m\x{f8}\x{f8}se" => "right",
  "m\x{c3}\x{b8}\x{c3}\x{b8}se" => "wrong",
}, \$output)
  or $output = $tt->error;

is($output, "var = right","xs");

#############################################################

unlink($filename);


This produces:

1..2
ok 1 - perl
not ok 2 - xs
#     Failed test (t/utf8key.t at line 46)
#          got: 'var = wrong'
#     expected: 'var = right'
# Looks like you failed 1 tests of 2.

Anyone got any ideas?  I'm loathed to crack open the XS stash and change
it as quite frankly transcoding between the different internal formats
Perl uses for string in XS sounds like it could be a lot of pain, and
could slow things down.  We could consider it another limitation of the XS
stash - that you simply can't use non-ascii stash keys if you want the
speed gain that the XS stash gives.

Mark.

        -- 
        #!/usr/bin/perl -T
use strict;
use warnings;
print q{Mark Fowler, mark@xxxxxxxxxxxxxx.xxx, http://twoshortplanks.com/};

_______________________________________________
templates mailing list
templates@xxxxxxxxxxxxxxxx.xxx
http://lists.template-toolkit.org/mailman/listinfo/templates