What's wrong with "smart quotes" and possibly how to fix it (Vanity)

me

[some tech guy]

So, I'm still seeing smart quotes mess up. There was a thread a little while ago whereby I said what the problem was, but it's still extant.

Here's the problem: All the web-tier stuff is using UTF-8 (which is a great idea), but *something*, probably the database tier, is using CP-1252.

I suspect either:

a) Database table for posts on SQL Server is set to collate and/or store the windows-1252/CP-1252 character set.

or

b) something between the web level and the DB level is doing character set interpretation in an incorrect way. There's 1252 in there somewhere, and that's what's breaking the quotes.

[some tech guy]

Just trying to be helpful.

[some tech guy]

http://stackoverflow.com/questions/2477452/%C3%A2%E2%82%AC-showing-on-page-instead-of

[some tech guy]

I’ll go further: the issue doesn’t occur on the front page, only when you view the thread. That’s where the problem is.

[sparklite2]

I haven’t used this yet, but it looks promising.

http://dan.hersam.com/tools/smart-quotes.html

[Slyfox]

I cant deny, I have sort of gotten used to not using apostrophes.

[Paladin2]

English has always been about merging languages and words. Just eschew punctuation and help move it along, like texting...

[Paladin2]

How do [] {} () || work out?

[Olog-hai]

Don't have to live without 'em. Code for straight apostrophes is ' for the record. They're waitin' for all of us.

[kevao]

Don’t know what you’re talking about.

[Stentor]

’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t’t€™â€™Ã¢â‚¬â„¢t’t’t

It's getting on my nerves.

[tenger]

Anyone check the radiator to see if it was full? I’m surprised people haven’t thought of that already...

[Mariner]

Pinging Jim.

1252 in the middleware.

[kevao]

It's getting on my nerves.

You got an apostrophe through clean. That’s good for you!

[Stentor]

You got an apostrophe through clean.

I lost it. I hate myself. Oh well, there's butter pecan in the freezer.

[cynwoody]

I’ll go further: the issue doesn’t occur on the front page, only when you view the thread. That’s where the problem is.

The problem is only with posted stuff. The server thinks it knows better than the user.

The most trivial example is posting non-HTML containing quotes. The server converts them from straight-up 7-bit ASCII to left and right curlies (smart quotes). They display just fine.

Then somebody selects them, posts them as a quote, adds commentary, and submits. Now they are garbage! LOL!

The root of the problem is, the server is looking at the UTF-8 input and supplying HTML entities for each non-7-bit ASCII byte it sees.

E.g., for a left curly double quote (“), the UTF-8 hex is e2 80 9c. The server translates those three bytes to “, which come out as “, namely, small-a with a circumflex, the euro symbol, and the oe ligature.

The solution is to convert that mess back to the original Unicode “.

This can easily be done client-side, using JavaScript. You just need some user scripts. See here for thread viewing and here for posting.

[MarchonDC09122009]

The apostophe quote weird character makes us look incompetent, and defaces the great factual content here.
Sys Admin, please escalate and fix FR - America’s black box recorder of why she fell and what went wrong.

[some tech guy]

Interesting. I really think it’s 1252 somewhere, but your commentary also makes some sense. It could be some interaction between the markup translation, detranslation to raw characters, and retranslation to viewable.

The bit I find particularly interesting is that the smart quotes work on the front page, but break on the thread view.

Here’s my dump from copying smart quotes to a file (in a UTF-8 console) and hexdumping it:

[redacted]-imac:~ [redacted]$ hexdump ~/freep.txt
0000000 c3 a2 e2 82 ac 0a
0000006

That looks very much like the issue referenced on stackoverflow, which is why I suspect 1252 in there somewhere.

[kevao]

“smart quotes” is a term of microaggression. Whether a quote is smart or not, should depend on its source, not on its encoding. Just saying....

[some tech guy]

What if the quotes are only pretending to be smart, but are actually vanilla? We should give them a safe space.

[kevao]

If the smart quotes are in black font, then yes. #blackfontsmatter