Strange characters showing up in FR posts -- here's a workaround

11/11/15

[markomalley]

I don't know if this has impacted anybody else, but in recent weeks, I've noticed that HTML special characters are not rendering properly in HTML posts.

For example, the character “ is rendering as “, the character ” is rendering as ”, and so on.

A screenshot of a typical thread is here:

Needless to say, that is really irritating.

---

There is a workaround for this: replace these special characters with their corresponding HTML codes.

The ones that FR is currently having problems with are as follows:

‘	"left single quote"	‘
’	"right single quote"	’
“	"left double quote"	“
”	"right double quote"	”
—	"em-dash"	—
–	"en-dash"	–
…	"ellipsis"	…

You can do a global search and replace with pretty much any decent text editor out there.

However, in my experience, I've seen that it takes a few passes to catch all of the special characters in a document. You may need to do up to 7 passes in order to catch all of the characters.

This makes for a decent looking post, but it is a lot of repetition.

And I'm lazy.

---

Since I'm lazy, I did a little bit of research in order to figure out a "constructive" way to be lazy in this situation. I was looking for a text editor, similar to Notepad++, that supports macros (since I'm on Linux, I can't use Notepad++, except under Wine).

In doing my research, I think that I've found the solution that does EXACTLY what we need: Geany. Geany is a simple IDE environment that is supported under Linux, Windows, and Mac OSX. There is a plugin written for this editor / IDE that will automatically replace HTML special characters like the ones above with the applicable HTML code, thus eliminating our problems in one fell swoop!

---

In order to use this capability, the first step is to download and install the software (open source). Be sure to install both the software and the plugins.

The user interface:

First, you need to enable the appropriate plugin. Open up the plugin manager window by selecting "Tools" -- "Plugin Manager":

Then select the "HTML Characters" plugin:

---

To use this editor / IDE, you will want to paste in the text you wish to be "fixed".

Select all of the text that you've pasted into the environment ([ctrl]-A).

Then click on "Tools" -- "HTML Replacement" -- "Replace Characters in Selection"

And, Voila, all the problematic special characters in the text have been replaced with the corresponding HTML code.

(Note that if you were using plain text, you will now need to put a paragraph tag "<p>" at the beginning of each paragraph for it to format properly)

[granite]

bfl

[bornred]

If I paste Unicode character 2019 into my reply and hit preview, the server sees the UTF-8 encoding e2 80 99 and (mis)translates each byte into a separate HTML entity: &acirc;&#128;&#153;

If it was working correctly, it would send back something like &#8217;

The POST is being sent as UTF-8, and the result is coming back in UTF-8, so the scripts on the backend are probably just using the wrong charset (iso-8859-1?) to decode the form data.

This may help: http://ahinea.com/en/tech/perl-unicode-struggle.html

[Fundamentally Fair]

I put my response in Notepad, then paste.

[ru4liberty]

My posts look fine until I click "review" to check for typos and/or formatting errors. That's when it tends to go bonkers with all the weird characters.

[KevinB]

Ping

[markomalley]

each user would run this program on each article they read (assuming they wish to do so). Yes or No?

Each article he posts.

[unixfox]

If I paste Unicode character 2019 into my reply and hit preview, the server sees the UTF-8 encoding e2 80 99 and (mis)translates each byte into a separate HTML entity: ’

I think you may be on to something. Maybe the SQL database is somehow using the wrong charset?

Who knows.

[Enlightened1]

Use this for now,

http://dan.hersam.com/tools/smart-quotes.html

[plain talk]

OK. Got it now. Thanks!

[V K Lee]

Thank you, Mark. Rather than taking the time to learn Greek, I’ll just attempt to remember to use the * for the moment. This all began when an infected link was posted a few weeks ago IIRC It is annoying, but nothing more than inconvenient in attempting to decipher. Too old to learn another language and too little time to spend in doing so.

[BlueDragon]

There is strange, and there is stranger.

Stranger In A Strange Land (Leon Russell And The Shelter People) - (1971)

Exodus 2:22