The massively popular social network Facebook uses images to display the e-mail addresses of your friends, making it impossible to copy the e-mail address or click on it to send e-mails to your friends, thus making Facebook’s own proprietary in-site messaging system more attractive to its users. Yesterday, Gervase Markham posited that it should be possible to determine the text displayed in the image programmatically by way of the canvas tag and some JavaScript. I’m writing this to confirm that it is indeed possible and has been achieved.
The extension I’ve written to do this is called Facebook Image-to-Email. On any Facebook page containing an e-mail address image, the extension converts the image to text using the following workflow:
- Copy the image to a canvas using drawImage()
- Scans through the canvas to find matches for a pre-determined set of character sprites using getImageData
- Replaces the image with a clickable text e-mail address
Here’s an example of a “before” view:
And “after”:
Of course, this appears to be pretty simple. Ironically, the hardest part of solving this problem was the only part that Gervase remarked would be trivial: matching the image against a set of known character patterns. Since the font is not monospaced, and certain letters bleed into each other when adjacent (such as “89” and “ef”), it wasn’t possible to just store the pixel values for each full letter. What I ended up doing was only matching against only the center of the letters (which are never affected by adjacent letters) and just ignoring character edges.
One other detail as to the implementation: there appears to be some sort of security restriction in Firefox on reading data from images that are not in the same domain as the script reading them. For example, trying to call getImageData() from the chrome on a canvas that contained an image loaded from facebook.com returned null every time; the same happened if the script was running locally but loading a remote image. For this reason, the actual scripting that converts the image to text has to be injected into each page that requires it so that it appears to be running in the same domain as the image.
I’m not claiming that this is the most efficient implementation, but it is definitely complete. In my testing so far, it has correctly identified 100% of the e-mail addresses displayed in the images.
Carrying over some thoughts I had, posting here instead of in Gerv’s blog.
As it is now, you’re comparing the entire string to match a character |uMap.toString() == map|, which is unnecessary. You only need to compare a string long enough to be completely unambiguous in relation to all other characters.
That is, the entries in uMap needn’t be as long as they are to still decode the character properly. That can save quite some time since comparing strings are O(length).
Potentially, one could even make a sort of decoding tree, to feed column for column of image data into, and when it has been unambiguously decoded, you reach the leaf, which is the amount to skip forward, and the character itself.
Great job! Now I’m glad I didn’t spend too long working on it myself!
Nit: “entries in uMap” should of course be “entries in rMaps”.
Another thing. By comparing arrays instead of strings (meaning having every rMaps[i].map as an array) you’d, first of all, compare 25% fewer items, since comparing commas in the string would be avoided. You’d also save on lots of string conversions, since getImageData returns an array in the first place.
I guess somewhere on http://developer.mozilla.org/en/docs/Safely_accessing_content_DOM_from_chrome might be the right hint on how to get the image data.
Thanks for providing it. In my first test it failed with and @pfizer.com address. It returned @pfizecom (no r, no period).
Great for all other cases I’ve tested though.
It will be interesting to see if you can get it to actually create an image from a miniaturised text logo then parse the result into a link, using examples generated here: tinyurl.com/247tyb
If not, expect facebook to switch to this method of obfuscation.
I’ve also noticed the problem with recognizing “r.”, which would seem to be due to the period being placed underneath the tip of the r… causing both to be incorrectly read.
Also “-” isnt being read.
BTW if anyone else is looking to port this to java, BufferedImage.getRaster().getDataBuffer() is what you are looking for to replace the canvas method. =)
Hey when do u get out of clases at end of day i can’t ever find u?