Any programmer working with any language should know about encoding.
Computers stores data in the form of bits i.e 1s and 0s. When you enter any character using keyboard, it saves that character as bytes. When English speaking people developed computers and OS they defined the basic/common encoding ASCII.
ASCII uses only 7 bits to store characters (alphabets, numbers, special chars...etc). 7 raise 2 is 128. So only 128 characters are supported in ASCII encoding system.
When Europeans, Japanese, Chinese and Indians wants to work with their characters it was not supported using English_American system.
Europeans thought of using the other bit in ASCII and adapt some of their characters so that they can store upto 256 characters. But still that was not enough for the world wide users.
Even US English speaking people came across problem when storing special characters like degree (°C), Square, Cube...etc which are not part of ASCII's 128 character list.
People started developing their own code pages at that time and developed abundant amount of code pages. Even US English speaking people came across
code page called "windows-1252" which is Latin based and extended 128 ASCII into 256. But, this created lots of confusion to encode/decode appropriately and to share information.
Smart people around the world planned to fix this issue by introducing concept called
Unicode. It supports lakhs of characters (
code points) so that all characters used in the world are handled.
Unicode comes with variations like UTF8, the most widely used file encoding system and UTF16 & UTF32. UTF8 means it start using 1 byte for character and if that is not sufficient it goes upto multiple bytes. Similarly UTF16 and 32.
Here are the FAQs
1) What is encoding?
It is the process of converting text into bytes
2) Why encoding needed?
Because the default ASCII encoding couldn't support all the characters used in the world except English countries.
3) As a developer where I need to be careful about encoding?
A) When ever you get chance to read files provided by others, have conscious effort on what encoding is used in the file. So that you could use same to decode in your language API
B) Know what is the default encoding your programming language supports. So that you can override that when required.
For Ex:
.NET supports UTF-16 for chars and strings
StreamReader constructor uses UTF-8 as default encoding if user doesn't specify the same
https://msdn.microsoft.com/en-us/library/yhfzs7at(v=vs.110).aspx#Anchor_2
https://msdn.microsoft.com/en-us/library/ms143456(v=vs.110).aspx#Anchor_2
4) Is encoding and encryption same?
No, Encoding is just way of formatting the data and setting rule how to read the data. But, encryption just alter the data and re-alter it at receiving end.
5) What is code page?
Every language speaking people defined the extension of ASCII characters as per their language from 128 to 255 is called their code page. For ex: windows-1252
6) Do you want to know more details on this encoding?
Reference 1:
http://kunststube.net/encoding/
Reference 2:
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
Happy Coding!