字符编码-java内部编码探讨

一、Stack Overflow 上的提问

What is the Java’s internal represention for String? Modified UTF-8? UTF-16?

Java uses UTF-16 for the internal text representation

The representation for String and StringBuilder etc in Java is UTF-16

https://docs.oracle.com/javase/8/docs/technotes/guides/intl/overview.html

How is text represented in the Java platform?

The Java programming language is based on the Unicode character set, and several libraries implement the Unicode standard. The primitive data type char in the Java programming language is an unsigned 16-bit integer that can represent a Unicode code point in the range U+0000 to U+FFFF, or the code units of UTF-16. The various types and classes in the Java platform that represent character sequences – char[], implementations of java.lang.CharSequence (such as the String class), and implementations of java.text.CharacterIterator – are UTF-16 sequences.

At the JVM level, if you are using -XX:+UseCompressedStrings (which is default for some updates of Java 6) The actual in-memory representation can be 8-bit, ISO-8859-1 but only for strings which do not need UTF-16 encoding.

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

and supports a non-standard modification of UTF-8 for string serialization.

Serialized Strings use UTF-8 by default.

And how many bytes does Java use for a char in memory?

A char is always two bytes, if you ignore the need for padding in an Object.

Note: a code point (which allows character > 65535) can use one or two characters, i.e. 2 or 4 bytes.

二、java内部编码是utf-16

代码一：

char c = '放';
System.out.printf("The value of char %c is %d.%n", c, (int) c);

String str = String.valueOf(c);
byte[] bys;
try {
	bys = str.getBytes("Unicode");
	for (int i = 0; i < bys.length; i++) {
		System.out.printf("%X ", bys[i]);
	}
	System.out.println();

	int unicode = (bys[2] & 0xFF) << 8 | (bys[3 & 0xFF]);
	System.out.printf("The unicode value of %c is %d.%n", c, unicode);

} catch (UnsupportedEncodingException e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
}

输出结果如下：

		
The value of char 放 is 25918.
FE FF 65 3E 
The unicode value of 放 is 25918.

代码二：

		
                 String a = new String("放");
		 System.out.println(strTo16(a));

/******************** another method ******************************/

public static String strTo16(String s) {
		String str = "";

		System.out.println(s.length());
		for (int i = 0; i < s.length(); i++) {
			int ch = (int) s.charAt(i);
			String s4 = Integer.toHexString(ch);
			str = str + s4;
		}
		return str;
	}

输出结果如下：

		
1
653e

UTF-16 首先用两个字节，表示Unicode编码，两个字节的情况下，编码和Unicode一模一样。2个字节能编码的字一共有2^16 = 65536个字。这种情况下，一个字符占两个字节。那么 utf-16 如何表示 大于 Unicode编码 65536的字呢。这个时候，就是采用4个字节编码了。

维基百科 Unicode已经超过13万编码了   https://zh.wikipedia.org/wiki/Unicode

三、编码举例

字符串“I am 君山”用 UTF-16 编码，下面是编码结果：

用 UTF-16 编码将 char 数组放大了一倍，单字节范围内的字符，在高位补 0 变成两个字节，中文字符也变成两个字节。从 UTF-16 编码规则来看，仅仅将字符的高位和地位进行拆分变成两个字节。特点是编码效率非常高，规则很简单，由于不同处理器对 2 字节处理方式不同，Big-endian（高位字节在前，低位字节在后）或 Little-endian（低位字节在前，高位字节在后）编码，所以在对一串字符串进行编码是需要指明到底是 Big-endian 还是 Little-endian，所以前面有两个字节用来保存 BYTE_ORDER_MARK 值，UTF-16 是用2 字节或4字节来表示的 UCS-2 或 Unicode 转换格式，通过代理对来访问 BMP 之外的字符编码。