Simple Estimation of memory usage of database data in Java

Database Data occupies memory in Java. Conclusion: 1. database records are stored in JAVA. It takes about 4 times of memory space to use an object (which is a common ORM processing method), and 10 times of space to use HashMap to store KV data. 2. if your main data is text, you can estimate the space by two times. The above is a general data test conclusion. For your reference

Database Data occupies memory in Java. Conclusion: 1. database records are stored in JAVA. It takes about 4 times of memory space to use an object (which is a common ORM processing method), and 10 times of space to use HashMap to store KV data. 2. if your main data is text, you can estimate the space by two times. The above is a general data test conclusion. For your reference

Simple Estimation of memory usage of database data in Java

Conclusion:

1. The database records are stored in JAVA. It takes about 4 times the memory space to use objects (The ORM is generally used). It takes 10 times the space to store KV files such as HashMap;

2. If your main data is text, you can estimate the space by two times.

The above is a general data test conclusion for your reference.

The space occupied by database records is relatively good. For example, an int occupies 4 bytes, bigint occupies 8 bytes, date occupies 3 bytes, datetime occupies 8 bytes, and varchar occupies variable length bytes. If you do not want to perform exact calculations, you can easily know the total space occupied by the table and the average length of each record in the database through statistics.

When we use JDBC to access the database, we are often asked about memory overflow. Because java is an object-oriented language, JVM is used for automatic memory recovery and memory computing cannot be performed in normal ways, this article provides an idea and reference for estimating memory.

First, the ing between database objects and memory in common JDBC is given.

MySQL	Oracle	JDBC
Int		Integer
Int unsigned		Long
BigInt		Long
BigInt unsigned		BigInteger
Decimal	Number	BigDecimal
Varchar	Varchar2	String
Date		Date
Datetime	Date	Timestamp
Timestamp	Timestamp	Timestamp
Clob	Clob	String
Blob	Blob	Byte []
Text	Clob	String
Float	Binary_float	Float
Double	Binary_double	Double

The above is a better understanding. Next we need to use the memory space of common JAVA objects. This can be done through the Instrumentation interface provided by JDK 5, or through the open-source sizeOf. jar to test, the author is through sizeOf. jar. The test results are as follows:

Object	64-bit JVM compression pointer	64-bit JVM non-compressed pointer
Integer	16	24
Long	24	24
Object	16	16
Date	24	32
Timestamp	32	40
String_0	48	64
String_1	56	72
String_10	72	88
String_100	248	264
StringBuilder	24	32
BigDecimal	40	48
BigInteger	64	80
HashMap	128	216
HashMap_0	72	96
HashMap_100	576	1112
HashMap_10000	65600	131160
ArrayList	80	144
ArrayList_0	40	64
ArrayList_100	440	864
ArrayList_10000	40040	80064
Shortlist	48	80
LinkedHashMap	96	144
ClassA	32	40
ClassB	40	48
ClassC	40	56

Because the current host is usually 64-bit, 64-bit JVM starts from JDK1.6.45. When the maximum JVM memory is smaller than 32 GB, the compression pointer feature is automatically enabled, so that the memory usage of objects is much less, the table above shows that at least 1/3 of the space is reduced.

The following is a test based on the database data.

Assume that the mysql database has an emp table with the following structure:

CREATE TABLE `emp` (  `id` int(11) NOT NULL,  `create_time` datetime DEFAULT NULL,  `modify_time` datetime DEFAULT NULL,  `name` varchar(16) DEFAULT NULL,  `address` varchar(256) DEFAULT NULL,  `age` smallint(6) DEFAULT NULL,  `height` decimal(10,2) DEFAULT NULL,  `weight` decimal(10,2) DEFAULT NULL,  `phone` varchar(20) DEFAULT NULL,  PRIMARY KEY (`id`)) ENGINE=InnoDB DEFAULT CHARSET=utf8

The sample data is as follows:

Hm. put ("id", 1988); hm. put ("createTime", new Date (); hm. put ("modifyTime", new Date (); hm. put ("name", "Zhang Sanfeng"); hm. put ("address", "room 188, No. 808 Xihu Avenue, Hangzhou, Zhejiang Province"); hm. put ("age", 88); hm. put ("weight", new BigDecimal (88); hm. put (& quot; height & quot;, new BigDecimal (188); hm. put ("phone", "1388888888 ");

Based on the above sample data, the valid data is about 80 bytes, And the occupied space in MySQL is about 120 bytes.

The following is the test space for converting to HashMap and Emp objects in java:

Object	64-bit JVM compression pointer	64-bit JVM non-compressed pointer
HashMap_empty	128	216
HashMap_full	1360	1832
Emp_empty	72	112
Emp_full	464	600

According to the test results above, the space occupied by data to JAVA has increased a lot. If the data is stored in HashMap with 64-bit compression pointers, it takes 1360 bytes. The space is about 11.3 times that of the database, if the object is an ordinary Emp object, 464 bytes are required, 3.8 times that of the database.

If we read emp information from the database by PAGE, 50 records are displayed on each page and saved in List, the HashMap needs to be 68KB, And the emp object needs to be 23KB.

Based on this simple test, we can conclude that:

Database records are stored in JAVA. It takes about 4 times of memory space to use an object (which is generally handled by ORM). It takes 10 times of space to use HashMap to store KV files.

If your data differs greatly from the reference data, such as the text of the main data, the space can be simply estimated by two times.

The above is a general data test conclusion for your reference.

The following is the test code:

Import net. sourceforge. sizeof. sizeOf; import java. io. IOException; import java. math. bigDecimal; import java. math. bigInteger; import java. SQL. SQLException; import java. SQL. timestamp; import java. util. *; public class TestSize {static {SizeOf. skipStaticField (true); // java. sizeOf will not compute static fields // SizeOf. skipFinalField (true); // java. sizeOf will not compute final fields // SizeOf. skipFlyweightObject (true); // java. sizeOf will not compute well-known flyweight objects} public static void main (String [] args) throws SQLException, IOException, IllegalAccessException {TestSize ts = new TestSize (); ts. testObjectSize (); ts. testDataSize (); System. out. println ("OK");} public void testObjectSize () {System. out. println ("Integer:" + SizeOf. deepSizeOf (new Integer (56); System. out. println ("Long:" + SizeOf. sizeOf (new Long (56L); System. out. println ("Object:" + SizeOf. sizeOf (new Object (); System. out. println ("Date:" + SizeOf. sizeOf (new Date (); System. out. println ("Timestamp:" + SizeOf. sizeOf (new Timestamp (System. currentTimeMillis (); System. out. println ("String_0:" + SizeOf. deepSizeOf (new String (); System. out. println ("String_1:" + SizeOf. deepSizeOf (new String ("1"); System. out. println ("String_10:" + SizeOf. deepSizeOf (new String ("0123456789"); System. out. println ("String_100:" + SizeOf. deepSizeOf ("0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"); System. out. println ("StringBuilder:" + SizeOf. deepSizeOf (new StringBuilder (); System. out. println ("BigDecimal:" + SizeOf. deepSizeOf (new BigDecimal ("34535643.23"); System. out. println ("BigInteger:" + SizeOf. deepSizeOf (new BigInteger ("34535643"); System. out. println ("HashMap:" + SizeOf. deepSizeOf (new HashMap (); System. out. println ("HashMap_0:" + SizeOf. deepSizeOf (new HashMap (0); System. out. println ("HashMap_100:" + SizeOf. deepSizeOf (new HashMap (100); System. out. println ("HashMap_10000:" + SizeOf. deepSizeOf (new HashMap (10000); System. out. println ("ArrayList:" + SizeOf. deepSizeOf (new ArrayList (); System. out. println ("ArrayList_0:" + SizeOf. deepSizeOf (new ArrayList (0); System. out. println ("ArrayList_100:" + SizeOf. deepSizeOf (new ArrayList (100); System. out. println ("ArrayList_10000:" + SizeOf. deepSizeOf (new ArrayList (10000); System. out. println ("inclulist:" + SizeOf. deepSizeOf (new inclulist(); System. out. println ("LinkedHashMap:" + SizeOf. deepSizeOf (new LinkedHashMap
  
   
(); System. out. println ("ClassA:" + SizeOf. deepSizeOf (new ClassA (); System. out. println ("ClassB:" + SizeOf. deepSizeOf (new ClassB (); System. out. println ("ClassC:" + SizeOf. deepSizeOf (new ClassC ();} public void testDataSize () throws IOException, IllegalAccessException {HashMap hm = new HashMap (); System. out. println ("HashMap_empty:" + SizeOf. deepSizeOf (hm); hm. put ("id", 1988); hm. put ("createTime", new Date (); hm. put ("modifyTime", new Date (); hm. put ("name", "Zhang Sanfeng"); hm. put ("address", "room 188, No. 808 Xihu Avenue, Hangzhou, Zhejiang Province"); hm. put ("age", 88); hm. put ("weight", new BigDecimal (88); hm. put (& quot; height & quot;, new BigDecimal (188); hm. put ("phone", "1388888888"); System. out. println ("HashMap_full:" + SizeOf. deepSizeOf (hm); Emp emp = new Emp (); System. out. println ("Emp_empty:" + SizeOf. deepSizeOf (emp); emp. setId (1, 1988); emp. setCreateTime (new Timestamp (System. currentTimeMillis (); emp. setModifyTime (new Timestamp (System. currentTimeMillis (); emp. setName ("Zhang Sanfeng"); emp. setAddress ("room 188, No. 808 Xihu Avenue, Hangzhou City, Zhejiang Province"); emp. setAge (28); emp. setWeight (new BigDecimal ("88"); emp. setHeight (new BigDecimal ("188"); emp. setPhone ("13888888888"); System. out. println ("Emp_full:" + SizeOf. deepSizeOf (emp);} class ClassA {} class ClassB extends ClassA {} class ClassC extends ClassB {} class Emp {private Integer id; private Timestamp createTime; private Timestamp modifyTime; private String name; private String address; private Integer age; private BigDecimal height; private BigDecimal weight; private String phone; public Integer getId () {return id ;} public void setId (Integer id) {this. id = id;} public Timestamp getCreateTime () {return createTime;} public void setCreateTime (Timestamp createTime) {this. createTime = createTime;} public Timestamp getModifyTime () {return modifyTime;} public void setModifyTime (Timestamp modifyTime) {this. modifyTime = modifyTime;} public String getName () {return name;} public void setName (String name) {this. name = name;} public String getAddress () {return address;} public void setAddress (String address) {this. address = address;} public Integer getAge () {return age;} public void setAge (Integer age) {this. age = age;} public BigDecimal getHeight () {return height;} public void setHeight (BigDecimal height) {this. height = height;} public BigDecimal getWeight () {return weight;} public void setWeight (BigDecimal weight) {this. weight = weight;} public String getPhone () {return phone;} public void setPhone (String phone) {this. phone = phone ;}}}