Java Ten-year technology stack [summary review]

The following is a list of the http://www.tvtv223.com/so/8/default/8.html#36-databases from the library MycatJava Technology Stack

Reference to a lot of information, here is no longer detailed list, you can search by yourself

1 Java Fundamentals: 1.1 Algorithms

• 1.1 Sorting algorithms: Direct insert sort, hill sort, bubble sort, quick sort, direct select sort, heap sort, merge sort, base sort
• 1.2 Two-fork search tree, red-black tree, B-tree, + + tree, LSM tree (corresponding applications, database, HBase, respectively)
• 1.3 Bitset solve problems such as data duplication and presence

1.2 Basic

• 2.1 Migration of string constant pools
• 2.2 String KMP algorithm
• 2.3 Equals and Hashcode
• 2.4 Generics, exceptions, reflections
• hash algorithm of 2.5 string
• 2.6 Hash Conflict Resolution: Zipper method
• 2.7 The principle of the Foreach Loop
• 2.8 The role of static, final, transient and other keywords
• The underlying implementation principle of the 2.9 volatile keyword
• What sort method is used by the 2.10 Collections.sort method
• 2.11 Future interface, futuretask implementation in common thread pooling, etc.
• Internal details of the intern method of the 2.12 string, changes in jdk1.6 and jdk1.7, and implementation of the internal CPP code stringtable

1.3 Design Patterns

• Single-Case mode
• Factory mode
• Decorator mode
• Viewer design Pattern
• Threadlocal design mode
• 。。。

1.4 Regular Expressions

• 4.1 Capturing groups and non-capturing groups
• 4.2 Greedy, barely, exclusive mode

1.5 Java memory model and garbage collection algorithm

• 5.1-class loading mechanism, that is, parental delegation model

• 5.2 Java Memory allocation model (default hotspot)

  Threads shared: Heap zone, Permanent zone thread exclusive: virtual machine stack, local method stack, program counter

• 5.3 Memory allocation mechanism: Young Generation (Eden area, two survivor area), old generation, permanent generation and their distribution process

• 5.4 Strong references, soft references, weak references, virtual references, and GC

• 5.5 Happens-before Rules

• 5.6 Order reordering, Memory fence

• 5.7 Memory generational improvements for Java 8

• 5.8 Garbage Collection algorithm:

  Mark-Clear (Deficiency: inefficient, memory fragmentation)

  The replication algorithm (which solves the above problem, but only half of the memory, for most of the object survival time of the scene, led to a default 8:1:1 improvement, the disadvantage is still need to use the outside to solve the problem may not be able to load)

  Tag Grooming

• 5.8 Common garbage Collectors:

  Cenozoic: Serial collector, parnew Collector, Parallel scavenge collector

  Older years: Serial old collector, Parallel old collector, CMS (Concurrent Mark Sweep) collector, G1 collector (spanning Cenozoic and older years)

• 5.9 Parameters for common GC:-XMN,-XMS,-xmx,-xx:maxpermsize,-xx:survivorratio,-xx:-P rintgcdetails

• 5.10 Common tools: JPS, Jstat, Jmap, Jstack, graphical tools jconsole, Visual VMS, MAT

1.6 Locks and the source code of concurrent containers

• 6.1 Synchronized and volatile understanding
• 6.2 The principle of unsafe class, use it to implement CAs. So the birth of Atomicinteger series, etc.
• 6.3 CAs may be the solution of ABA problems, such as the number of changes to add, version number
• The implementation principle of the 6.4 Synchronizer Aqs
• 6.5 exclusive lock, shared lock, reentrant exclusive lock Reentrantlock, shared lock implementation principle
• 6.6 Fair lock and non-fair lock
• 6.7 The realization principle of the read-write lock Reentrantreadwritelock
• 6.8 Locksupport Tools
• 6.9 condition interface and its implementation principle
• 6.10 HashMap, HashSet, ArrayList, LinkedList, HashTable, concurrenthashmap, treemap implementation principle
• Concurrency Problems of 6.11 HashMap
• The realization principle of 6.12 Concurrentlinkedqueue
• 6.13 Fork/join Frame
• 6.14 Countdownlatch and Cyclicbarrier

1.7-Wire Cheng code

• 7.1 Internal Execution principle
• 7.2 Differences in various thread pools

2 Web aspect: Architecture design of 2.1 SPRINGMVC

• 1.1 Servlet Development problems: Mapping problem, parameter acquisition problem, format conversion problem, return value processing problem, view rendering problem
• 1.2 Springmvc Several major components and interfaces developed to address these issues: handlermapping, Handleradapter, Handlermethodargumentresolver, Httpmessageconverter , Converter, Genericconverter, Handlermethodreturnvaluehandler, Viewresolver, Multipartresolver
• 1.3 The relationship between Dispatcherservlet, container and component
• 1.4 Description of the SPRINGMVC process for the overall request
• 1.5 Springboot

2.2 SPRINGAOP Source

• 2.1 AOP Implementation Classification: Compile time, byte code before loading, three time after bytecode loading to implement AOP

• 2.2 Deep understanding of the roles: AOP Federation, ASPECTJ, JBoss AOP, Spring Self-implemented AOP, spring embedded ASPECTJ. Especially if you can differentiate between the two by code

• 2.3 Interface Design:

  Concepts or interfaces defined by the AOP Alliance: Pointcut (concept, no corresponding interface defined), Joinpoint, Advice, Methodinterceptor, methodinvocation

  SPRINGAOP interfaces and their implementation classes defined for the advice interface described above: Beforeadvice, Afteradvice, Methodbeforeadvice, Afterreturningadvice Aspectjmethodbeforeadvice, Aspectjafterreturningadvice, Aspectjafterthrowingadvice, ASPECTJ for the above interface Aspectjafteradvice.

  SPRINGAOP defined Advisoradapter interface: Convert the above advise to Methodinterceptor

  SPRINGAOP defined Pointcut interface: contains two properties Classfilter (Filter Class), Methodmatcher (filtering method)

  Expressionpointcut interface defined by SPRINGAOP: pointcut expression that introduces ASPECTJ in implementation

  SPRINGAOP defined Pointcutadvisor interface (combining the above advice interface and Pointcut interface)

• 2.4 Springaop Call Flow

• 2.5 SPRINGAOP their own implementation (representing the character Proxyfactorybean) and the ASPECTJ implementation of the way to differentiate

2.3 Spring Transaction system source code and distributed transaction JOTM Atomikos source code implementation

• 3.1 Problems with JDBC transactions
• 3.2 Hibernate improvements to the transaction
• 3.3 For a wide variety of transactions, spring defines the interface of the transaction architecture and how the JDBC transaction and hibernate transactions are fused
• 3.4 Three types of transaction models contain roles and their respective responsibilities
• 3.5 Transaction Code also implementation of business code separation (aop+threadlocal)
• 3.6 Spring Transaction Interceptor Transactioninterceptor Panorama
• 3.7 X/open DTP model, two-phase commit, JTA interface definition
• The realization principle of 3.8 jotm and Atomikos
• 3.9 propagation properties of a transaction
• 3.10 Propagation_requires_new, propagation_nested the principle of realization and difference
• 3.11 Principles of the hang and restore of things

2.4 Database Isolation Level

• 4.1 READ UNCOMMITTED: not submitted
• 4.2 Read Committed: Read Committed
• 4.3 repeatable read: Repeatable READ
• 4.4 Serializable: Serialization

2.5 database

• 5.1 Optimization of database performance

• 5.2 Deep understanding of MySQL's record Locks, Gap Locks, Next-key Locks

  For example, under what circumstances the deadlock will occur:

  Start transaction; DELETE from t WHERE id = 6; INSERT into T VALUES (6); Commit

• 5.3 Lock of INSERT INTO SELECT statement

• 5.4 The acid characteristic concept of the transaction

• 5.5 InnoDB's MVCC understanding

• 5.6 Undo Redo Binlog

  • 1 undo redo can be persisted, what is their process? Why choose Redo to do persistence?
  • 2 undo, Redo combine to achieve atomicity and persistence, why is undo log persisted prior to redo log?
  • 3 Why should undo rely on redo?
  • 4th log content can be a physical or logical log? What are their respective strengths and weaknesses?
  • 5 Redo Log is finally used for physical log plus logical log, physical to page,page internal logic. What's the problem? How to solve? Double Write
  • 6 Undo Log Why not use the physical log and the logical log?
  • 7 Why should I introduce checkpoint?
  • 8 after introducing checkpoint to ensure consistency needs to block user operation for a period of time, how to solve this problem? (The problem is still pervasive, and Redis, zookeeper have similar scenarios and different coping strategies), with synchronous checkpoint and asynchronous checkpoint
  • 9 General process of internal 2PC of transactions (with 2 persistence, redo log and binlog persistence) when Binlog is turned on
  • 10 Explain the above procedure, why is Binlog persisted after redo log, before the storage engine commit?
  • 11 Why should I maintain the order of write Binlog and execute storage engine commit operations between transactions? (That is, the transaction written to the Binlog log must first commit)
  • 12 in order to ensure the above order, the previous approach is locking Prepare_commit_mutex, but this greatly reduces the efficiency of the transaction, how to implement the Binlog group commit?
  • 13 How do I implement a group commit for the persistence of redo log? The process of 2PC within the transaction, 2 persisted operations can be group commit, greatly improve the efficiency

2.6 ORM Framework: MyBatis, Hibernate

• 6.1 The most primitive jdbc->spring of the JDBCTEMPLATE->HIBERNATE->JPA->SPRINGDATAJPA evolution of the road

2.7 Springsecurity, Shiro, SSO (Single sign-on)

• 7.1 Session and Cookie differences and links and how the session is implemented
• 7.2 springsecurity certification process and its relationship to the session
• 7.3 CAs for SSO (see CAS (01)-Introduction)

2.8 logs

• 8.1 JDK comes with logging, log4j, LOG4J2, Logback
• 8.2 Façade commons-logging, slf4j
• 8.3 Log conversions for the 6 melee types above

2.9 DataSource

• 9.1 c3p0
• 9.2 Druid
• 9.3 JdbcTemplate the use and management of connection during SQL statements execution

2.10 HTTPS Implementation principle 3 distributed, Java Middleware, Web server, etc.: 3.1 Zookeeper source code

• 1.1 Client Architecture
• 1.2 Server-side standalone and cluster versions, corresponding request processors
• 1.3 Set-up and activation process of Session of Cluster edition
• 1.4 Leader election process
• 1.5 detailed parsing of transaction logs and snapshot files
• 1.6 Implementing distributed locks, distributed ID Distributors
• 1.7 Implementation of the leader elections
• 1.8 Zab Protocol Implementation Consistency principle

3.2 Serialization and deserialization framework

• Study on 2.1 Avro
• Study on 2.2 Thrift
• Study on 2.3 protobuf
• Study on 2.4 protostuff
• 2.5 Hessian

3.3 RPC Framework Dubbo source code

• 3.1 The implementation of the Dubbo extension mechanism, comparing the SPI mechanism
• 3.2 Release process of the service
• 3.3 Subscription process for services
• Design of 3.4 RPC communication

3.4 NiO module and corresponding Netty and Mina, thrift source

• 4.1 TCP handshake and disconnection and finite state machine
• 4.2 Backlog
• 4.3 BIO NIO
• 4.4 The difference between blocking/non-blocking, synchronous/asynchronous
• 4.5 Blocking IO, nonblocking io, multiplexed io, asynchronous IO
• 4.6 Reactor Threading Model
• 4.7 JDK poll, Epoll and bottom poll, epoll docking implementation
• 4.8 Netty Own Epoll realization
• 4.9 Kernel layer poll, epoll the approximate implementation
• 4.10 Epoll Edge triggering and horizontal triggering
• Eventloopgroup Design of 4.11 Netty
• Bytebuf Design of 4.12 Netty
• 4.13 Netty's Channelhandler
• 0 copies of 4.13 Netty
• 4.14 Netty threading model, especially with regard to business threads and the release of resources

3.5 Message Queuing Kafka, ROCKETMQ, Notify, Hermes

• 5.1 Kafka's File storage design
• Copy replication process for 5.2 Kafka
• 5.3 Kafka copy of the leader electoral process
• 5.4 Kafka Message Loss issue
• The message order problem of 5.5 Kafka
• 5.6 Kafka ISR Design and half-contrast
• 5.7 Kafka itself is doing a very lightweight to stay efficient, many advanced features do not: Transaction, priority message, message filtering, more importantly, the service governance is not sound, once the problem, can not be intuitively reflected, not very suitable for the data demanding enterprise-class systems, It is suitable for large amounts of concurrency, such as logs, but allows a small amount of lost or duplicated scenarios
• Business design of 5.8 Notify and ROCKETMQ
• 5.9 File-based Kafka, ROCKETMQ, and database-based notify and Hermes
• 5.10 What to consider when designing a messaging system
• 5.11 Lost messages, duplicate messages, high availability, etc. topics

3.6 Database Sub-list mycat3.7 NoSQL database mongodb3.8 kv Key System memcached Redis

• 8.1 Redis maintenance and management of clients, read/write buffers
• Implementation of 8.2 Redis transaction
• Implementation of 8.3 Jedis client
• Implementation of 8.4 Jedispool and Shardedjedispool
• 8.5 Redis epoll Implementation, looping file events and time events
• 8.6 Redis's RDB persistence, save and Bgsave
• 8.7 redis aof Command Append, file write, file sync to disk
• 8.8 Redis aof Rewrite, measures taken to reduce blocking time
• LRU Memory recovery algorithm for 8.9 redis
• 8.10 Redis's Master slave replication
• 8.11 Redis's Sentinel high-availability scenario
• Cluster sharding scheme for 8.12 Redis

3.9 Web server tomcat, Ngnix design principles

• 9.1 Tomcat's overall architecture design
• 9.2 Tomcat concurrency control for communications
• 9.3 HTTP request arrives at Tomcat's entire processing flow

3.10 Elk Log Real-time processing query system

• 10.1 Elasticsearch, Logstash, Kibana

3.11 Service aspects

• 11.1 SOA and MicroServices
• 11.2 Consolidated deployment of services, multi-version automatic fast switching and rollback

See the practice of multi-application deployment technology based on Java container

• 11.3 Governance of services: current limit, downgrade

See the Zhang Kaitao of the Great God of Architecture series

Service Flow limit: Token bucket, leaky bucket

Service degradation, service fuse, service isolation: Netflix's Hystrix components

• 11.4 Linear expansion of the service

  How non-stateful services do linear scaling:

  such as general Web applications, directly using hardware or software to do load balancing, simple rotation mechanism

  How a stateful service does linear scaling:

  Extensions like Redis: Consistent hash, migration tool

• 11.5 Service Link monitoring and alerting: CAT, Dapper, Pinpoint

3.12 Spring Cloud

• 12.1 Spring Cloud Zookeeper: For service registration and discovery
• 12.2 Spring Cloud Config: Distributed configuration
• 12.2 Spring Cloud Netflix Eureka: Registering and discovering for rest services
• 12.3 Spring Cloud Netflix hystrix: Service isolation, fusing, and downgrading
• 12.4 Spring Cloud Netflix zuul: Dynamic routing, API Gateway

3.13 Distributed Transactions

• 13.1 JTA Distributed Transaction Interface definition for this integration with the spring transaction system
• 13.2 TCC Distributed Transaction concept
• 13.3 TCC Distributed Transaction Implementation Framework case 1:tcc-transaction
• 13.3.1 tcccompensableaspect section intercept create root transaction
• 13.3.2 The Tcctransactioncontextaspect slice causes the remote RPC call resource to join the above transaction as a contributor
• 13.3.3 Tcccompensableaspect Slice creates a branch transaction based on the markup of the TransactionContext passed by the remote RPC
• 13.3.4 all RPC calls complete, the root transaction starts committing or rolling back, performing commit or rollback of all participants
• 13.3.5 the commit or rollback of all participants, or through a remote RPC call, the provider end executes the Confirm or Cancel method of the corresponding branch transaction
• 13.3.6 transaction storage, cluster sharing problem 13.3.7 transaction recovery, avoid cluster repeat recovery
• 13.4 TCC Distributed Transaction Implementation Framework case 2:BYTETCC
• 13.4.1 JTA Transaction Management implementation, analogy JOTM, Atomikos and other JTA implementation
• 13.4.2 the storage and recovery of transactions, whether the cluster shares problems the caller creates the Compensabletransaction transaction, and joins the resource
• 13.4.3 Compensablemethodinterceptor interceptors inject compensableinvocation resources into spring transactions
• 13.4.4 Spring's distributed transaction manager creates a transaction as coordinator Compensabletransaction type, binds to the current thread, and creates a JTA transaction
• 13.4.5 when executing operations such as SQL, the JDBC and other Xaresource resources used are added to the JTA transaction.
• 13.4.6 Dubbo RPC Remote Call, Compensabledubboservicefilter creates an agent Xaresource, joins the above compensabletransaction type transaction, And in the RPC call procedure, the TransactionContext party creates the branch's compensabletransaction transaction, joins the resource, and then commits the JTA transaction
• 13.4.7 RPC Remote Call came to the provider end, Compensabledubboservicefilter creates a corresponding Compensabletransaction type transaction based on the TransactionContext passed over
• 13.4.8 provider end, execute when meeting @transactional and @compensable, as a participant open the try phase of the transaction, that is, create a JTA transaction
• 13.4.9 provider-End Try execution completes start preparing the commit of the try, simply commits the above JTA transaction, returns the result to the RPC caller to decide whether to rollback or commit
• 13.4.10 the commit or rollback of the transaction after all execution, if the commit commits the JTA transaction (including the submission of xaresource resources such as JDBC), commits the Compensabletransaction type transaction after the commit is successful. If the JTA transaction commits fail, the Compensabletransaction type transaction needs to be rolled back.
• 13.4.11 Compensabletransaction The commit of a type transaction is the submission of compensableinvocation resources and RPC resources, each of which invokes the confirm of each compensableinvocation resource , and the submission of compensableinvocation resources for each RPC resource submission
• 13.4.12 at this point, each compensableinvocation resource's confirm is ready to open a new transaction, and the current thread's compensabletransaction type transaction already exists. So opening a transaction here is just a matter of creating a new JTA transaction.
• 13.4.13 for this, each of the compensableinvocation resources of the Confirm open transactions, and began to repeat the above process, for the JDBC and other resources are added to the newly created JTA transaction, While RPC resources and compensableinvocation resources are still joined to the current thread-bound compensabletransaction type Transaction
• 13.4.14 the current Compensableinvocation resource's confirm open transaction executes, the commit is executed, and the commit of the JTA transaction is still performed, and the commit is completed. The confirm of a compensableinvocation resource completes, continuing the confirm of the next compensableinvocation resource, A new JTA Transaction RPC Resource submission is also restarted (commit of the party compensabletransaction transaction)
• 13.4.15 when all the confirm of the compensableinvocation resource are executed, the commit of the RPC resource is started, a remote call is made, the remote provider Branch transaction is committed, and the remote call process passes the transaction ID
• 13.4.16 provider end, according to the transaction ID passed to find the corresponding compensabletransaction transaction, start to perform the commit operation, the completion of the commit operation to return the end of the response
• 13.4.17 the coordinator receives a response and resumes execution of the next RPC resource submission, and when all RPC resources have completed the corresponding commit, the coordinator is the complete transaction

3.14 Consistency algorithm

• 14.1 raft (see Raft algorithm appreciation)

  • 14.1.1 leader election process, leader election constraints, to include all commited entries, to achieve log more than half of the log is the latest
  • 14.1.2 Log copy process, leader to all follower send appendentries RPC request, half follower reply OK, you can commit the entry, and then respond to the client OK
  • 14.1.3 after the above leader received more than half of the copy, hang, then the subsequent leader can not directly submit to the more than half of the previous term entry (this part has a detailed case to prove, and can say the root cause), the current practice is to create an empty entry in the present term, Then if these newly created entry are copied most of the time, then the more than half entry of the previous term can be submitted
  • 14.1.4 leader once it thinks a term can be submitted, update its own commitindex, Apply entry to the state machine at the same time, and then in the next heartbeat communication with follower, bring the commitindex of leader to follower, let them update, and apply entry to their state machine
  • 14.1.5 from the above process can be seen, as a client, it may occur: Leader that a client request can be submitted (the corresponding entry has been more than half copied), at this time leader hung, have not been able to reply to the client, That is to say to the client, although the request failed, but the request corresponding to the entry is persisted, but sometimes it is the request failed (more than half of the replication success) did not persist successfully, that is, the request failed, the server side may be successful or failed. So at this time need to work on the client side, that is, Cleint end multiplicity Try to use the previous request data to retry, instead of using new data to retry, the server side must be implemented idempotent.
  • 14.1.6 Cluster membership changes
• The ZAB protocol used by the

• 14.2 Zookeeper (see Zookeeper consistency algorithm appreciation)

  • 14.2.1 leader the electoral process. Important: For the collection of votes in different states of the server, voting is required to elect a server that contains all the logs as leader
  • 14.2.2 leader and follower data synchronization process, full-volume synchronization, differential synchronization, Corrections and truncation between logs to ensure consistency with leader. and follower join the system that has completed the election, at this point of synchronization: blocking the leader processing write requests, completing the differences between the log synchronization, but also to handle the existing in-progress request synchronization, complete synchronization, unblocked.
  • 14.2.3 broadcast phase, that is, the normal processing of the client's request, a half response can reply to the client.
  • 14.2. Recovery and persistence of the Chronicles of 4th. Persistence: persist once per a certain number of transaction logs, leader before the election. Recovery: Simply consider that a transaction request that has been written to the log counts as a committed request (regardless of whether it has been copied more than half before), committing all commits. The specific recovery is: Restore the snapshot log first, and then apply the corresponding transaction log

• 14.3 Paxos (see Paxos algorithm proving process)

  • 14.3.1 Paxos's operation process:

    Phase 1: (a) a proposer selects a motion with number N and sends prepare request to all acceptor

    Phase 1: (b) If the acceptor has responded to the prepare request that the bill number is smaller than n, then it is committed to no longer respond to the prepare request or the acceptance request that the motion number is less than N, and find out the value of the largest motion that has been accept returned to the proposer. If the number of the response is larger than N, the prepare request is ignored directly.

    Phase 2: (a) If proposer receives more than half of the acceptors response, a motion (N,V) is proposed, and V is the value of the largest accept motion in all of the acceptor responses above, or proposer its own value. The bill is then sent to all the acceptor. This request is called the Accept request, this step is the so-called request to send a motion, and the previous prepare request is more a construction of the final bill (N,V) process.

    Phase 2: (b) Acceptor received a motion numbered N, if Acceptor has not responded to a prepare request for a motion greater than N, Acceptor will accept the motion, or refuse

  • 14.3.2 Paxos Certification process:

    1 Plenty of questions

    2 Initial accept of acceptor

    3 p2-Requirements for results

    4 p2a-to Acceptor's accept request

    5 p2b-Request for proposer motion (as required by the results)

    6 p2c-Requirements for proposer motions (as required by practice)

    7 Prepare process and P1A

    8 8 Optimization Prepare

  • 14.3.3 Base Paxos and Multi-paxos

4 Big Data Direction 4.1 Hadoop

• 1.1 Usergroupinformation Source code interpretation: Jaas Authentication, user and group relationship maintenance
• Implementation of 1.2 RPC communication
• 1.3 Procedures for agent users
• 1.4 Kerberos Authentication

4.2 MapReduce

• 2.1 MapReduce theory and its corresponding interface definition

4.3 HDFS

• 3.1 MapFile, Sequencefile
• 3.2 ACL

4.4 YARN, Mesos resource scheduling 4.5 Oozie

• 5.1 Oozie Xcommand Design
• The realization principle of 5.2 dagengine

4.6 Hive

• Thrift RPC Communication Design for 6.1 HiveServer2 and Metatore
• Optimization process of 6.2 hive
• 6.3 HiveServer2 Certification and authorization
• 6.4 Metastore Certification and authorization
• 6.5 HiveServer2 User Transfer process to Metatore

4.7 Hbase

• 7.1 HBase Overall architecture diagram
• 7.2 HBase's Wal and MVCC design
• 7.3 Client-side asynchronous batch flush to find the regionserver process
• 7.4 Zookeeper on HBase nodes explained
• 7.5 The Mini, major merge in HBase
• High availability for 7.6 region vs. Highly available implementations of the Kafka partition
• 7.7 Regionserver RPC Call isolation issues
• 7.8 Data from memory brush write to HDFs granularity problem
• Design of 7.9 Rowkey
• 7.10 Memstore and LSM

4.8 Spark

• 8.1 Different ways to deploy
• The way to realize 8.2 sparksql
• 8.3 ...

Java Ten-year technology stack [summary review]