My 2015 Technical Learning Journal

My 2015 Technical Learning Journal

2015 will soon be over, in a hurry another year. Back to summarize the finishing, found that this year is quite substantial. In the normal work, learn a lot of new things, can't help feeling very gratified! I started writing one months ago and finally finished writing this article before 2016. About this article, although called Journal, but out of the programmer's "obsessive-compulsive disorder", or system classification, the classification method refers to the ThoughtWorks technology radar Tecniques, Languages & Frameworks, Tools, Platforms, the tecniques of which is changed to theory.

1. Theory

From this year began gradually in-depth study of the theory of computer science, continue to follow the course of their own established learning steps to go down.

1.1 Regain Csapp

This year should have been "impact" two or three times Csapp, the effect is quite satisfactory, almost read the thick of the whole book, Harvest quite abundant! From the assembly language, the principle of computer composition, compiling links, system programming, are considered a systematic study. The specific learning notes are as follows, where (2) and (3) the personal feeling is still well organized:

• Six-Star Classic Csapp notes (1) Computer system parade
• Six-Star Classic Csapp notes (2) operation and presentation of information
• The six-Star classic Csapp note (3) machine-level representation of the program
• Six-Star Classic csapp-notes (7) Loading and linking (top)
• Six-Star Classic csapp-notes (10) system I/O
• "Six-Star Classic csapp-notes (11) Network Programming"
• Six-Star Classic csapp-notes (12) concurrent programming (UP)

1.2 Algorithms & Leetcode

This year read a few algorithmic books, but are in accordance with the content of the parallel look, have not read ... At the same time also hands-on with C + + brush some leetcode topics. C + + Although very complex, but do algorithm problem can actually only use the core of the part, as the enhanced version C, you can quickly get started.

The most rewarding thing in this study is how to prove the correctness of the program and algorithm , and how to deal with various cornercase and thinking loopholes when implementing the algorithm. Although the understanding and research is not deep enough, but it does broaden the thinking, let me start thinking systematically how to write the correct code in a scientific way. For details, see the "Programmer's Path to Practice" topic and several Leetcode problem-solving notes:

• The path of Programmer's cultivation-(1) Basic (middle): Performance analysis
• The path of Programmer's cultivation-(1) Foundation (bottom): proof of correctness
• The path of Programmer's cultivation-(2) linear table (top): Arrays and Lists
• The path of Programmer's cultivation-(2) linear table (bottom): Stack, queue
• The path of Programmer's cultivation-(3) sort (UP): basic sort
• The path of Programmer's cultivation-(4) Search (middle): Two forks find tree
• "Leetcode Problem Solving-list (2.2.0) Basic class"
• "Leetcode Problem Solving-list (2.2.1) addtwonumbers"
• "Leetcode Problem Solving-list (2.2.2) reverselinkedlist"
• "Leetcode Problem Solving-list (2.2.3) partitionlist"
• "Leetcode Problem Solving-list (2.2.6) rotatelist"
• "Leetcode-Tree (5.0.0) Basic class"

1.3 Netty Network programming

The project has come into contact with Netty, which is indeed very powerful. Through the use of Netty, we have also collated the knowledge needed to develop a network communication middleware. But limited to the TCP/IP and other network knowledge to do system learning, so the Netty bottom understanding is not deep enough, I hope the future can be added to this part.

• "Using Netty to analyze Redis network protocol"
• "Netty 4 source Code Analysis: Server Start"
• "Netty 4 source Code Analysis: Request Processing"
• "Developing middleware with Netty: Fundamentals of Network Programming"
• "Developing middleware with Netty: Optimizing for high concurrency performance"

1.4 Programmer's self-cultivation

Compiling links has always been a daunting area for me. This year by learning Csaap, incidentally to compile links to learn to open a head. Now also can solve some assembly and C language link wrong, also is a little achievement. At the same time also tried the ANTLR, really good, later if you write some small things to use it. The deepest sentiment to this part is: The original from the high-level language polymorphism to the compilation of JMP is such a thing ah!

• "Programmer self-accomplishment: (1) Target Document"
• "ANTLR v4 Introductory Tutorials and examples"
• Examples of ANTLR and stringtemplate: automatically generating unit test classes

2. Tools

A lot of new tools have been introduced this year, thanks to these tools and authors, so that I can focus more effectively on what I'm doing. you will find that sometimes you simply use a certain tool, and your life changes radically . Like just replaced Sublime+markdown write article, only found that the original can write so many articles each month, and typesetting is 10 times times more beautiful than before! Just began to read in the Watercress reading others recommended books and records of their own reading books, an inattentive, found that they have read so many good books.

2.1 Linux Mint

Thanks to the multi-system support of mint, I finally installed Linux on my own computer, and I have been using Cygwin and virtual machines under Windows. Use of Linux after the feeling of thinking are open, want to use what software to see what source line, deeply deeply affected by the software in the World of freedom! See "Linux Mint 171 Week Experience" for details.

2.2 Sublime Text

After the trial of various text editor, finally no longer "wandering". Sublime text almost satisfies my various needs, the development of various languages, code base management and the following articles to be written. For details, see "Sublime Text 3, C + + development environment building".

2.3 Markdown

This is one of the most important improvements of the year! concentrate on writing content, reduce the interference of style and other trivial things, with sublime text editor and preview plugin is absolutely perfect match! Look back to the previous post written in Word pasted into the CSDN editor, the layout is simply "miserable" ... Now with the markdown, writing efficiency and typesetting quality have greatly improved, MD Plain text file is much smaller than Doc. Plus CSDN official support, although the rendering of efficiency is not particularly beautiful, but can be assured that the use of (in the CSDN promotion markdown of the essay activities also won a T-shirt:).

3. Language and Framework 3.1 Golang

There are many open-source projects on GitHub that golang the language. Taking advantage of the opportunity to study the codis of pea pods, he also learned the basic grammar of Golang. Some places are really new, more convenient than using C directly, may be the gospel of some backstage programmers. But some places are really not used to ...

• Java Programmer's Golang Getting Started Guide (top)
• Java Programmer's Golang Getting Started Guide (bottom)

3.2 YCSB

This year, in the pre-research on some new technologies, Yahoo's YCSB testing framework was used extensively. At first did not know very well, and later familiar with the internal source code after the feeling handy. The operating type and scale of the load, the distribution of the resulting data, and so on can be configured or extended, the function is very complete! See "YCSB Performance Test Tool use" for details.

3.3 Jgroups-raft

Based on jgroups network stack to the implementation of raft consistency protocol, try it out or good! Can be seen as a zookeeper or ETCD in the consistency of the algorithm layer, the algorithm can be connected behind a variety of content, from simple data additions and deletions to complex remote operations. This is also a small harvest this year, you can make a variety of distributed small things behind the Jgroups-raft , instead of relying on heavy zookeeper service cluster. See the principles and examples of distributed conformance Protocol (raft) and the final consideration of distributed development in this paper.

3.4 Thrift

Facebook's thrift still works, and it's easy to start distributed development of multi-component and cross-language at once, as well as support for data types such as enumerations and collections. A complete set of scenarios, from serialization to RPC infrastructure, is available free of charge compared to Google's protobuf. But it seems that thrift does not improve the more advanced calls such as watcher, while noting that the simplest tsimpleserver is a single-threaded blocking server, a request is processed to accept the next connection request and can only be used for testing. For details, see the Modern RPC guide for Java programmers.

3.5 Cucumber

Experience a BDD that may not be so standard and authentic, but it does inspire in the process. At the same time in the mock frame, but also by the powermock powerful shock! Private,static,final methods can be mock, the treatment of various incurable diseases. It can also be used directly with frameworks such as Mockito or Easymock,junit or testng, without having to completely re-learn a new set of APIs.

4. Platform 4.1 in-memory Computing

IMC is the main research direction this year, in addition to writing a lot of relevant articles, many of the Redis articles echo good, it seems that Redis is still very fire. The access links to my Redis series articles are sorted by content:

• "Relational query in Redis"
• Redis monitoring tools, commands and tuning
• "Redis Source Learning: Strings"
• "Redis Source Learning: Lua Script"
• "Pea clip Redis solution codis Source Analysis: Proxy Agent"
• "Pea clip Redis solution codis Source Analysis: Dashboard"
• Comprehensive analysis of the principles and applications of Redis cluster
• "Redis Cluster architecture optimization"

Through the IMC field of learning opportunities, but also in-depth thinking about the distributed system, in particular, look at the post-distributed development of the thinking. It is fortunate to have the opportunity to enter a good field of study after many years of work, and hopefully the future will have a deeper study of IMC and the theory of distributed systems.

4.2 Virtualization

This year I took some time to learn about the amazing Docker of the fire. The result is that Docker didn't disappoint me, it's really good stuff! Now you can write some of your own to use Docker, the code together with the environment to package submissions, developed in different places is very convenient! In the future, you will want to learn more about what technologies are used in Docker. For details, see "Getting Started with Docker: basic usage."

5.Hacking

In addition to the "seriousness" of the learning route, of course, you can learn some interesting new things to do.

5.1 Buffer bombs

As the teaching material of the top elite University of Csapp, some classical experiments in the world can find the courseware of CMU on the Internet, and the most interesting experiment is the buffer overflow attack experiment. It makes you a hacker, try to attack a security vulnerability of the small program, when you hit the success of the CMU to see the congratulatory words, may be excited to tears, haha!

• Csapp Buffer Overflow attack Experiment (top)
• Csapp Buffer Overflow Attack experiment (bottom)

5.2 Linux 0.11 kernel

The Linux kernel is also one of my most interesting areas, although I don't usually do embedded development. This year to pick up the "Orange's: An operating system implementation", but not completely imitation, but reference to the Orange's and Linux 0.11 two version of the Kernel, self-implementation of a simplified version of the kernel, and finally did not finish the process scheduler ... Using this as a theme, we learned about the use of GitHub, GDB, make, Docker and other tools, and learned about the various technologies required for a modern compilation or C-language project.

• "Operating system kernel hack: (a) experimental environment construction"
• OS kernel hack: (ii) Fundamentals of underlying programming
• "Operating system kernel hack: (iii) boot program production"
• OS kernel hack: (iv) kernel prototype

6. Thinking and understanding 6.1 about distributed development

The first thought about distributed development is: code and Data . In traditional software development, the meaning of code and data is more common, but the code corresponds to the application, and the data corresponds to the database. Into the distributed world, see a dazzling variety of background middleware inevitably some confusion, the following is a brief introduction of my mainstream middleware learning process:

• load Balancing : Always listen to people say virtual IP or VIP, wait until the actual installation of the LVS have an intuitive impression, the original simple load balancing has a lot of knowledge, but also to see in three, four or seven layers to do distribution.
• Web server : This part is the most lacking, the Apache and Nginx completely do not understand, and then a little familiar with even read some nginx source code, hand-written plug-ins, only to find nginx really good ah!
• Application Server : This part should be the most familiar to traditional Java developers, from heavyweight WebLogic to the most popular Tomcat and even the most lightweight jetty, every Java programmer can name a few of their own application servers. Later, I came into contact with PHP's fastcgi, further broaden my horizons.
• Message Queuing : activemq sure a lot of people have used, very early know JMS. But the world of Message Queuing is not small, there are rabbitmq and so on high-performance message middleware.
• Cache Middleware : The most familiar is Ehcache, and then naturally transitions to memcached and Redis, which are found in almost any large website backstage.
• Task Scheduler : The asynchronous execution of background tasks is also essential in large Web sites. The earliest use was the most popular quartz. Later used commercial Autosys, supporting a variety of complex execution conditions and job nesting dependencies, very powerful but do not know how the internal mechanism. And then later with the Tbschedule and quartz distributed version, there is a certain understanding of the distributed scheduling inside.

Middleware is software glue between applications, and it focuses on a feature that enables developers to focus on developing business systems. Although each product is designed to be very different from the application scenario, we can still classify it in terms of code and data. Distributed code is actually a Web service, dispatch task, Mr Task, etc., distributed data is distributed storage, Message Queuing, distributed cache and so on. It may be a lot clearer to think like this.

The second thought is about the root cause of the problem of distributed development, and state management should definitely be one. When it comes to the design goals of distributed systems, our brains immediately jump out of the familiar vocabulary of performance, scalability, usability, extensibility, security, and more. But careful thinking will reveal that many of these goals are inextricably linked to state. High performance to consider the state of concurrency control, scaling to consider the state of consistency assurance, high availability to consider the state of backup and recovery. As an example of developing decentralized (peer-centric) distributed systems, there are some insights about state management:

• Consistency guarantee : Use zookeeper as a complete conformance scheme, or use Jgroups-raft semi-finished products as the implementation of the Protocol layer.
• State Transitions : manual implementation of a large amount of work, and the function is not complete, you can use the squirrel and other open source implementation, support external state transformation, internal state transition, event timing trigger, nested status and other functions.
• Intermediate State : Consistency protocols such as Jgroups-raft are required for communication time, and timeouts are considered downtime. So if the protocol layer behind the remote call to do operations, it is necessary to split the state into the intermediate state and final state. After the receives the request from the consistency protocol layer, it returns directly after starting the asynchronous task, when the entire system enters an intermediate state and leader waits for the asynchronous response of all nodes. After each node's asynchronous task executes, it responds to leader. When leader collects all responses, all nodes are notified through the consistency protocol layer into the final state .
• Interface anti-weight : The components of a distributed system often have various problems, so it is unavoidable to retry calls to the interface. Therefore, the interface should be idempotent, that is, the ability to automatically prevent heavy, duplicate input data detection.
• State Persistence : Jgroups-raft is available in two ways: wal log (Write-ahead Logging) and snapshot . The Wal guarantees that the operation logs are successfully written to the persisted storage before performing the actual operation, and the log size can be compressed by regular snapshots.
• State Recovery : The Jgroups-raft on each node during a reboot will be "replayed" through the log at this node, which is problematic for asynchronous intermediate states. so the log and snapshot can be read ahead of time to restore the state machine to the correct state, so that the completed log operation will be rejected by the state machine .
• Local State : Messages sent and received through the consistency protocol layer must be the same for each node. So if each node needs its own message to be entered, the sends global data through the consistency protocol layer, and each node extracts its desired data based on the flag bit in the data .

6.2 About Software Engineering

Only one project has been developed this year, and a decentralized middleware has been in development recently. Because the middle of a period of time without the development of specific business systems, so re-start to do the project and the software development process has some new sentiment, especially how to make a reasonable design according to the requirements, according to the design to produce the correct code. This year's learning is BDD (behavior-driven development), and then re-comb the individual understanding of the various steps of software engineering:

• Requirements Analysis : requirements of a variety of specifications, have met with a brief BRD documents, detailed PRD documents, and even directly in the message described in a paragraph. I think the specification in BDD and the BRD document correspond to the most accurate, is to explain the functional characteristics, typical scenarios, and the most important to the user value. Rather than the detailed input and output parameters in the PRD and the interface design. The roles involved are BA business analysts, product managers, etc.
• Architecture Design : For the system and the component hierarchy within the system, involving the role of architects, development backbone, but also with the project manager to assign personnel, tasks and hours to make a reasonable scheduling.
  
  • Technology selection: Which middleware to choose, whether there are new technologies to pre-research, technical risk points
  • External System interactions: which external systems are involved, timing diagrams called at the system level, and developer coordination
  • Component Partitioning and Interaction: What components can be divided internally, how to communicate, and the overall flowchart
• Detailed design : For each component within the module hierarchy, involving the role of architects, development backbone or ordinary developers. This layer is previously understood to be somewhat blurred, and may be diluted in some business systems, but it is critical in a technically strong development framework or middleware development.
  
  • Module partitioning and collaboration: Access control between modules, which core class for each module does facade or mediator
  • Internal data Model: Independent model, with DTO decoupling for communication between systems
  • Core algorithm Design
  • Glossary of Business and technical terms: affects the naming of various elements, all developers need to unify
  • Coding Specifications and annotation styles
• Coding Phase : For class and method hierarchies. The main is the class design (class diagram, time series diagram, design pattern), in the coding phase may be in the perfect testcase while constantly refactoring. So this part is not stable, to avoid over-design. The roles involved are architects, all developers.

To develop the software can not be separated from the construction of infrastructure. As a result of this software engineering, you can use your company's technical assets or personal experience as early as possible to prepare your infrastructure in advance:

• Environment :
  
  • Dependency Management: Maven/gradle
  • Build: Ant/maven/make
  • Version control: Svn/git
  • CI Continuous Integration: Jenkins
  • Unified IDE and plug-in, with CI
• Project Structure :
  
  • Engineering: Scaffolding generation or handwriting
  • Main packages and folders: named according to glossary, including Java, JS, config file, etc.
  • Main classes and files
• Code :
  
  • Syntax enhancements: JDK 8, Apache Common, Google Guava
  • Dependency Injection: Spring or lighter Guice
  • Configuration parameters and file resolution
  • Exception handling: Assertion, Uniform exception class, exception error code, exception prompt (for internationalization)
  • Concurrency control: The lock and Countdownlatch provided by JDK concurrent are very useful
  • Thread pool: Same as JDK concurrent, guava provides enhanced
  • Event Bus: Guava
  • LOG: SLF4J with specific implementation, customized for the project unified log output format
  • Mathematical functions: Encryption and decryption algorithms, etc.

Personal understanding, whether using BDD or TDD, should at least have a clear understanding of key technology selection, top-level architecture design, and mid-tier module design in mind , rather than just hitting the code directly on the BRD. In addition, do not strictly abide by the rules, a set of the most suitable for their own methodology is also very good, you are not accustomed to the practice of the rules may indeed not be so applicable, may also be your level can not understand and control, no relationship slowly, do not bite the bullet to accept the whole, and finally get a little writing code mood.

now feel that the software development process is like playing chess, the most important thing is to control power . If the pre-design relies on the understanding and experience of business and technology, then the real start to do is to rely on the code of control. In the high-speed development, in many places can "point to stop", in accordance with the overall design control details, so that the code to go to the place, and then gradually re-construction to complete.

6.3 Technical Debt

After summing up, we found some blind spots outside of our own technical route, also can be said to be owed "debt":

• Web and mobile development : Always envy programmers who can visualize their results, but are not interested in web and mobile development, ready to take a time to focus on learning bootstrap, AngularJS, React.js, such as the Popular Front-end framework, such as CODIS with the background program dashboard to practiced hand
• Java 8: Once was so fond of Java,java 5, 6 has a little new features, will study hard half a day. Java 6 seems to be less enthusiastic, and even this big change in Java 8 is indifferent. What's the reason? I think one of the last two years in my spare time to dig deep bottom, architecture, Linux, C and so on; the second is that the company project has been slow to upgrade with JDK versions and has been stuck in JDK 6.
• Methodology and application Framework : As Business systems development work is reduced, there is no particular focus on the new framework of methodologies and application layers for research and development. Although the technical pre-research has been writing prototypes and experimental code, the amount of coding did not decline, but to be wary of the content of the study and the Code quality degradation.
  
  • Methodology: Whether it is a business system or prototype development, is the opportunity to practice, TDD, refactoring, etc. must be constantly practiced to understand
  • Application framework: Spring perimeter products like spring boot, and the latest version of Springside are good re-learning opportunities
• Amateur Practice Program : Although most books read with hands-on exercises and reading notes, they do not write gadgets that are useful to themselves or others. In the future, you should do more hands-on, write something like "use QT to develop a simple browser (a)" In such fun and can help others with small open-source software.

The 2015 summary is here, and 2016 has a bigger challenge! Continue to refuel, cherish every minute!

My 2015 Technical Learning Journal