What open source software and technologies are used behind Twitter?

Without open-source software, Twitter will not exist. Each tweet you send and receive will require open-source software in the process of sending it to mobile terminals and PCs. We are curious about how many open source software Twitter uses. In addition, we want to know the impact of open source on the culture of Twitter.

We interviewed Chris aniszczyk, the open source Manager of Twitter, to share the open source story of Twitter with us. Aniszczxk will give a keynote speech at this month's linuxcon (San Diego, CA, August 29-31): the open source technology behind a tweet. Casino

Let's take a look at aniszczy's View of open source and Twitter's open source culture.

Please give us a brief introduction to the "the open source technology behind a tweet" report you will make on linuxcon "?

On the surface, Twitter provides a simple Real-Time Message Service that transmits a message unit called tweets consisting of 140 bytes. Let's take a closer look at the complexity of running this service: more than 0.4 billion of the tweets are sent every day. At such a scale, you must deal with interesting real-time engineering problems. In this report, I will explain how we can address these challenges and why we choose open source software to address them. The report covers the entire lifecycle of a tweet, from our backend service to front-end display. I expect the audience to have a better understanding and understanding of open-source technologies after hearing the report, and know what happened behind a simple tweet when it appeared in their timeline.

How many open source software is used behind a tweet communication scenario?

We use a lot of open-source software. In my opinion, it is natural to make this decision. Because the use of open-source software allows us to customize code in the process of rapid development of companies and services to meet our rapid iterative engineering needs. On Twitter, when we plan to launch a new project, we will always evaluate our needs and the capabilities that open source can provide to us, and tend to customize open source software to meet our needs. In this way, most of Twitter is built on open source software. As a result, the open source method is now a common part of our culture. In addition, a positive feedback was obtained from the open-source community and fed back to the open-source community. We also shared a lot of our code on GitHub.

Below are examples of some open source software we use:

• MySQL is widely used and is the main storage tool of tweet. We have developed MySQL fork in the open for cooperation with the open source community;
• Cassandra, hadoop, lucence, pig, and some other Apache Foundation projects are used by our architecture to enhance the service capabilities of data analysis and search. We also gave back to these projects and funded the Apache Software Foundation.
• Memcached is widely used in our Cache architecture to cope with the increasing traffic we tell. We have recently opened up twemcache, which is developed based on the memcached code.

In addition, we provide some open-source software:

• Lago is a traffic generation tool developed by us to help us test services before they are launched in the production environment. Lago provides the traffic and precision needed to test our services for the Twitter environment.
• Zipkin is a distributed tracing tool developed by us to help us collect timing data, which is generated for unrelated services in Twitter API requests.
• Scalding is a Scala library. Using this library, you can easily develop mapreduce jobs on hadoop and use the built-in integrated Scala and JVM.

I would also like to mention Apache mesos, which makes it very easy to develop distributed applications and share data center resources. We use it in many places on Twitter, from executing Job Analysis on hadoop to running Rails Applications. It is really one of the basic applications of Twitter. You can view this speech for more information.

What does Twitter work like? Is your culture influenced by open source?

If you spend some energy in the open-source community, you will realize that the open communication of information will have a positive impact on the whole world. On Twitter, we keep this in mind that every employee has the willingness and opportunity to participate. We hold the all-hands meeting every week. At this meeting, sharp questions are raised and answered, because we keep an open mind and trust each other from inside to outside. Furthermore, we established an open source office about a year ago to support open source organizations that are vital to us. We are grateful for the great work made by the open-source community and want to maintain a healthy relationship with the open-source community.

In terms of Engineering Culture, Twitter itself is a real-time event-driven engineering problem. We also turn our engineering culture into real-time response and event-driven. We want agile, small-step Iterative engineering cultures that develop along with the growth of the entire company. More than 0.4 billion million tweets are sent every day, and a large number of tweets are delivered. We also hold hackweeks every quarter. Employees can spend a week working on a variety of projects they are truly passionate about. These projects do not need to be related to their daily responsibilities.

What open source software and technologies are used behind Twitter?