crate insider

%, thus can greatly reduce sparksql processing data when the data access content, especially in the spark1.6.x to introduce filters, in some cases can greatly reduce the disk IO and memory consumption. For example 4, spark1.6.x+ parquet greatly improve the throughput of data scanning, which greatly improves the speed of data query. Spark1.6.x and spark1.5.x compared to the increase of about 1 time times the speed, in the spark1.6.x operation of the Parquet CPU use is also greatly optimized, effe

Technology insider InnoDB storage Engine Flush List Dirty Page List Redo Log Redolog buffering To prevent dirty pages from going down when they are flushed to disk, you must first Redolog and then modify the page; Data recovery through Redolog (ACID-D persistence ) when a database outage occurs Default size 8M, via innodb_log_buffer_size The time to flush the redolog buffer to the Redolog file

Original: SQL Server 2008 Technical Insider--t-sql query notes1, SQL Programming has many unique features, such as: set-oriented thinking mode, query elements of the logical processing sequence, three-valued logic. If you do not have the knowledge to start using SQL programming, you will get redundant, low performance code, and difficult to maintain.2. The component responsible for generating the actual work plan (execution plan) in SQL Server is: que

Tags: Database engine mysql InnoDBMySQL Technology insider-INNODB Storage Engine-Reading notes (one) As PHP development, the use of MySQL is always indispensable Blog Link http://itsong.net/articles/466.htmlChapter One MySQL architecture and storage engine MySQL is designed as a database of single-process multithreaded architectures ./mysql --help | grep my.cnfYou can see where the MySQL DB instance will look for the config

Tags: mysql innodb reading notesMySQL Technology insider-INNODB Storage Engine-Reading notes (ii) As PHP development, the use of MySQL is always indispensable Series Article blog Link http://itsong.net/articles/466.htmlChapter III file MySQL with InnoDB several types of files parameter file, configuration path, initialization parameters, memory size, etc. Log files, including error log, binary log, slow query log, query log

segment, but the actual network cannot transmit such large data, and the MTU, when the message length exceeds the MTU, is divided into several packet lengths of the MTU (the last segment may be small), and then the transmission is carried out separately.IP segmentation is usually reorganized on the destination host (intermediate devices need to be reorganized if the entire IP packet needs to be viewed).Fragmentation and reassembly work consumes CPU and memory, and may consume excess network ban

, ! (IPCB (SKB)->flags ipskb_rerouted));}Ip_finish_outputstatic int ip_finish_output (struct sk_buff *skb) {#if defined (config_netfilter) defined (CONFIG_XFRM)/ * Policy Lookup after SNAT yielded a new policy * /if (SKB_DST (SKB)->xfrm! = NULL) { IPCB (SKB)->flags |= IPSKB _rerouted; Return Dst_output (SKB); } #endif if (Skb->len > Ip_skb_dst_mtu (SKB) !skb_is_gso (SKB)) return ip_fragment (SKB, Ip_ FINISH_OUTPUT2);//segment Else return Ip_finish_

kernel is compiled CONFIG_IRQBALANCE , interrupt load balancing in the SMP architecture will be present in the kernel as a module.Late_initcall (balanced_irq_init); #define Late_initcall (FN) module_init (FN) //include\linux\init.hIn the balanced_irq_init() function, a kernel thread is created to interrupt load balancing:static int __init balanced_irq_init (void) { ... printk (kern_info "Starting balanced_irq\n"); if (Kernel_thread ( BALANCED_IRQ, NULL, Clone_kernel) >= 0) return 0;else prin

*/ . Suspend = E100_suspend, . Resume = E100_resume, #endif . Shutdown = E100_shutdown, . Err_handler = e100_err_handler,};static int __init e100_init_module (void) { if (((1 Some of these function pointer prototypes:#define DRV_NAME "e100" static int __devinit e100_probe (struct Pci_dev *pdev, const struct pci_device_id *ent) { C2/>struct Net_device *netdev; struct NIC *nic; int err; if (! ( Netdev = Alloc_etherdev (sizeof (struct NIC))) {

notifications:As with real devices, virtual devices also focus on special event notifications that occur with other components of the system. Virtual devices are based on the logic of real devices, and real devices cannot deliver notifications to virtual devices, so notifications for virtual devices need to be done separately. virtual Appliance cannot perceive hardware-triggered notificationsAdditional features of virtual appliancesThe advantages of virtual appliances are usedRegister_netdevice

unsigned long state; The bitmap identifier, whose possible value is enumerated by tasklet_state_xxx atomic_t count; Counter, 0 indicates that the micro task is closed and is not executable. Non 0 indicates that the micro task has been opened Void (*func) (unsigned long); The function to execute unsigned long data; The parameters of the above function};

0 */routing/device Hook */ #define netlink_unused 1/ * UNUSED number */#define NETLINK_USERSOCK 2/ * Reserved for user mode socket protocols */#define Netlink_firewall 3/ * Unused number, formerly Ip_queue */.....The advantage of NetLink's interaction with the kernel relative to other (IOCTL, etc.) user spaces is that with NetLink, the kernel can proactively transmit kernel data to the user, not just as a response to a user requ

the elem is judged. In addition, to determine the validity of the Elem is "! =", so that you can filter both null and undefined, but do not filter the integer number 0, if Elem is a numeric type, by letting the Elem add an empty string, the Elem is converted to a string.If!elem is true, that is, Elem can be converted to false, then the loop is skipped and the next loop is executed. This line of code is used to filter the empty string case. If Elem is an integer number 0, because it has been con

"WCF Technology Insider" translation 15:1th Part _ 3rd Chapter _ Message Exchange Mode, topology and choreography: Message topology, message orchestration and summary of this chapter Message topology The message topology describes how messages are sent between one or more senders and recipients. A message topology can describe a simple application-application connection, but it can also describe complex applications-enterprise connections. In subseq

a transaction commits, its update results are seen by other transactions. Can solve the dirty read problem.repeatedread: In a transaction, the result of reading the same data is always the same, regardless of whether there are other transactions to manipulate the data, and whether the transaction is committed. Can resolve dirty read, non-repeatable read.serialization: Transaction serialization execution, the highest isolation level, sacrificing the concurrency of the system. All problems with c

versioning) information, and so on. SQL Server allows users to create temporary tables for their own use, where the physical location of these temporary tables is tempdb. Note: Each time the SQL Server instance is restarted, the contents of the database are deleted and recreated as a copy of the model. Therefore, When you need to create objects for testing purposes, and you do not want to keep them in the database after testing is complete, you can usually create them in tempdb. Even if you for

the Orders table that are placed by the specified customer within the specified date range, and also takes the behavior affected by the query as an output parameter.View Code Tips: Stored procedures can encapsulate business logic processing, better control security (to help avoid SQL injection), and improve execution performance (reduce network traffic). (3) Trigger: A special stored procedure that, whenever a particular event occurs, invokes the trigger and runs its code. SQL S

segments, data written in parallel on each disk, but no redundancy, easy to loseRAID1 most reliable, 22 mirror, slow reading speedRAID5 (commonly used) also adds parity information when storing data, data and inspection are stored on each disk, one disk is broken, the disk can be recovered through the inspection information of other disks, and requires at least three disks.RAID10(commonly used) first do RAID1 and then do RAID0. Two is a set of groups within a group of backups, inter-group paral

,Options:Map[String,String],data:dataframe): Resolveddatasource = {Dataframtwriter Source:/*** Specifies the behavior when data ortable already exists. Options include:*- `Savemode.overwrite` : Overwrite the existing data.*- `Savemode.append` : Append the data.*- `Savemode.ignore` : Ignore the operation (i.e. no-op).*- `savemode.errorifexists`: Default option, throw an exception at runtime.** @since 1.4.0*/defMode(savemode:savemode): Dataframewriter = { This.Mode= SavemodeThis }Import Java.util

of space, greater than 255 bytes required 2 bytes of space. So the maximum amount of space for varchar (10) is 11 bytes, with 1 bytes to hold the character length.7.BINARY and varbinary: similar to char and varchar, binary and varbinary store binary strings, rather than character strings, which do not have the concept of a character set, sorting and comparison by binary values. N in BINARY (n) and varbinary (n) all represent byte lengths. The binary padding character is 0x00, and the char paddi