Data mining-understanding data

1. a dataset consists of data objects. A Data Object (sample, instance, data point, object, and data tuples) represents an object.

 

Ii. Attribute types

An attribute is a data field that represents a feature of a data object. The attribute can be nominal, binary, ordinal, or numerical value.

Observation is the observation of a given attribute.

1. Nominal attributes: names of some objects.

2. Binary Attribute: Boolean attribute.

3. ordinal property: There is a meaningful degree between values.

4. Numeric attribute: there is a quantity of values. (Range scale and ratio scale)

{

Range Scale: can compare the difference between the sum. There is no inherent zero point.

Ratio scale: it has an inherent zero point and can calculate the ratio of multiples.

}

5. discrete property: a finite or infinite number of values, which can be expressed without an integer.

6. Continuous attribute: Numeric attribute. Generally, floating point values are used.

7. mathematical expectation: mean value.

 

III. Basic statistical description

The basic statistical description can be used to identify the nature of data and filter data that does not conform to the main nature (noise and outlier)

The basic statistical description can be divided into: central trend, data distribution, and graphic display.

1. Central trend:

Mean-average

Median-the value at the most intermediate position

Mode-Maximum number of occurrences

Middle column-maximum mean

2. Data Distribution:

Range-difference between the maximum value and the minimum value

Quantile: the relationship between attribute values and probabilities. Probability: P = 1/2n, 0 <p <1.

Quartile-values are sorted by size and quartile

Quartile range-Q3 (value at 0.75)-Q1 (value at 0.25)

Five-digit Summary-min, Q1, median, Q3, Max

Box chart-min ---------- | Q1 | median Q3 | ----------------- Max

Variance-the average value of the square of each data and the average value. Deviation between attributes and mean.

Standard deviation-square root of variance

3. graphic display:

Bar Chart (bar chart and frequency histogram): displays the frequency data.

Pie Chart, quantile chart, divided into number-quantile chart,

Scatter chart: Relationship trend of two attributes.

Iv. Data similarity and heterogeneity

1. Data Matrix: an array of objects and attributes, that is, n objects x P attributes

Example: Name age sex

A 16 1-object O1

B 16 1-object O2

C 16 1-object O3

[O1name, o1age, o1sex

O2name, o2age, o2sex

O3name, o3age, o3sex]

2. homogeneous matrix: array of objects and objects, that is, n objects x n objects. The closeness between n objects.

 

Example: Name age sex

A 16 1-object O1

 

B 16 1-object O2

 

C 16 1-object O3

 

[0

 

D (2, 1) 0

 

D (3, 1) D (3, 2) 0]

D (I, j) is the opposite-sex measurement between object I and j. The larger the value, the larger the difference. Conversely, similarity measurement SIM (I, j) = 1-D (I, j)

The following describes d (I, j )................

(1) Measurement of the closeness of a nominal property: Calculated using the non-matching rate. Formula: d (I, j) = (p-M)/P

P-Total number of attributes, and m-same number of attributes. P-M indicates the number of different attributes.

(2) similarity measurement of Boolean attributes:

　　　

Object I	Object J
		1	0	Sum
	1	Q	R	Q + R
	0	S	T	S + T
	Sum	Q + S	R + T	P = q + S + T + R

 

 

 

 

Now let's look at the similarity: Q and T. That is, similarity measurement: d (I, j) = (q + T)/P = (q + T)/(q + S + T + r)

Conversely, the opposite sex is a different measurement value .. That is, S and R, D (I, j) = (S + r)/P

Of course, what we calculate is symmetric binary. What is a symmetric Binary Attribute? Both are meaningful and important in reality.

Next, asymmetric binary similarity is assumed that the Boolean value 0 does not make much sense in real life.

In this way, asymmetric binary similarity: SIM (I, j) = Q/P, because the original similarity values are Q and T, but t does not make much sense.

(3) closeness measurement of numerical attributes: Euclidean distance, Manhattan distance, and minovsky distance...

Data mining-understanding data