ORACLE,MONGODB Data Processing related

Task one: The project run log is automatically backed up and processed into MongoDB.

Difficulty: string cutting, date-time format processing, Linux timed tasks

Process: Log processing No outside of the valid fields from the unstructured log text interception, extract valid information for the next analysis to use. The log information in this project is actually quite structured, the problem of string cutting is mainly a row of data cut once can not complete the acquisition of all information, need to use different cutting methods to obtain.

Some minor problems were encountered while processing the resulting string time. 01/JAN/2018:01:14:42, this format of the text time, directly with the SimpleDateFormat object according to ("YYYY/MMM/DD:HH:MM:SS") processing will produce an exception.

Exception in thread "main" java.text.ParseException:Unparseable Date: "01/jan/2018:01:14:42"

The problem is mainly in the month of Jan, online data found that the English month in accordance with the "MMM" format can be resolved, but many blogs ignore the location of the region information, which led to the above exception occurred.

Therefore, the region should be explicitly defined when processing this time format. SimpleDateFormat ("Yyyy/mmm/dd:hh:mm:ss", locale.us), in order to follow the "MMM" to parse the English month.

Because the backup work is scheduled to run on the server for a long time, the log of the previous day is processed regularly. Therefore, I chose to use Maven to write a small project, which can simplify the packaging process, but also suitable for scheduled tasks.

In the Linux timed task, referring to 52355681 of this blog, Bo Speakers of the more detailed. So the problem is solved.

Task two: Oracle data scheduled backup to MongoDB

Due to project reasons, the data is mainly stored in Oracle database, and the teacher wants to save this part of data in MongoDB for easy analysis.

The small problem encountered in this process is when MAVEN is packaged. Use the assembly packaging plug-in for a packaged package with dependencies.

MVN assembly:assembly, an error occurred during the packaging process

Failed to execute goal in Project Readoracle:could not resolve dependencies for project Readoracle:readoracle:jar:1.0-sna Pshot:failure to find com.oracle:ojdbc6:jar:11.2.0.1.
0 in Https://repo.maven.apache.org/maven2 is cached in the local repository

The problem is quite clear, because Ojdbc6:jar can not find the reason, the online search is due to Oracle's OJDBC licensing issues, resulting in the MAVEN central warehouse does not have this related jar package. So you need to package it with a local jar file,

However, the author uses a local Nexus server and should not have this problem. Looked up a lot of information, also did not understand exactly what reason.

Finally solved the problem in 77503266, still not very clear why. Explore the MAVEN packaging process later.

Finish.

ORACLE,MONGODB Data Processing related