How does spark SQL handle date types, time type JSON each object cannot wrap # #问题描述json How does the File date type work? How do I convert from a character type to a date or datetime type?
the JSON file is as follows, with character format type of date
"{" "Name": "Andy", "age": +, "time": "2015-03-03t08:25:55.769z"} {"name": "Justin", "age": +, "time": "2015-04-04 t08:25:55.769z "} {" name ":" Pan "," age ": $," Time ":" 2015-05-05t08:25:55.769z "} {" name ":" Penny "," Age ":" Time " : "2015-05-05t08:25:55.769z"} "' Default inferred Schema: ' Root |--_corrupt_record:string (nullable = True) |--Age:long (nullabl E = True) |--name:string (nullable = True) |--time200:string (nullable = True) '
Test Code "val fileName =" Person.json "val sc = sparkutils.getsclocal (" JSON file Test ") Val SqlContext = new Org.apac He.spark.sql.SQLContext (SC) val jsonfile = SqlContext.read.json (fileName) Jsonfile.printschema () "# #解决方案
### Programme I,JSON data time is long seconds or milliseconds
### Scenario two, custom schema ' val fileName = ' Person.json ' val sc = sparkutils.getsclocal ("JSON file Test") val sqlcontext = new Org.apache.spark . sql. SqlContext (SC) val schema:structtype = Structtype (mutable. Arrayseq ( Structfield ("name", StringType, True), Structfield ("Age", StringType, True), Structfield ("Time", Timestamptype, true)); val jsonfile = SqlContext.read.schema (Schema). JSON (FileName) Jsonfile.printschema () jsonfile.registertemptable ("person") Val Now:timestamp = New Timestamp (System.currenttimemillis ()) val teenagers = Sqlcontext.sql ("SELECT * FROM Person WHERE age >= <= and time <= ' +now+ "'") Teenagers.foreach (println) &NBSP;&N bsp; val dataFrame = Sqlcontext.sql ("SELECT * from", WHERE age >=, and age <=-and time <= ' 2015-03 -03 16: 25:55.769 ' ") Dataframe.foreach (println)" # # #方案三, SQL Build table CREATE TABLE SQL ' create temporary table person IF not EXISTS [(Age:long, Name:string, Time:timestamp)] USING org.apache.spark.sql . JSONOPTIONS (path ' Person.json ')Syntax create [temporary] TABLE [IF not EXISTS] [(Col-name Data-type [, ...])] USING [OPTIONS ...] [As] ""
### Programme IV,with textfile convert
From for notes (Wiz)
How spark SQL Handles date types
