Streamsets HTTP client && JSON parse && local FS use

Streamsets contains a rich set of components, Origin processer destination

Test examples are integrated with the HTTP client and JSON processing

Start the service

Using Docker

• Create pipeline
  
• Add an HTTP client component
  
• Configuring the HTTP Client

  HTTP Service Address: Https://raw.githubusercontent.com/rongfengliang/json-learning/master/user3.json

• Add JSON parse
  
• Configuring JSON Parse

  Note that the path is/starts with a way like jsonpath and JSON patches, depending on the code

• Add Local FS Component

  Output to temp file system
  

• Connecting components
  

Run

The converted Data

{"name":"json-patch","version":"1.0.0","main":"index.js","license":"MIT","dependencies":{"generate-schema":"^2.6.0","json-merge-patch":"^0.2.3","json-templater":"^1.2.0","json8-patch":"^0.4.1","jsonpatch":"^3.0.1","jsonschema":"^1.2.4","uglify-js":"^3.4.5","uglifyify":"^5.0.1","uglifyjs":"^2.4.11"},"apps":"{\"users\":{\"name\":\"dalong\",\"age\":33}}","type":"dGhpcyBpcyBhIGV4YW1wbGU=","scripts":{"t":"node t","p":"node p","s":"node s","gs":"node gs","build":"browserify t.js | uglifyjs -c > bundle.js","live":"live-server","m":"node m"},"devDependencies":{"browserify":"^16.2.2","live-server":"^1.2.0"},"result":{"users":{"name":"dalong","age":33}}}

Resources

Https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Origins/Origins_ Overview.html#concept_hpr_twm_jq

Streamsets HTTP client && JSON parse && local FS use