Working with JSON
The @dpkit/json
package provides comprehensive support for loading and saving data in JSON and JSONL (JSON Lines) formats. It leverages Polars DataFrames for efficient data processing and supports flexible data transformations through dialect configurations.
Installation
Section titled “Installation”The JSON package is part of dpkit’s modular architecture:
npm install @dpkit/json
Basic Usage
Section titled “Basic Usage”Loading JSON Data
Section titled “Loading JSON Data”import { loadJsonTable } from "@dpkit/json"
// Load from local fileconst table = await loadJsonTable({ path: "data.json" })
// Load from remote URLconst table = await loadJsonTable({ path: "https://example.com/data.json"})
// Load multiple files (concatenated)const table = await loadJsonTable({ path: ["file1.json", "file2.json"]})
Loading JSONL Data
Section titled “Loading JSONL Data”import { loadJsonlTable } from "@dpkit/json"
// Load JSONL (JSON Lines) formatconst table = await loadJsonlTable({ path: "data.jsonl" })
Saving Data
Section titled “Saving Data”import { saveJsonTable, saveJsonlTable } from "@dpkit/json"
// Save as JSONawait saveJsonTable(table, { path: "output.json" })
// Save as JSONLawait saveJsonlTable(table, { path: "output.jsonl" })
Data Formats
Section titled “Data Formats”The package supports two main JSON formats:
JSON Format
Section titled “JSON Format”Standard JSON arrays of objects:
[ {"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
JSONL Format
Section titled “JSONL Format”Newline-delimited JSON objects:
{"id": 1, "name": "Alice"}{"id": 2, "name": "Bob"}
Dialect Support
Section titled “Dialect Support”Dialects provide flexible data transformation capabilities:
Property Extraction
Section titled “Property Extraction”Extract data from nested objects using the property
option:
// Input: {"users": [{"id": 1, "name": "Alice"}]}const table = await loadJsonTable({ path: "data.json", dialect: { property: "users" }})
Item Keys Filtering
Section titled “Item Keys Filtering”Select specific fields using itemKeys
:
// Only load 'name' field from each recordconst table = await loadJsonTable({ path: "data.json", dialect: { itemKeys: ["name"] }})
Array Format Handling
Section titled “Array Format Handling”Handle CSV-style array data with itemType: "array"
:
// Input: [["id", "name"], [1, "Alice"], [2, "Bob"]]const table = await loadJsonTable({ path: "data.json", dialect: { itemType: "array" }})