A Practical Guide to JSON Parsing with Python

Key blog summary

• Shows how to deal with “dirty” or non‑standard JSON using libraries like chompjs and dirtyjson, which are essential when working with real‑world JavaScript-heavy sites.​
• Explains how to use Python’s standard json module (load, loads, dump, dumps) for everyday JSON parsing tasks, including reading and writing files.​
• Covers practical web scraping patterns such as discovering JSON APIs via browser dev tools, extracting JSON from script tags, and parsing hidden data for a web scraping company.​
• Introduces performance‑oriented tools like orjson and ijson to speed up parsing and enable streaming of very large JSON files without exhausting memory.​

In the modern landscape of data extraction, JSON (JavaScript Object Notation) has quietly become the backbone of the internet. For any web scraping company or data engineer, the ability to fluently handle JSON is no longer just a “nice-to-have”—it is a critical survival skill.

While traditional scraping often focuses on parsing HTML with tools like BeautifulSoup, modern web applications rely heavily on client-side rendering. This means the valuable data you want isn’t always in the HTML markup; it’s often hidden in API responses or embedded directly into JavaScript variables as JSON.

This guide goes beyond the basics. We will explore how to python parse json efficiently, handle the messy “dirty” JSON found in real-world scraping, and scale your operations for massive datasets.

Before diving into the code, it is essential to understand why we prioritize JSON. When you load a dynamic website (like an infinite-scroll e-commerce page), the browser is likely fetching data in the background.

• Structure: JSON is structured data. You do not need to write complex XPaths or CSS selectors to “guess” where the price is. It is simply product[‘price’].
• Completeness: APIs often return more data than what is displayed on the screen, such as inventory counts, internal IDs, and timestamp data.
• Efficiency: Parsing a 50KB JSON file is significantly faster and less CPU-intensive than parsing a 2MB HTML DOM.

Python comes equipped with a robust built-in library simply called json. For 90% of general use cases, this is your starting point.

The two most critical functions you will use are loads() (load string) and load() (load file).

1. Python JSON Parse (String)

When you fetch data from an API using a library like requests, the response usually comes as a text string.

import json

json_string = '{"id": 101, "name": "ScraperBot", "active": true}'

# Convert string to Python dictionary
data = json.loads(json_string)

print(f"Bot Name: {data['name']}")

2. Python Reading JSON File

When you have saved data locally and need to ingest it for analysis:

# Reading from a local file
with open('data_dump.json', 'r') as file:
    dataset = json.load(file)

print(f"Total Records: {len(dataset)}")

Conversely, after you scrape json data python scripts generally need to save it.

• json.dumps(data): Converts a dictionary to a JSON string.
• json.dump(data, file): Writes the dictionary directly to a file.

Pro Tip: Always use indent=4 during development to make the file human-readable, but remove it in production to save space.

This is where we move from “Python basics” to “Industry Expert” territory. A common mistake beginners make is trying to scrape the HTML of a dynamic site.

Strategy 1: The Network Tab Inspection

Before writing a single line of scraping code, open your browser’s Developer Tools (F12) and switch to the Network tab. Select Fetch/XHR.

Reload the page. If you see requests returning JSON, you have struck gold. You can mimic these requests in Python to get the data directly, bypassing the HTML entirely.

Strategy 2: Extracting from <script> Tags

Modern frameworks like Next.js or Nuxt.js often embed the initial state of the application into a <script> tag in the HTML. This is technically JSON, but it is wrapped inside HTML.

Here is how to scrape json data python developers often miss:

import json






import re
from bs4 import BeautifulSoup

html_content = """
<html>
  <body>
    <script id="__NEXT_DATA__" type="application/json">
      {"props":{"pageProps":{"product":{"id": "A123", "price": 99.99}}}}
    </script>
  </body>
</html>
"""

soup = BeautifulSoup(html_content, 'html.parser')
script_tag = soup.find('script', id='__NEXT_DATA__')

if script_tag:
    # Python load json directly from the tag content
    data = json.loads(script_tag.string)
    price = data['props']['pageProps']['product']['price']
    print(f"Price found: ${price}")

In the real world, data is rarely clean. You will often encounter “JSON” inside JavaScript files that isn’t technically valid JSON. It might have trailing commas, single quotes, or comments—all of which will cause Python’s standard json library to crash.

The Problem:

var product = {
    'id': 123,  // Single quotes and trailing comma!
    name: "Widget" // Unquoted key!
}

The Solution:
For a web scraping company, the library chompjs is a game-changer. It is designed to “chomp” through JavaScript objects and turn them into valid Python dictionaries.

import chompjs

js_blob = "{'id': 123, 'name': 'Widget',}" # Invalid JSON
clean_data = chompjs.parse_js_object(js_blob)

print(clean_data['name']) # Output: Widget

Using chompjs is significantly more reliable than writing your own Regular Expressions to clean up data.

Sometimes you need to parse a file that is larger than your available RAM (e.g., a 10GB export of product data). Attempting to json.load() this will crash your script.

The solution is streaming. The library json allows you to read a JSON file iteratively, piece by piece, rather than loading the whole thing into memory.

import ijson

filename = "massive_product_feed.json"

# Imagine the JSON is a list of objects: [{"id":1}, {"id":2}, ...]
with open(filename, 'rb') as f:
    # "item" refers to the objects inside the main array
    for item in ijson.items(f, 'item'):
        process_product(item) # Handle one product at a time

This technique allows a machine with 4GB of RAM to process a 100GB JSON file flawlessly.

Even experienced developers encounter json.decoder.JSONDecodeError. Here is how to handle them professionally.

1. The BOM (Byte Order Mark) Issue

Sometimes a file looks like valid JSON but fails to parse because of invisible characters at the start of the file.
Fix: Use encoding=’utf-8-sig’ when opening the file.

with open('data.json', 'r', encoding='utf-8-sig') as f:
    data = json.load(f)

2. The “Expecting Property Name” Error

This usually means keys are not wrapped in double quotes (standard JSON requirement).

Fix: If you are scraping from a webpage source, this is likely a JavaScript Object, not JSON. Switch to chompjs or dirtyjson.

Once you have mastered python json parse techniques, you need to store the data efficiently.

1. JSON Lines (JSONL): Instead of one giant JSON array [{}, {}, {}], store one JSON object per line.
   • Why? You can append new scraped rows without reading/rewriting the whole file. It is also compatible with BigQuery and standard log processors.
2. Compression: Text-based JSON compresses incredibly well. Saving files as .json.gz can save 80% of disk space.

import gzip
import json

with gzip.open('data.json.gz', 'wt', encoding='UTF-8') as zipfile:
    json.dump(data, zipfile)

Robust JSON parsing with Python sits at the heart of modern data extraction, especially for any serious web scraping company that relies on API responses and JavaScript-rendered content. By combining the standard json module with specialised tools like orjson, ijson, chompjs, and dirtyjson, developers can reliably handle everything from clean API payloads to huge, messy datasets embedded deep inside web applications.