1 Matching Annotations
  1. Apr 2022
  2. pythonspeed.com pythonspeed.com
    Processing large JSON files in Python without running out of memory
    1
    1. pyxelr 17 Apr 2022
      In the previous version, using the standard library, once the data is loaded we no longer to keep the file open. With this API the file has to stay open because the JSON parser is reading from the file on demand, as we iterate over the records.

      For ijson.items(), the peak tracked memory usage was 3.6 MiB for a large JSON, instead of 124.7 MiB as for the standard json.load()

      Python ijson JSON
    Visit annotations in context

    Tags

    Annotators

    URL

    pythonspeed.com/articles/json-memory-streaming/