Uploading Huge data to ERPNext

Why We Needed This

1. Massive Data Volume: Sometimes you’re dealing with hundreds of thousands of rows—too much for ERPNext’s UI to handle smoothly. Splitting your data into hundreds of CSV files becomes necessary.
2. Command-Line Efficiency: Using bench data-import, you can process each file one-by-one from the terminal. It’s faster, more reliable, and avoids browser timeouts or upload errors.
3. Hidden Pitfalls with Logs: If you reuse the same Data Import record or forget to clear old logs, ERPNext throws weird errors like:

if log.success or len(import_log) < self.data_import.payload_count:
TypeError: '<' not supported between instances of 'int' and 'NoneType'

The Solution That Worked

To overcome the import issues and log interference, we built a simple command-line loop that does three key things for each file:

1. Delete all previous Data Import Logs at the beginning of each cycle, so no leftover records cause row skips or comparison errors.
2. Import the file via bench command, which is reliable and handles large datasets far better than the GUI.
3. Repeat for all CSV files split from the original dataset.

#!/bin/bash

SITE="benchui2.erpgulf.com"
DOCTYPE="Sales Invoice"
DIR="/mytmp"
PREFIX="split100_part_"
EXT=".csv"

for i in $(seq 1 100); do
    FILE="$DIR/${PREFIX}${i}${EXT}"
    echo "🧹 Cleaning old Data Import Logs..."
    
    bench --site "$SITE" console <<EOF
import frappe
frappe.db.delete("Data Import Log", {})
frappe.db.commit()
EOF

    echo "🚀 Importing file $i/100: $(basename "$FILE")"
    
    bench --site "$SITE" data-import --doctype "$DOCTYPE" --file "$FILE" --type Insert

    echo "✅ Done with: $(basename "$FILE")"
done

Benefits of This Approach

1. No skipped rows due to stale log entries
2. No errors from comparing int to NoneType
3. No manual clean-up between import runs
4. Completely automated for any number of files

You can modify the script to validate files before import, log results to a file, or even retry failed ones later.

----------------

How to see existing logs in the DB

bench --site benchui2.erpgulf.com console

logs = frappe.db.get_all(
   ...:     "Data Import Log",
   ...:     fields=["name", "data_import", "log_index", "success", "row_indexes", "exception", "docname"],
   ...:     order_by="log_index"
   ...: )
   ...: 
   ...: for log in logs:
   ...:     print(log)

----------

Delete existing sales if you need to import again

bench --site benchui2.erpgulf.com console

invoices = frappe.get_all("Sales Invoice", filters={"docstatus": 0}, pluck="name")
  
for name in invoices:
       try:
           frappe.delete_doc("Sales Invoice", name, force=1)
           frappe.db.commit()
           print(f"✅ Deleted: {name}")
       except Exception as e:
           print(f"❌ Failed: {name} - {e}")

----------

Get invoice count

bench --site benchui2.erpgulf.com console

frappe.db.count('Sales Invoice')

frappe.db.count('Sales Invoice', {'docstatus': 1})

----------------

Bulk Submit invoices, on a massive scale.

#!/bin/bash

SITE="benchui2.erpgulf.com"  # change this to your actual site name

echo "Let me collect all draft invoices..."

bench --site "$SITE" console <<EOF
invoices = frappe.get_all("Sales Invoice", filters={"docstatus": 0}, fields=["name"])
counter = 0
for invoice in invoices:
    doc = frappe.get_doc("Sales Invoice", invoice["name"])
    doc.submit()
    frappe.db.commit()
    counter += 1
    print(f"{counter}. Submitted Invoice: {doc.name}")

print(f"Successfully submitted {counter} invoices. Thank you!!")
exit()
EOF

A multi threaded approach to save time while submitting huge number of invoices is explained here