DATA QUALITY

A useful dataset starts with a schema

How to define fields, handle missing values and check duplicate records.

ScrapeTech Solutions · · 5 MIN READ

Start with the decision the data supports

A dataset is easier to use when every field has a reason to exist. For a price comparison, that might mean a stable product identifier, the current price, its currency and the source URL. A catalog analysis may also need variants and availability. Define those needs before collecting a larger batch.

Keep a short field dictionary alongside the sample. For each field, document its type, whether it is required and how a missing value is represented. This gives the person receiving the file a way to distinguish a valid empty value from a collection problem.

Choose a small, explicit schema

The product sample in this hub uses the following fields. It is a demonstration format, so adapt it to the receiving system rather than treating it as a universal standard.

The sample contains synthetic products and placeholder domains. It includes an intentional duplicate so that the workbench can demonstrate its exact-match check.

Separate missing values from zero and false

A price of zero is different from an unknown price. Likewise, false is a meaningful stock value. The workbench treats absent fields, null and blank strings as empty cells, while preserving numeric zero and Boolean false.

Its empty-cell count is a structural check, not a business validation score. A missing optional description may be acceptable; a missing product identifier may make a record unusable. Decide which fields are required in your project brief and review those fields explicitly.

Understand the duplicate rule

The browser workbench removes records only when their complete JSON content matches, ignoring object-key order. A record with the same SKU and a different price is retained. This protects a changed observation from being silently discarded by an identifier-only rule.

For recurring collection, define the record key and observation time separately. Two prices observed on different days may be valid historical records. For a current catalog snapshot, you might instead require a single record per product and variant. The right choice depends on how the output will be used.

Check the export as well as the input

JSON preserves arrays, nested objects and Boolean values. CSV is convenient for tabular review, but nested values need an agreed representation. This workbench encodes nested values as JSON text inside the CSV cell. Null and missing values become blank cells; if their distinction matters, keep the JSON export.

CSV text fields that begin with formula-like prefixes are prefixed with an apostrophe by the exporter. That can change the displayed text, so check the export settings expected by the receiving spreadsheet or importer.

A useful acceptance check: inspect identifiers, types, empty required fields, duplicate rules and a handful of source-to-record comparisons. Keep the accepted sample with the schema version.

Try the JSON / CSV workbench or download the product sample.

← All engineering notesDiscuss a project ↗