Skip to content

Data flow - Json file with 20800 rows loses data #14

@ValerioFG

Description

@ValerioFG

Hello,
I'm trying to run a data flow with a json file that has an array of JSONs.
There are 20800 items in the array, the json is loaded but when I try to use the file data is lost during the data flow.

Even a simple data flow like :
JSON FILE -> SELECT (employee_id, employee_code) -> Sink (csv) will result in the lost of multiple employee_code.

Json data looks like this in array:

[]

{ "employee_id": 999, "employee_code": "12230", "name": "Ptricia", "last_name": "aaaaa", "gender": "Female", "birth_date": "1999-10-29T00:00:00", "nationality": "UK", "marital_status": "Single", "document_id": "AAF497", "document_type": "Select From Below", "social_security_number": "SS24xx27D", "document_valid_to": "2017-05-22T00:00:00", "authorization_type": "Select From Below", "registration_time": "2020-11-09T00:00:00", "address_road_type": "Select From Below", "address": "9 Somewhere", "address_postcode": "BR444", "address_town": "London", "email": "al1454sto@hotmail.com", "telephone1": "4484", "emergency_contact_name": "dddasd", "emergency_contact_last_name": "Jacdddk", "emergency_contact_relationship": "dddd", "emergency_contact_telephone": "22422", "bank_account": "42424", "disability_level": "Select From Below", "academic_education": "0", "salary": 0.0, "status": "Select From Below", "level": "Select From Below", "scale": "Select From Below", "temporary_contract_reason": "Select From Below", "additional_information": "", "custom_field_collection": [ { "name": "Student Loan", "value": "L1", "value_id": "1" }, { "name": "Title", "value": "Miss", "value_id": "3" }, { "name": " Employee has been issued a contract on e ", "value": "Yes", "value_id": "1" }, { "name": "Proof of Eligibility Reason", "value": "EEA Passport from Eea Country or Switzerland", "value_id": "25" }, { "name": "Tax code declaration", "value": "A This is my first job since last 6th of April", "value_id": "1" }, { "name": "Working Time Directive", "value": "No", "value_id": "2" }, { "name": "Pay type", "value": "Hourly", "value_id": "1" }, { "name": "Pay frequency", "value": "2 Weekly", "value_id": "1" }, { "name": "Pay group", "value": "Crew", "value_id": "1" }, { "name": "TUPE Transfer", "value": "UNDEFINED", "value_id": "0" }, { "name": "Sin definir", "value": "21/03/2020" }, { "name": "Sin Definir", "value": "" }, { "name": " App ID", "value": "" }, { "name": "Reason for change", "value": "" }, { "name": "Secondary email address", "value": "" }, { "name": "Building Society Ref", "value": "" }, { "name": "Sort Code", "value": "23212" }, { "name": "Visa Number", "value": "" }, { "name": "I can confirm I have seen the original RTW document on date ", "value": "03/08/2020" }, { "name": "Proof of Eligibility Expiry Date", "value": "0" }, { "name": "Student Loan Start Date", "value": "" }, { "name": "Student Loan End Date", "value": "" }, { "name": "Address Line 2", "value": "0" }, { "name": "Employee earcu", "value": "UNDEFINED", "value_id": "0" }, { "name": "Múltiple Sin Definir", "value": "UNDEFINED", "value_id": "0" }, { "name": "Múltiple Sin Definir", "value": "UNDEFINED", "value_id": "0" }, { "name": "Account Number", "value": "5454" }, { "name": "Account Name", "value": "A hbhb" }, { "name": "Bank Name", "value": "Halifax" }, { "name": "Nomination", "value": "UNDEFINED", "value_id": "0" }, { "name": "Salary Banding", "value": "UNDEFINED", "value_id": "0" }, { "name": "Reason for salary change", "value": "UNDEFINED", "value_id": "0" } ] }

Any help?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions