-
Notifications
You must be signed in to change notification settings - Fork 2
Description
Hello,
I'm trying to run a data flow with a json file that has an array of JSONs.
There are 20800 items in the array, the json is loaded but when I try to use the file data is lost during the data flow.
Even a simple data flow like :
JSON FILE -> SELECT (employee_id, employee_code) -> Sink (csv) will result in the lost of multiple employee_code.
Json data looks like this in array:
[]
{ "employee_id": 999, "employee_code": "12230", "name": "Ptricia", "last_name": "aaaaa", "gender": "Female", "birth_date": "1999-10-29T00:00:00", "nationality": "UK", "marital_status": "Single", "document_id": "AAF497", "document_type": "Select From Below", "social_security_number": "SS24xx27D", "document_valid_to": "2017-05-22T00:00:00", "authorization_type": "Select From Below", "registration_time": "2020-11-09T00:00:00", "address_road_type": "Select From Below", "address": "9 Somewhere", "address_postcode": "BR444", "address_town": "London", "email": "al1454sto@hotmail.com", "telephone1": "4484", "emergency_contact_name": "dddasd", "emergency_contact_last_name": "Jacdddk", "emergency_contact_relationship": "dddd", "emergency_contact_telephone": "22422", "bank_account": "42424", "disability_level": "Select From Below", "academic_education": "0", "salary": 0.0, "status": "Select From Below", "level": "Select From Below", "scale": "Select From Below", "temporary_contract_reason": "Select From Below", "additional_information": "", "custom_field_collection": [ { "name": "Student Loan", "value": "L1", "value_id": "1" }, { "name": "Title", "value": "Miss", "value_id": "3" }, { "name": " Employee has been issued a contract on e ", "value": "Yes", "value_id": "1" }, { "name": "Proof of Eligibility Reason", "value": "EEA Passport from Eea Country or Switzerland", "value_id": "25" }, { "name": "Tax code declaration", "value": "A This is my first job since last 6th of April", "value_id": "1" }, { "name": "Working Time Directive", "value": "No", "value_id": "2" }, { "name": "Pay type", "value": "Hourly", "value_id": "1" }, { "name": "Pay frequency", "value": "2 Weekly", "value_id": "1" }, { "name": "Pay group", "value": "Crew", "value_id": "1" }, { "name": "TUPE Transfer", "value": "UNDEFINED", "value_id": "0" }, { "name": "Sin definir", "value": "21/03/2020" }, { "name": "Sin Definir", "value": "" }, { "name": " App ID", "value": "" }, { "name": "Reason for change", "value": "" }, { "name": "Secondary email address", "value": "" }, { "name": "Building Society Ref", "value": "" }, { "name": "Sort Code", "value": "23212" }, { "name": "Visa Number", "value": "" }, { "name": "I can confirm I have seen the original RTW document on date ", "value": "03/08/2020" }, { "name": "Proof of Eligibility Expiry Date", "value": "0" }, { "name": "Student Loan Start Date", "value": "" }, { "name": "Student Loan End Date", "value": "" }, { "name": "Address Line 2", "value": "0" }, { "name": "Employee earcu", "value": "UNDEFINED", "value_id": "0" }, { "name": "Múltiple Sin Definir", "value": "UNDEFINED", "value_id": "0" }, { "name": "Múltiple Sin Definir", "value": "UNDEFINED", "value_id": "0" }, { "name": "Account Number", "value": "5454" }, { "name": "Account Name", "value": "A hbhb" }, { "name": "Bank Name", "value": "Halifax" }, { "name": "Nomination", "value": "UNDEFINED", "value_id": "0" }, { "name": "Salary Banding", "value": "UNDEFINED", "value_id": "0" }, { "name": "Reason for salary change", "value": "UNDEFINED", "value_id": "0" } ] }
Any help?