Data Schema for Uber Drivers' SARs

[Gilligan511]

@emmanuel-hestia @pdehaye @wilderino

this folder contains the results from the SAR of an Uber driver in Geneva. There are three main files (a pdf and two csv files), in addition to a record of the email conversation the driver had with Uber and the original ZIP file.

@wilderino Is the schema we would develop from this redundant with the driver2019 schema?

Also, here is a link to the consent form we are providing drivers; it's limiting in what Hestia employees can do with raw data from SARs, but my understanding is that developing the schema is within the limits as long as you don't process the data.

[emmanuel-hestia]

Hello, thank you very much for the information. I am studying the files to see how they can integrate with our analysis tools. You input is most helpful, we appreciate it!

…

On Mon, 11 Jul 2022, at 10:46, Gilligan511 wrote: @emmanuel-hestia <https://github.com/emmanuel-hestia> @pdehaye <https://github.com/pdehaye> Hi Emmanuel, this <https://drive.infomaniak.com/app/drive/193995/files/173857> folder contains the results from the SAR of an Uber driver in Geneva. There are three main files (a pdf and two csv files), in addition to a record of the email conversation the driver had with Uber and the original ZIP file. Are you able to use this to create a data schema of the Uber data (if you think this would be useful for hestia tools - thinking that in a near future we want drivers to analyze their data this way). — Reply to this email directly, view it on GitHub <#6>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AU33BROSX3TC2YVR6MQKRN3VTPNNLANCNFSM53GXFTAA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Gilligan511]

@emmanuel-hestia @wilderino

Following up on the issue above - I am attempting to help an Uber driver who has provided us with the results from a Subject Access Request. He recovered the data on September 24, 2020, which I believe is the most up to date driver SAR we have so far (correct me if I am wrong).

When I upload the data zip file here here for analysis with our tools, it does not work.

The tool is looking for the file names
"1 - Driver Detailed Device Data.csv"
"2 - Rider Detailed Device Data.csv"
"Trip*.csv".

However, the September 2020 compressed folder received from Uber contains three files:
"Driver Detailed Device.csv"
"[DRIVER NAME] - SAR.pdf"
"Rider Detailed Device.csv"

The pdf file has information not in the other csvs, such as driver rating.

I tried renaming "Driver Detailed Device.csv" and "Rider Detailed Device.csv" to "1 - Driver Detailed Device Data.csv" and "2 - Rider Detailed Device Data.csv". The experience displayed no results for "1 - Driver Detailed Device.csv" and the map and spreadsheet on the "2 - Rider Detailed Device.csv" tab appeared but the map seemed to be missing any plotted points. "Trip*.csv" was still an error message because it is not in the compressed file.

The uber sample data is based on 2019 or 2018, correct? Can we change the experience so that I can make it usable for the 2020 data?

Below are screenshots from my attempt to upload after I changed the names of the csv files:

[wilderino]

Sorry, for that... I didn't know the format/name of the files and I had to do an experience with the sample data-set so it's "normal" that it doesn't work for new SAR requests.

For "Driver Detailed Device.csv" and "Rider Detailed Device.csv" it's weird it doesn't work after the renaming. Could you send me the names of the columns and the first line (for the types) of the files ? For Driver Data could you do "Inspect Element"->"Console" and check if there is an error ?

For the PDF, we can't handle them for now (we have no ways to read the data from the PDF) so the Trips tab won't be usable...

[emmanuel-hestia]

Hello,

I see two CSV files:

• 'Driver Detailed Device.csv'
• 'Rider Detailed Device.csv'

The headers are:

• 'Driver Detailed Device.csv'
  date,time,lat,lng,speed,horizontal_accuracy,gps_time,vertical_accuracy,os,os_version,model,language,version,carrier,imei_number,serial_number,carrier_mcc,carrier_mnc,ip_address,battery_level

• 'Rider Detailed Device.csv'
  date,time,lat,lng,speed,horizontal_accuracy,gps_time,vertical_accuracy,os,os_version,model,language,version,carrier,imei_number,serial_number,carrier_mcc,carrier_mnc,pin_location_lat,pin_location_lng,ip_address,battery_level

Compared to what we used to have, the data schemes are as:

Driver	Rider	Form (?)
date	date	-
time	time	-
lat	lat	lat
lng	lng	lng
speed	speed	speed
horizontal_accuracy	horizontal_accuracy	horizontal_accuracy
gps_time	gps_time	-
vertical_accuracy	vertical_accuracy	vertical_accuracy
os	os	os
os_version	os_version	os_version
model	model	model
language	language	language
version	version	version
carrier	carrier	carrier
imei_number	imei_number	imei_number
serial_number	serial_number	serial_number
carrier_mcc	carrier_mcc	carrier_mcc
carrier_mnc	carrier_mnc	carrier_mnc
-	pin_location_lat	-
-	pin_location_lng	-
ip_address	ip_address	ip_address
battery_level	battery_level	battery_level
-	-	app_name
-	-	counter
-	-	city_name
-	-	city_name_id
-	-	battery_status
-	-	cpu_abi
-	-	google_advertising_id
-	-	google_play_services_status
-	-	google_play_services_version
-	-	id
-	-	ios_advertiser_id
-	-	manufacturer
-	-	region_iso2
-	-	rooted
-	-	voiceover
-	-	wifi_connected
-	-	year_class
-	-	driver_id
-	-	driver_online
-	-	driver_status
-	-	trip_uuids
-	-	surge
-	-	trip_id
-	-	enabled_user_notification_types
-	-	epoch_ms
-	-	last_user_action
-	-	last_user_action_epoch_ms
-	-	altitude
-	-	course
-	-	gps_time_ms
-	-	session_id
-	-	session_start_time_ms
-	-	type

[wilderino]

The experience for the 2 csv should be fixed (just need to redeploy the website). @fquellec is trying to see if we can read the pdf.

[wilderino]

So we found a solution for the PDF but it's not perfect. In Excel we can import data from a PDF file. Once we imported the trips in the Excel we can convert it to a CSV file that can be used in experience (the name needs to be in the format Trips*.csv). If you want I can send you the CSV directly.

[Gilligan511]

Hi all, hope you had a good holiday weekend.

Last week, Jessica and I sat down with the driver who received the SAR mentioned above to look at their recovered data in https://digipower.academy/experience/uber-driver

There are two issues they identified.

The first is the most relevant to the Geneva workshop with drivers on Friday. The data in the pdf that became the "Trips.csv" file was returned from Uber without a unit, and it turns out Uber returned the distance for the rides in miles, not kilometers. The driver initially thought that the visualization and the displayed data on Digipower Academy was incorrect because the distance did not match the records in the app, which are in kilometers.

Things become a little complicated, because we have no way of knowing if the next files recovered by a driver from Uber will have distance in miles or kilometers.

I think the solution would be to have the "distance" column in the Trips tab have an option to convert miles to kilometers with this text displayed in a popup or something similar:

"If the distance you recovered from Uber does not match the distance recorded in your phone for any particular trip, it is possible that Uber returned your data in miles instead of kilometers. Click here to convert the data kilometers."

Right now, I don't think there is a better solution than to have drivers be able to manually check against the distance recorded in their apps, which is something that can be facilitated in workshops or done individually.

The second issue is that the timestamps in the "Trips.csv" have an hour added to them when displayed on Digipower Academy. The rest of the row from the csv is displayed correctly, so maybe this is a timezone issue?

Let me know if I can clarify anything.

[Gilligan511]

Adding to this that I figured out the time difference varies sometimes 1 hr, and sometimes 2hr. So, it's probably the difference between Central European Time and Greenwitch.