Skip to content
This repository was archived by the owner on Dec 20, 2018. It is now read-only.

Conversation

@devlucasc
Copy link

These changes allow to set option that make conversion of timestamp in long format using a specified precision, for example, microseconds.
The setting of precision option can make avro compatible with BigQuery format, that reads timestamp in microseconds.

These changes allow to set option that make conversion of timestamp in long format using a specified precision, for example, microseconds.
The setting of precision option can make avro compatible with BigQuery format, that reads timestamp in microseconds.
@codecov-io
Copy link

codecov-io commented Mar 4, 2018

Codecov Report

Merging #271 into master will increase coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #271      +/-   ##
=========================================
+ Coverage   92.21%   92.3%   +0.09%     
=========================================
  Files           5       5              
  Lines         321     325       +4     
  Branches       43      48       +5     
=========================================
+ Hits          296     300       +4     
  Misses         25      25

@gengliangwang
Copy link
Contributor

Hi @devlucasc ,
In my opinion:

  1. If we should add this option, we need to add it for all the data sources, instead of just for AVRO.
  2. We can use .map or SQL statement to change the output timestamp unit, which is simple and straight forward to me.

@devlucasc
Copy link
Author

Hi @gengliangwang

  1. The problem only occur in avro data sources
  2. Map is not good when has a complex structure as arrays, structs or array of structs that contains a timestamp field. In this case is needed to make map, flatmap and explode to treat.

Kind regards.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants