[WIP] GDrive abstract file batch source plugin extention #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
docs/GoogleDrive-batchsource.md
Outdated
| If no such file can be found, an error will be returned. | ||
|
|
||
| **Sample Size:** The maximum number of rows that will get investigated for automatic data type detection. | ||
| The default value is 1000. This is only used when the format is 'xls'. |
Collaborator
There was a problem hiding this comment.
this will be used in tsv and csv also
Collaborator
Author
There was a problem hiding this comment.
Update to
**Sample Size:** The maximum number of rows that will get investigated for automatic data type detection.
The default value is 1000. This is used when the format is `xls`, `csv`, `tsv`, `delimited`.
docs/GoogleDrive-batchsource.md
Outdated
| The default value is 1000. This is only used when the format is 'xls'. | ||
|
|
||
| **Override:** A list of columns with the corresponding data types for whom the automatic data type detection gets | ||
| skipped. This is only used when the format is 'xls'. |
Collaborator
There was a problem hiding this comment.
used in csv, tsv also
Collaborator
Author
There was a problem hiding this comment.
Update to
**Override:** A list of columns with the corresponding data types for whom the automatic data type detection gets
skipped. This is used when the format is `xls`, `csv`, `tsv`, `delimited`.
src/main/java/io/cdap/plugin/google/drive/source/GoogleDriveFileSource.java
Outdated
Show resolved
Hide resolved
|
|
||
| @Override | ||
| public FileStatus[] listStatus(Path path) throws FileNotFoundException, IOException { | ||
| return GoogleDriveUtils.listStatus(driveService, path); |
Collaborator
There was a problem hiding this comment.
are we handling filters for this
Collaborator
There was a problem hiding this comment.
added support for filters
| // Query Google Drive for files in the directory | ||
| String query = "'" + dirId + "' in parents and trashed = false"; | ||
| FileList result = driveService.files().list() | ||
| .setQ(query) |
Collaborator
There was a problem hiding this comment.
existing filters should be passed here
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
3 times, most recently
from
a09cada to
d832951
Compare
…actor Code Refactor
…-fields Make fields nullable
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
from
58ad16a to
54913a2
Compare
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
from
54913a2 to
e8dd7e0
Compare
…ileBatchSource-ui [PLUGIN-1906] Add AbstractFileBatchSource GDrive [UI + Docs]
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
from
e8dd7e0 to
2034022
Compare
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
from
b453f4a to
439d403
Compare
psainics
force-pushed
the
feat/AbstractFileBatchSource
branch
from
439d403 to
4dc5202
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
New google drive plugin will
UI Changes
Reordering
Moved
AuthenticationaboveBasic[OLD]
[NEW]
New Fields
1) Structured Schema Required
Default Value
TrueDoc
UI [under basic]
Visibility
2) Format
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true3) get-schema
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true3) Path Field
4) Regex Path Filter
5) Read Files Recursively
false6) Allow Empty Input
false7) Sample Size
Doc
UI [under basic]
Default Value
1000Visibility
Structured Schema Requiredis set to true8) Override
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true9) Sample Size
Doc
UI [under basic]
Default Value
1000Visibility
Structured Schema Requiredis set to true10) Delimiter
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format is delimited11) Enable Quoted Values
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format like csv12) Use First Row as Header
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format like csv and xls13) File Encoding
Doc
UI [under advance]
Visibility
Structured Schema Requiredis set to true and format like csv and xls14) Terminate Reading After Empty Row
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format xls15) Select Sheet Using
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format xls16) Sheet Value
Doc
UI [under basic]
Visibility
Structured Schema Requiredis set to true and format xlsCode Changes
GoogleDriveFileSourceGoogleDriveFileSystemGoogleDriveRecordReader,GoogleDriveSourceFileFromFoldertoStructuredRecordto make it compatible with abstract file sourceGoogleDriveSourceConfig