New Source: HDFS, Hive Integration #35250
Replies: 20 comments
-
|
Just curious about what is the ~size of the files that you are moving from/to HDFS in your use case? |
Beta Was this translation helpful? Give feedback.
-
|
@ChristopheDuong It depends. Sometimes around 1 GB. Tens of GB occasionally. : ) |
Beta Was this translation helpful? Give feedback.
-
|
Hi, did airbyte integrate the hive and hdfs already? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Any updates on this? |
Beta Was this translation helpful? Give feedback.
-
+1 request |
Beta Was this translation helpful? Give feedback.
-
+1 more request |
Beta Was this translation helpful? Give feedback.
-
|
hi everyone, could you please upvote the original issue? we can only see which tickets are most requested if you leave an 👍🏼 emoji reaction on the issue itself |
Beta Was this translation helpful? Give feedback.
-
|
+1 more request For Hive Destination |
Beta Was this translation helpful? Give feedback.
-
|
Would love to see HDFS integration! |
Beta Was this translation helpful? Give feedback.
-
|
+1 more request For Hive/Hdfs/HBase source |
Beta Was this translation helpful? Give feedback.
-
|
+1 more hive/hdfs target |
Beta Was this translation helpful? Give feedback.
-
|
Tagging this for destinations and sources teams until we split this issue into specific work items for specific connectors |
Beta Was this translation helpful? Give feedback.
-
|
Any update on this? Does the S3 connector work if a team is using EMRFS based out of S3? It doesn't appear so, but want to confirm. |
Beta Was this translation helpful? Give feedback.
-
|
S3 connectors (both source and destination) work with any S3 compatible endpoints. AFAIK, we have tested them with min.io |
Beta Was this translation helpful? Give feedback.
-
|
@grishick I'm sorry, my question was quite vague. My team uses Hive via EMRFS on S3, and I think this issue re:partitions is more relevant. I need to do more research, but I don't think the existing S3 destination allows for creating the partitions required for Hive to work properly. Would be interested in hearing otherwise, however! |
Beta Was this translation helpful? Give feedback.
-
+1 more request |
Beta Was this translation helpful? Give feedback.
-
+1 more request) |
Beta Was this translation helpful? Give feedback.
-
|
+1 more request |
Beta Was this translation helpful? Give feedback.
-
|
+1 more request For Hive/Hdfs/HBase source |
Beta Was this translation helpful? Give feedback.
-
|
We are still using Sqooq. |
Beta Was this translation helpful? Give feedback.
-
Tell us about the new integration you’d like to have
Which source and which destination? Which frequency?
Describe the context around this new integration
Which team in your company wants this integration, what for? This helps us understand the use case.
hdfs distcpto move files from hdfs to another hdfsDescribe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
HDFS DFS commandto move files from hdfs to hdfsselect querythrough jdbc from hive server.┆Issue is synchronized with this Asana task by Unito
Beta Was this translation helpful? Give feedback.
All reactions