🆕 New Source: SAP HANA

[jgngo]

Tell us about the new integration you’d like to have

• SAP HANA to Postgres

Describe the context around this new integration

• Creating a finance data mart
• Sync every hour

Describe the alternative you are considering or using

• Prefect or Airflow

https://help.sap.com/viewer/4fe29514fd584807ac9f2a04f6754767/2.0.03/en-US/20a1569875191014b507cf392724b7eb.html

1. Investigate sandbox environment

• https://developers.sap.com/tutorials/hxe-ua-install-using-docker.html
• or create a sandbox in cloud.

2. Generate new connector and implement connections https://developers.sap.com/tutorials/hana-clients-jdbc.html
3. Create unit test
4. Integration test
5. Comprehensive:

• Investigate and prepare use cases:
  https://help.sap.com/viewer/6811c09434084fd1bc4f40e66913ce11/7.4.19/en-US/4ec07a0b6e391014adc9fffe4e204223.html
• Implement comprehensive test.
  6: Documentation, Prepare pull request, pass all checks.

┆Issue is synchronized with this Asana task by Unito

[marcosmarxm]

Thanks @jgngo I added your request to the roadmap. We just released Airbyte CDK that improves building new connectors. Are you willing to build this connector? (it's not a problem if not)

[jgngo]

Would love to build one, if I get time to do it.

[sherifnada]

blocked on setting up a SAP Hana instance in AWS/GCP due to the instance being very cumbersome to setup.

[trymzet]

Sorry if this is SPAM, but I randomly came across this article which describes setting up Hana in a docker container: https://feitam.es/how-implement-cdc-change-data-capture-of-database-hana-apache-kafka-connect-sap-connector-incremental-mode-to-postgresql/. You may find it useful.

[sherifnada]

Thanks @trymzet ! We'll take a look at it soon once we get to implementing this connector

[jomach]

I'm working since some months on building a kafka source connector for SAP. Be aware you will have Problems on calculating deltas. As SAP Tables do not have "updatedAt" column for example.

[jomach]

You can go over triggers but you could go over ODP queues. See https://init-software.de/odpsc/#/

[sherifnada]

Thanks for the heads up @jomach

[ofranz]

Any updates or timeline on this connector?

[matheus-rossi]

Hello! I've worked with SAP HANA recently, developing an incremental data pipeline to extract data from HANA.

As @jomach commented, the absence of columns like "updatedAt" was a challenge in the beginning, but I found an internal column present in all SAP HANA tables, which is ("$rowid$").

Based on this community post: https://blogs.sap.com/2023/03/12/rowid-function-in-HANA/

I documented this in my docs; it may be useful: https://matheusrossi.com/development/erp/sap-hana/

@sherifnada Maybe this ("$rowid$") column could be used to perform incremental reads from tables...

[QuanLab]

$rowid$ seems to be working fine in this case. Is there anyone working on this?

[jrosskopf]

If you guys are not necessarily interested in SAP HANA via the database, but higher level integration on SAP ERP tables, BAPIs, but also SAP Business Warehouse (BW) cubes and querys and ODP data sources (e.g. CDS views, but also HANA tables), then maybe have a look at: https://github.com/datazoode/erpl-airbyte-connectors

We just did an experiment with Airbyte to see whether there is demand from the community. In case you are interested, open an issue in the repo or directly contact us via: https://erpl.io/contact.html

[jomach]

Update: ODP data sources were killed by SAP and cannot be used by external extractors anymore. So I guess we will need to go with https://github.com/datazoode/erpl-airbyte-connectors

[jrosskopf]

@jomach, thank you for your reply, and the plug the ERPL airbyte connector. Just to clarify, ERPL also can replicate data via the ODP RFC interface, but also supports BAPIs, read table or read from BW cubes / queries via BICS.

Maybe also 2cents on the killing of ODP. They literally say, that the do not support ODP via RFC anymore and suggest ODP OData for 3rd party integrations. Of course their own tools will continue using ODP RFC, as it is currently the only reliable, scalable data replication method on SAP application server level. However, what is interesting is:

• That it is not clear how this reduction of scope of the RFC interface interferes with the exiting contracts of customers. And
• What the many other third party integrations like Azure Data Factory, Matillion, Google Cloud Data Fusion, Theobald, Informatica, ... to just name a few ... will do.

From SAPs side is is just a way to push their Datasphere Product, which at the current state is super immature and has ridiculously expensive data egress rates.

[jomach]

IMHO is ridiculous that they force clients that put clients data into the SAP BW and then SAP say oh no you cannot pull the data out...

[jomach]

Currently I'm using the field REQTSN on SAP BW. This is a incremental Timestamp

[jomach]

@jgngo does the erpl support incremental loading ?

[ngohuuduc]

Will this connector run on JDBC ? Since we want to replace our current JDBC legacy pipeline

[jomach]

What is the issue from jdbc for you?

[ngohuuduc]

Ah.
We are looking to replace our current ETL solution with Airbyte from SAP HANA (JDBC) and hope this will work soon.