Make Openlineage Run ID/Job ID configurable

[Chaho12]

Docs say that run id code, job id code is fixed to query id.
How about changing it so it is configurable? Run ID has to be unique, but Job ID doesn't seem so.
What do you think @mgorsk1 ?

Also, we were thinking of having a dynamic way to add facet/filtering would be great too!

[mgorsk1]

Hey @Chaho12, as per OpenLineage specification, Job name/id needs to uniquely identify the ETL process - configuring it globally would make every query run in Trino to be treated as same ETL job which I don't think would be correct. Currently we have Job name which is also not ideal (query id will change with every query execution even if query sql doesn't change) but it's closer to OL spec. We could do some hashing to replace query id with query sql hash in job name but that's also prone to too fine grained id generation.

As for filtering yes, this is something already mentioned by community and I think a great idea, feel free to submit a PR - i think schema/tabke filtering would be a good start.

[Chaho12]

query id will change with every query execution even if query sql doesn't change

Yes this is the issue. Unlike airflow or others, same SQL running twice will each have a different job id so it will not combine/merge into one history, resulting in too many jobs (I notice over 90k job history is a burdern in server, slows down fetching API's).

[Chaho12]

Maybe job id as {catalog.schema.table} like SQL? or sth on the line so that user can configure ?