OpLog Delivery: How does this work?
How does this work?
This delivery model requires the client to set up their own database with the help of our scripts through a shared git repo, and do data ingestion themselves through S3. This approach requires at least 4 weeks to set up, because it takes more time to ingest all of that data.
Technical readme is at https://gitlab.com/mixrank/mdp/-/blob/master/README.md. This link can not be accessed by clients; clients will get a forked repo with a copy at a different location.
What steps are required from the user’s side?
We would need GitLab username(s) for which we will grant access to the repo for setting up your PostgreSQL database. That repo contains a README.md with instructions on how to set up your own database and how to consume changes.
We also need an S3 ARN from the user to grant access to the S3 bucket with all the data. If you're not using AWS, we could instead create a keypair under our organization.
What is the size of the initial export? Size of update files?
The complete dataset is about 1.5TB compressed, but the size varies depending on what all data feeds are included and the geographies it covers.