Real-time data retrieval | Voters

Real-time data retrieval

generally available

Thu

Stacker takes too long to update with new data from the airtable base (without having to press Airtable update frequently) We need to do stuff with those data relatively quickly via our portal (within a minute or two).

March 27, 2020

Sze Wing Choi

marked this post as

generally available

Hi everyone,
I'm Sze Wing, a Product Manager at Stacker. Everyone now has access to upgrade their Stacker connection to Airtable which is super exciting! You can read more about how to do that here. https://support.stackerhq.com/hc/en-us/articles/14372703347219-Upgrading-Airtable-data-connections#why-upgrade--0-1
As mentioned below, once you're upgraded, you will automatically have access to real time sync on connections that you have upgraded. You can see an article with more details about that here - https://support.stackerhq.com/hc/en-us/articles/16129474206355-Airtable-real-time-sync#what-is-airtable-real-time-sync-0-0 
Please reach out to us if you have any questions, we're really excited about this improvement.
Sze Wing

Team Stacker

Hi everyone, we're beginning to roll this out as a two part upgrade:
Firstly, we'll be introducing an upgrade for Stacker's connection to Airtable. If you're interested in the why and how of this, you can read this help doc: https://support.stackerhq.com/hc/en-us/articles/14372703347219
Once you're upgraded, you will automatically have access to real time sync on connections that you have upgraded. The team have been working super hard on this and I'm sure a lot of you will be thrilled to see it live. Please do read our help docs article about Real Time Sync to hear more: https://support.stackerhq.com/hc/en-us/articles/16129474206355-Airtable-real-time-sync#what-is-airtable-real-time-sync-0-0
We'll be putting these upgrades out gradually to all workspaces, so you may not see the upgrade button immediately, but don't worry, we'll get to you.

Rutger Hofste

Amazing, this would address our biggest limitation for our user experience. It would be great to have a Stacker webhook so a service like Zapier could trigger Stacker to sync from Airtable.

Hasan Karakus

Airtable webhooks API now with OAuth (so stacker can create the webhooks) could solve this :)

We use the webhooks API to sync data between Airtable and Firebase and its instant

Sze Wing Choi

marked this post as

in progress

Sze Wing Choi

Hi everyone,
I understand that you've all been waiting for this request to be fulfilled for a long time so thank you for your patience with us.
With the release of the new Airtable APIs at the end of 2022 we've changed our approach to how we will achieve real-time data, different to that for the beta.
However, to use these APIs we need to create a path for customers to upgrade their Airtable data connections to use the new authentication APIs. The work for this is in progress (as you can see on on our roadmap) and following that we'll be able to start working on bringing real-time Airtable data to all of you.
For those paying for the 5 minute Fast Sync plan at the moment, we'll be reaching out shortly to explain how this plan will be deprecated.
Thanks,
Sze Wing (PM @ Stacker)

Team Stacker

Merged in a post:

Instant Sync

Karim Badra

Many times inputs from Stacker generate automations on Airtable which take 15 minutes to reflect. If you buy an upgrade plan, it makes it 5 minutes. However, even the update creates an unnecessary and problematic lag for users to see the automations generated instantly.

February 21, 2023

Sam Dickie

Hi all, 
I wanted to provide a quick update on some of the work we have been doing recently relating to 'real-time data retrieval`.
We understand how important it is to get data to sync faster between Airtable and Stacker — and we know we should do a better job of updating you on our progress. We’re aware that it is frustrating that Stacker’s sync time is between 5 mins and 15 mins, and this seems unreasonably long compared to some alternative solutions that read data from Airtable completely live. Our technical architecture for ensuring we can serve customers’ apps fast even when they have tens of thousands of records is to cache the data from Airtable within Stacker. This is necessary to achieve high performance with high data volume apps with the current limitations of the Airtable APIs.
We've been experimenting continuously in the background for the past few quarters. Most of these experiments didn't yield the performance we'd expect, so we've not released them publicly. 
However, there are still solutions! And we’ve been working on a few different options one of which we’re now actively testing with a small group of customers. The solution is to allow for push updates from Airtable to directly update Stacker and works by adding a script to each table in Airtable. This gives live data sync between Stacker and Airtable and conserves Airtable API requests — which are rate limited.
At present this requires some manual setup, so we’ve been onboarding customers gradually to make sure we identify all the issues before widening the rollout. We’ll post an update to this thread when we’re ready to onboard additional customers.

Ryan Mammina

Sam Dickie: This is great, a manual push would be sufficient for our use case, we are already running several Airtable automations, so running a push at the end would be easy to integrate into our workflow.

Jeff Miller

Sam Dickie: This is a great approach! I would love to be in the beta testers group for this feature. Right now, I am using a multi-step setup involving Airtable scripts and Zapier webhooks to "push" updates to Stacker. It sounds like what the solution you guys are working on is much easier!

Peter Nelson

Sam Dickie: I've got a customer app that would probably be a good candidate for testing this new approach. Let me know if I can help beta test this.

Jake Russell

Sam Dickie: Thank you for the update. Can I be included in the early stages of testing? I have a consultant on my team who knows how to support a manual set up and any other required advanced knowledge. We will help with getting it ready for rollout. Thank you for your consideration.

Dylan Angell

Sam Dickie: I'm the Data Engineer for a company that would also love to be in the beta test if you're in need of volunteers. Plenty comfortable with any scripting setup necessary.

Jean

Sam Dickie: Hey, would love to be included in the beta ! Thanks a lot

Jake Russell

Stacker team, can you provide an update on the status of this feature? Based on comments, it sounds like this has been in Planned stage for a while now. I would love to get a sense of the current timing here. Thank you.

Jake Russell

100% agree with this. We need instant sync. I'm 99% certain another service I've tested, Softr, has an instant sync. To charge more money for 5 minutes sync (not even instant sync), seems unreasonable. I mean let's just upgrade everyone to instant sync. The technology is there.

→