"The Truth is in the Cloud" — Why iCloud Data is Failing You, and How Data Abstract can Help.

The past few weeks, everybody has been complaining about iCloud. Or more precisely, they have been complaining about a small subfeature that is crucial to app developers: Core Data syncing via iCloud. Developers are concerned that data syncing via iCloud is difficult and unreliable, and that when things go wrong (and the invariably will), they have little to no recourse to investigate or debug the issues.

Frankly, i’m not surprised about these issues, because from where i stand, Core Data in iCloud uses a fundamentally wrong approach that can only fail. Let’s see how that is so.

Do you remember when Steve Jobs took the stage at WWDC a couple of years back and introduced iCloud? It was going to be better than all of Apple’s previous cloud services, and to explain the reason why, Steve used one catch phrase: “The truth is in the cloud”. Developers rejoiced and gave a standing-o — and rightfully, because Steve was correct.

For a cloud service to work reliably, the “truth” — that is, the authoritative representation of what your data is — needs to be in one central place. The cloud.

And Steve was not lying to us. That claim he made is true for most of iCloud, and if we’re honest, iCloud is working pretty well for most of the services it provides. Cloud-based documents work great. Cloud-based settings (the Key-Value-Store) work great. All the non-developer features of iCloud (Calendar/Contacts, Backups, heck, even things like iTunes Match) work pretty well. The only thing that is causing developers (and by extension, users) headaches is “Core Data in iCloud”. Why is that?

It’s because Core Data in iCloud violates the core promise Steve made about why iCloud is better. Core Data’s truth is not in the cloud. Instead, Core Data merely uses iCloud as a dumb conduit to sync data back and forth between the different devices a user runs your app on — and leaves the onus for sorting out the sync to the individual devices.

And as we all know, “syncing is hard”. And worse, syncing is impossible, if the app doing the syncing has no control over the data flow (for example when the user disables iCloud for a while), and the developer has no way to debug what is going on.

If 15+ years of working with networked data and disconnected clients has taught me anything, it is that this is an unsalvagable situation. In order to have data in your apps and reliably reproduce and “sync” it between multiple devices, you need to take control of that data, and have a single truth source — ideally in the cloud.

It just so happens that here at RemObjects Software, we have spent the past 10 years thinking about these problems, and have developed a technology and libraries around it that solve them.

This blog post is less about selling you on Data Abstract (although i of course hope that you will have a look at it), and more about showing how taking control and hosting your own data will solve your problem in ways that Core Data in iCloud can not — and give your app additional capabilities, to boot. So i will look at this from the perspective of using Data Abstract, our product, but many of the ideas covered will be applicable just as well, should you decide to “roll your own” or use a different infrastructure.

What is Data Abstract?

Data Abstract is a combination of a client library that you link into your iOS or Mac application (it is also available for other platforms), and server-side infrastructure in form of our free Relativity Server.

One of the “downsides” of giving up Core Data/iCloud is that you will need to look at hosting your own data. That seems like a daunting task, but is really not as difficult, or as costly as it might seem. And in fact, you will come to see it as an upside in the long run.

To get started, in most cases a tiny and cheap Amazon EC2 server instance will do the job — we’re talking monthly costs of around $20 or so. This will be enough to handle the initial load of most apps — and of course, if your app sells really well, the extra costs for upgrading the specs, or “scaling out” to more than one server will be peanuts to you, anyways ;).

Setting up the server is really simple. Most Linux servers you can rent or host, including EC2, already come with a database such as mySQL or PostgreSQL (we use the latter a lot, ourselves). All you need to do is install our Relativity Server, and you’re set. You can of course also use Azure, or other more traditional hosting solutions (and we are working on providing a ready-to-use EC2 image that will let you get set up with this even quicker, soon).

You can then design your database as you see fit, and Relativity Server lets you expose the data to your client applications — you decide what tables get exposed and who can access what, and you can write custom logic to control and influence data access and updates via JavaScript. The Relativity Server Explorer makes this easy and visual.

On the client side, Data Abstract is an easy-to-use static library that you can link into your application, and it will handle all the data access for you. The library is built on modern Cocoa principles, for example it uses GCD blocks to handle network access in the background, making it easy to keep your app responsive. Also, it is designed from the ground up for offline data, meaning you do not need to spend a lot of effort worrying about what happens when your user loses network connection — they can work with data while on a plane, and sync back up later when they get back online.

We have a lot of development topics around DA and Relativity server covered in our documentation Wiki, a good starting point is this page: Developing Database Applications for Mac and iOS.

In addition to providing all that Core Data/iCloud promises (but reliably), hosting your own data also provides additional benefits that cannot be achieved (by design) with iCloud data, even if it were working perfectly otherwise:

• In addition to each user having their own private data, your app can share data between users, where applicable — be it between individual users, or publicly with all users.
• You can build apps for other platforms, including Android, Windows, or Mac apps not sold in the Mac App Store, that share the same data. (Data Abstract has dedicated libraries for creating native apps for both Android and Windows, as well.)
• You can build a web front-end for users to view their data, either via server technologies such as ASP.NET, or client technologies such as a pure JavaScript based browser client. (Again, Data Abstract provides libraries for both, and our library for JavaScript is even completely free.)
• Most importantly, though, you as the app developer are in full control of the data. If one of your users calls you and says that something is wrong with their data, you’ll know exactly where to look.

Summary

I hope you found this article interesting, and that is has given you a fresh take on the intricacies of adding data “syncing” to your app — on iOS and elsewhere.

The key point i hope you will take a way from this is that Core Data in iCloud, being essentially a client-to-client sync mechanism, just like MobileMe and others before it, is fundamentally flawed, so that the myriad of problems many developers are seeing are hardly surprising. And that the solution to that is taking control of your own data, and hosting it with a single “truth in the cloud”.

I’d be lying if i said i didn’t hope you would consider using Data Abstract as your solution for this problem. We’re a small team of developers dedicated to this problem, and i believe we have a great and unique solution to it that provides a lot of benefit and can save you a lot of time and resources. Data Abstract sells as developer license, with no deployment costs or royalties.

Let us know what you think!

Whatever solution you choose, we wish you best of luck with your data-driven apps, and are looking forward to what you’ll produce!

Yours,
marc hoffman

Chief Architect,
RemObjects Software