Není k dispozici v češtině.
My application Reader‘s Diary is on Google Play store for more than 4 years. It was my first more complex application when I started self-learning programming. And you can guess that from some still remaining fragments of horrible code:) I wrote application primarily for myself, I use it whenever I start or finish reading a book. However, during these years it found few hundred active users. As my coding and UI skills improved, so has app, and now it is quite different from it‘s first version. Some poor initial design choices are however still there.
Planning new version
I was quite happy with the previous version. It fulfilled all my needs. Not taking small fix into account, it was in store for 18 months. In the meantime, I got some requests from users. Some wanted to sync their books, some wanted to add book cover pages. Showing cover images was an easy thing. I already have a function to search books by ISBN code using Google Books service so I only needed to save image url from the search result. For sync, I decided to use Firebase.
Firebase Auth
First thing to solve when adding app synchronization is user authentication. Fortunatelly, this can be easily done with Firebase Auth. Firebase Auth let you authenticate your users using Firebase server. There are many ways Firebase can authenticate your users. Most simple and obvious are Email/Password, Google and Facebook accounts but you can use even phone authentication via sms codes.
Firebase UI Auth
API is easy to use but you need some UI for it. Some login form with all these email/password, Google, Sign In, Sign Up buttons. If your design skills aren‘t perfect, results won‘t be perfect as well. Luckily, you don’t need to create UI yourself. There is Firebase Auth library that let you use finished and tested UI.
Using library is easy:
fun startFirebaseAuthForResult(activity: Activity) {
activity.startActivityForResult(
AuthUI.getInstance()
.createSignInIntentBuilder()
.setAvailableProviders(Arrays.asList(
AuthUI.IdpConfig.EmailBuilder().build(),
AuthUI.IdpConfig.GoogleBuilder().build()
))
.setIsSmartLockEnabled(false)
.setTosAndPrivacyPolicyUrls(TERMS_OF_USE, PRIVACY_POLICY_URL)
.setTheme(R.style.GreenTheme)
.setLogo(R.drawable.ic_launcher)
.build(),
FIREBASE_AUTH_REQUEST)
}
What is not so easy is to customize UI for your needs. You can use styles for this but it won‘t always work how you wish it to. You can see from my screenshot I was unable to change button text color to white without changing also color of other things on the Firebase UI screens.
Initial synchronization thoughts
Ok, we have authentication ready, now for main part: synchronization. Firebase offers two options: Firebase Realtime database and Cloud Firestore.
Store and sync data with our NoSQL cloud database. Data is synced across all clients in realtime, and remains available when your app goes offline.
The Firebase Realtime Database is a cloud-hosted database. Data is stored as JSON and synchronized in realtime to every connected client. When you build cross-platform apps with our iOS, Android, and JavaScript SDKs, all of your clients share one Realtime Database instance and automatically receive updates with the newest data.
Use our flexible, scalable NoSQL cloud database to store and sync data for client- and server-side development.
Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud Platform. Like Firebase Realtime Database, it keeps your data in sync across client apps through realtime listeners and offers offline support for mobile and web so you can build responsive apps that work regardless of network latency or Internet connectivity. Cloud Firestore also offers seamless integration with other Firebase and Google Cloud Platform products, including Cloud Functions.
You can read about differences between two databases here and here. Cloud Firestore is currently in beta and also there are much more projects using Realtime database, so I decided to use this. On the paper, synchronization with Firebase is pretty easy. You add value listener and wait for updates. And post things. You don‘t even need to care about storing data offline, Firebase will take care. This approach has two big BUTS. First, Firebase sends a lot of data. And charges for it. So if you are not careful something like this can happen to you. Second, you will surely get new data when you go online but there is no guarantee that posted data will really be sent to the server. So if you mustn’t lose any data, you need to implement some kind of local storage anyway.
Database Design
First, if you have app with offline database (typically SQLite), you might need to do some changes here.
Records need to be unique. In SQLite, there is _id column that will probably be primary key autoincrement. But this column is of little use if you need to sync data across devices and don‘t want to see many duplicates. Best solution is to add column with UUID.
A universally unique identifier (UUID) is a 128-bit number used to identify information in computer systems”
You can generate UUID on Android simply using command
UUID.randomUUID()
There is also an issue with deleting records. Reader‘s Diary app had only one table – books. If the user deleted book, it was simply deleted from the database. Not anymore with cloud backup. We need to let backend and clients know that record was deleted. I added other table called deleted containing just book UUID. When the book is deleted, it is first added to deleted table and only then deleted from main books table.
Second, you need some structure of your JSON tree on Firebase. I decided for this structure:
- UUID of user (provided by Firebase Auth)
- books
- deleted
There’s one more thing to think about: Database rules. You want to enable users only read their own books and also insert books only into their own space. There is Rules tab in Firebase database settings. You can set rules for reading, writing and data validations. To simply enable users writing and reading only their own records, you can use like this:
{
"rules": {
"$uid": {
"books": {
".read": "$uid == auth.uid",
".write": "$uid == auth.uid",
},
"deleted": {
".read": "$uid == auth.uid",
".write": "$uid == auth.uid",
}
}
}
}
Sync
I wanted sync to be simple and not too data use intensive. In order to stay on free Spark plan, I strove to upload and download as few data as possible. You can see Firebase plans comparison here. Basically, you can have 100 simultaneous connections, store 1GB of data and download 10 GB per month.
To save some database space, my Firebase record keys are just few letters long, not full variable names. This is how I did this for author variable in my data class:
@get: PropertyName("au") @set: PropertyName("au")var author: String = ""
I also wanted to minimize downloads and simultaneous connections. In order to do so, I decided not really use full Firebase Realtime capabilities. Instead of adding valueEventListener and observing changes while the application is running, sync is performed only when an application is opened or user swipes to refresh.
To download as few data as possible, only changed books are uploaded and downloaded on sync. Newly created or updated books are flagged dirty in database. When sync is started they are sent with sync start timestamp in milliseconds. This should be rightly set by server, Firebase Cloud functions could be used for it, but as there is limit 125000 method invocations per month (so 125000 books can be created or updated) that I don‘t feel is high enough if active users count rises, I decided to go with device time. Yes, the user can break his sync if he plays with device time but so far there was nobody reporting me sync problems.
So how do I sync?
I am using simple coroutine for this. First updated books are uploaded with the current timestamp. It doesn‘t really matter if a record is created or updated, UUID is used as child reference so a record is created or overridden. I set child value and addOnCompleteListener and onFailureListener. Inside onComplete listener successfully uploaded book‘s dirty flag is set to false and successCount is increased.
Then (or rather same time as all Firebase Realtime database functions are asynchronous with callbacks, more about it later) deleted books are deleted on Firebase. First null value is set for book object, effectively deleting it, second inside /deleted record is created containing deletion timestamp.
Then Firebase is queried for books updated after previous sync (saved timestamp when sync started). All updated books are received at the same time, saved to database (overridden if already exist) and valueEventListener is removed.
Similar happens for deleted books, these are also queried by timestamp and deleted from the local database.
There is one problem to solve. As I wrote before, all Firebase Realtime database functions are asynchronous with callbacks. So how do I know, that synchronization is finished? I have flags for the number of items to upload, number of items to delete, if download of books is finished and if download of deleted books is finished. Inside callbacks true is set for download finished flag or uploaded count is increased for items to be upload and then there is check if all conditions of finished sync are met. So order doesn’t really matter, it just works.
This is surely not the bulletproof way and skilled tester could break it, however, I think this is a good start for a small app or your Minimum viable product and it can be always improved later.
Looking back on statistics, I can see Firebase is clearly still sending a lot of data. Today (September 8th) only 1.5MB data is stored in the database but 4.3MB was downloaded since the beginning of the month. I am clearly not only one with this problem. If you know where the problem might be, please let me know.