wbrawner/Store
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update README.md

Browse source
This commit is contained in:
Mike Nakhimovich 2017-02-10 12:17:28 -05:00 committed by GitHub
parent a8293a5994
commit 5ed868ba7a
1 changed files with 87 additions and 72 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

151
README.md
View file

@ -3,7 +3,7 @@
# Store # Store
Android library for async data loading from multiple sources. NOTE: While this library is production ready it is still in early stages and some apis may be renamed/changed. Android library for async data loading from network or 2 levels of caching. NOTE: While this library is production ready it is still in early stages and some apis may be renamed/changed.
### The Problems: ### The Problems:
@ -17,66 +17,66 @@ Store provides a level of abstraction between our UI elements and data operation
### Overview ### Overview
A Store is responsible for managing a particular data request in an application. When you create an implementation of a Store, you provide it with a Fetcher<link>. Additionally, you can define how your Store will cache data in-memory and on-disk, as well as how to parse it. Since you'll be getting back an Observable of your data, threading is a breeze! Once a store is built, it will handle the logic around data flow, allowing your views to use the best data source and ensuring that the newest data is always available for later offline use. Stores can be customized to work with your own implementations or use our included middleware. A Store is responsible for managing a particular data request in an application. When you create an implementation of a Store, you provide it with a `Fetcher`. Additionally, you can define how your Store will cache data in-memory and on-disk, as well as how to parse it. Since you'll be getting back an Observable of your data, threading is a breeze! Once a store is built, it will handle the logic around data flow, allowing your views to use the best data source and ensuring that the newest data is always available for later offline use. Stores can be customized to work with your own implementations or use our included middleware.
Store leverages RxJava and multiple request throttling to prevent excessive calls to the network and disk cache. By utilizing our library, you eliminate the possibility of flooding your network with the same request while adding 2 layers of caching (memory + disk). Store leverages RxJava and multiple request throttling to prevent excessive calls to the network and disk cache. By utilizing our library, you eliminate the possibility of flooding your network with the same request while adding 2 layers of caching (memory + disk).
### Fully Configured Store ### Fully Configured Store
Let's start by looking at what a fully configured store looks like, we will then walk through simpler examples building up functionality: Let's start by looking at what a fully configured store looks like, we will then walk through simpler examples building up functionality:
```java ```java
Store<Foo> store = ParsingStoreBuilder.<BufferedSource, String>builder() Store<ArticleAsset, Integer> articleStore = StoreBuilder.<Integer, BufferedSource, ArticleAsset>parsedWithKey()
.fetcher(this::ResponseAsSource) //OkHttp responseBody.source() .fetcher(articleId -> api.getArticleAsBufferedSource(articleId)) //OkHttp responseBody.source()
.persister(SourcePersisterFactory.create(context.getFilesDir()) .persister(FileSystemPersister.create(FileSystemFactory.create(context.getFilesDir())))
.parser(GsonParserFactory.createSourceParser(gson, Foo.class)) .parser(GsonParserFactory.createSourceParser(gson, ArticleAsset.Article.class))
.open(); .open();
``` ```
With the above setup you have: With the above setup you have:
+ In Memory Caching + In Memory Caching for rotation
+ Disk caching + Disk caching for when you're offline
+ Parsing through streaming api + Parsing through streaming api to keep memory consumption to a minimum
+ Ability to get cached data or bust through your caches + Rich API to ask for data whether you want cached/new or a stream of future data updates.
And now for the details: And now for the details:
### Creating a Store ### Creating a Store
Create a store using a builder, the only requirement is to include a `.fetcher()`. Create a store using a builder, the only requirement is to include a `.Fetcher<ReturnType,KeyType>` that returns an Observable<ReturnType> and has a single method `fetch(key)`
``` java ``` java
Store<Article> articleStore = StoreBuilder.<String>builder() Store<ArticleAsset, Integer> store = StoreBuilder.<ArticleAsset,Integer>key()
.nonObservableFetcher(barCode -> api.getArticle(barcode.getKey())) .fetcher(articleId -> api.getArticle(articleId)) //OkHttp responseBody.source()
.open();
Store<Article> articleStore = StoreBuilder.<String>builder()
.fetcher(barCode -> retrofitApi.getArticleObservable(barcode.getKey()))
.open(); .open();
``` ```
Stores use generic keys as identifiers for data. A Key can be any value object that properly implements toString and equals and hashCode. When your Fetcher function is called, it will be passed a particular Key value. Similarly, the key will be used as a primary identifier within caches (Make sure to have a proper hashCode!!)
### Barcodes ### Our Key implementation - Barcodes
For convenience we included our own key implementation called a BarCode. Barcode has 2 fields `String key and String type`
``` java ``` java
BarCode barcode = new BarCode("Article", "42"); BarCode barcode = new BarCode("Article", "42");
``` ```
When using a Barcode as your key, you can use a StoreBuilder convenience method
Stores use BarCodes as identifiers for data. A BarCode is a class that holds two strings, type and value. The two values act as unique identifiers for your data. When your Fetcher function is called, it will be passed the BarCode. Similarly, the barcode will be used as a key in your cache(s). ``` java
Store<ArticleAsset, Integer> store = StoreBuilder.<ArticleAsset>barcode()
.fetcher(articleBarcode -> api.getAsset(articleBarcode.getKey(),articleBarcode.getType()))
.open();
```
### Public Interfaces for Accessing Data - Get, Fetch, Stream
### Public Interface - Get, Fetch, Stream, GetRefreshing
```java ```java
Observable<Article> article = store.get(barCode); Observable<Article> article = store.get(barCode);
``` ```
The first time you subscribe to `store.get(barCode)`, the response will be stored in an in-memory cache using the BarCode as a key. All subsequent calls to `store.get(barCode)` will retrieve the cached version of the data, minimizing unnecessary data calls. This prevents your app from fetching fresh data over the network (or from another external data source) in situations when doing so would unnecessarily waste bandwidth and battery. A great use case is any time your views get recreated after a rotation, they will be able to request the cached data from your store. Having your data available has helped us retain less without or view layer The first time you subscribe to `store.get(barCode)`, the response will be stored in an in-memory cache. All subsequent calls to `store.get(barCode)` with the same Key will retrieve the cached version of the data, minimizing unnecessary data calls. This prevents your app from fetching fresh data over the network (or from another external data source) in situations when doing so would unnecessarily waste bandwidth and battery. A great use case is any time your views get recreated after a rotation, they will be able to request the cached data from your store. Having your data available has helped us retain less without or view layer
So far our Store’s data flow looks like this: So far our Store’s data flow looks like this:
![Simple Store Flow](https://github.com/nytm/Store/blob/master/Images/store-1.jpg) ![Simple Store Flow](https://github.com/nytm/Store/blob/master/Images/store-1.jpg)
`store.get()` -> return memory cached version if exists, otherwise -> fetch new networkResponse -> save in memory -> return newly cached response from memory
By default 100 items will be cached in memory for 24 hours. You may pass in your own instance of a Guava Cache to override the default policy. By default 100 items will be cached in memory for 24 hours. You may pass in your own instance of a Guava Cache to override the default policy.
@ -100,6 +100,9 @@ Calls to both `fetch()` and `get()` emit one value and then call `onCompleted()`
### Stream ### Stream
You may also call `store.stream()` which returns an Observable that emits each time a new item was added to the store. Think of stream as an Event Bus-like feature that allows you to know when any new network hits happen for a particular store. You can leverage the Rx operator `filter()` to only subscribe to a subset of emissions. You may also call `store.stream()` which returns an Observable that emits each time a new item was added to the store. Think of stream as an Event Bus-like feature that allows you to know when any new network hits happen for a particular store. You can leverage the Rx operator `filter()` to only subscribe to a subset of emissions.
### Get Refreshing
There is 1 more special way to subscribe to a Store - getRefreshing(key). Get Refreshing will subscribe to get() which returns a single response, unlike Get, Get Refreshing will stay subscribed. Anytime you call store.clear(key) anyone subscribe to getRefreshing(key) will resubscribe and force a new network response.
### Inflight Debouncer ### Inflight Debouncer
@ -108,11 +111,11 @@ There is an inflight debouncer as well to prevent duplicative requests for the s
### Adding a Parser ### Adding a Parser
Since it is rare that data comes from the network in the format that your views need, Stores can delegate to a parser. by using a `ParsingStoreBuilder<T,V>` rather than a `StoreBuilder<T>.` ParsingStoreBuilder has an additional method `parser()` which can take a Parser<Raw, Parsed> Since it is rare that data comes from the network in the format that your views need, Stores can delegate to a parser. by using a `StoreBuilder.<BarCode, BufferedSource, Article>parsedWithKey()
```java ```java
Store<Article> store = ParsingStoreBuilder.<BufferedSource, String>builder() Store<Article,Integer> store = StoreBuilder.<Integer, BufferedSource, Article>parsedWithKey()
.nonObservableFetcher(barCode -> source) //OkHttp responseBody.source() .fetcher(articleId -> api.getArticle(articleId))
.parser(source -> { .parser(source -> {
try (InputStreamReader reader = new InputStreamReader(source.inputStream())) { try (InputStreamReader reader = new InputStreamReader(source.inputStream())) {
return gson.fromJson(reader, Article.class); return gson.fromJson(reader, Article.class);
@ -131,7 +134,7 @@ Our updated data flow now looks like this:
### Middleware - GsonSourceParser ### Middleware - GsonSourceParser
There is also a seperate middleware lib with parsers to help in cases where your fetcher is a Reader, BufferedSource or String and your parser is Gson: There are also seperate middleware libraries with parsers to help in cases where your fetcher is a Reader, BufferedSource or String and your parser is Gson:
- GsonReaderParser - GsonReaderParser
- GsonSourceParser - GsonSourceParser
- GsonStringParser - GsonStringParser
@ -140,18 +143,21 @@ These can be accessed via a Factory class (GsonParserFactory).
Our example can now be rewritten as: Our example can now be rewritten as:
```java ```java
Store<Article> store = ParsingStoreBuilder.<BufferedSource, Article>builder() Store<Article,Integer> store = StoreBuilder.<Integer, BufferedSource, Article>parsedWithKey()
.nonObservableFetcher(this::getResponse) .fetcher(articleId -> api.getArticle(articleId))
.parser(GsonParserFactory.createSourceParser(gson, Article.class)) .parser(GsonParserFactory.createSourceParser(gson, Article.class))
.open(); .open();
``` ```
In some cases you may need to parse a top level JSONArray, in which case you can provide a TypeToken. In some cases you may need to parse a top level JSONArray, in which case you can provide a TypeToken.
```java ```java
Store<List<Article>> store = ParsingStoreBuilder.<BufferedSource, List<Article>>builder() Store<List<Article>,Integer> store = StoreBuilder.<Integer, BufferedSource, List<Article>>parsedWithKey()
.nonObservableFetcher(this::getResponse) .fetcher(articleId -> api.getArticles())
.parser(GsonParserFactory.createSourceParser(gson, new TypeToken<List<Article>>() {})) .parser(GsonParserFactory.createSourceParser(gson, new TypeToken<List<Article>>() {}))
.open(); .open();
Similar we have a middleware artifact for Moshi & Jackson too!
``` ```
### Disk Caching ### Disk Caching
@ -167,13 +173,13 @@ Now our data flow looks like:
Ideally, data will be streamed from network to disk using either a BufferedSource or Reader as your network raw type (rather than String). Ideally, data will be streamed from network to disk using either a BufferedSource or Reader as your network raw type (rather than String).
```java ```java
Store<String> store = ParsingStoreBuilder.<BufferedSource, String>builder() Store<Article,Integer> store = StoreBuilder.<Integer, BufferedSource, Article>parsedWithKey()
.nonObservableFetcher(this::ResponseAsSource) //OkHttp responseBody.source() .fetcher(articleId -> api.getArticles())
.persister(new Persister<BufferedSource>() { .persister(new Persister<BufferedSource>() {
@Override @Override
public Observable<BufferedSource> read(BarCode barCode) { public Observable<BufferedSource> read(Integer key) {
if (dataIsCached) { if (dataIsCached) {
return Observable.fromCallable(() -> userImplementedCache.get(barCode)); return Observable.fromCallable(() -> userImplementedCache.get(key));
} else { } else {
return Observable.empty(); return Observable.empty();
} }
@ -181,11 +187,11 @@ Store<String> store = ParsingStoreBuilder.<BufferedSource, String>builder()
@Override @Override
public Observable<Boolean> write(BarCode barCode, BufferedSource source) { public Observable<Boolean> write(BarCode barCode, BufferedSource source) {
userImplementedCache.save(barCode, source); userImplementedCache.save(key, source);
return Observable.just(true); return Observable.just(true);
} }
}) })
.parser(GsonParserFactory.createSourceParser(gson, String.class)) .parser(GsonParserFactory.createSourceParser(gson, Article.class))
.open(); .open();
``` ```
@ -199,12 +205,12 @@ If using SQLite we recommend working with SqlBrite. If you are not using SqlBrit
### Middleware - SourcePersister & FileSystem ### Middleware - SourcePersister & FileSystem
We've found the fastest form of persistence is streaming network responses directly to disk. As a result, we have included a seperate lib with a reactive FileSystem which depends on Okio BufferedSources. We have also included a SourcePersister which will give you disk caching and works beautifully with GsonSourceParser. Now we are back to our first example: We've found the fastest form of persistence is streaming network responses directly to disk. As a result, we have included a seperate lib with a reactive FileSystem which depends on Okio BufferedSources. We have also included a FileSystemPersister which will give you disk caching and works beautifully with GsonSourceParser. Now we are back to our first example:
```java ```java
Store<String> store = ParsingStoreBuilder.<BufferedSource, String>builder() Store<Article,Integer> store = StoreBuilder.<Integer, BufferedSource, Article>parsedWithKey()
.nonObservableFetcher(this::ResponseAsSource) //OkHttp responseBody.source() .fetcher(articleId -> api.getArticles(articleId))
.persister(SourcePersisterFactory.create(context.getFilesDir())) .persister(FileSystemPersister.create(FileSystemFactory.create(context.getFilesDir())))
.parser(GsonParserFactory.createSourceParser(gson, String.class)) .parser(GsonParserFactory.createSourceParser(gson, String.class))
.open(); .open();
``` ```
@ -212,43 +218,52 @@ Store<String> store = ParsingStoreBuilder.<BufferedSource, String>builder()
As mentioned, the above builder is how we work with network operations at New York Times. With the above setup you have: As mentioned, the above builder is how we work with network operations at New York Times. With the above setup you have:
+ Memory caching with Guava Cache + Memory caching with Guava Cache
+ Disk caching with FileSystem (you can reuse the same file system impl for all stores) + Disk caching with FileSystem (you can reuse the same file system impl for all stores)
+ Parsing from a BufferedSource to a <T> (String in our case) with Gson + Parsing from a BufferedSource to a <T> (Article in our case) with Gson
+ in-flight request management + in-flight request management
+ Ability to get cached data or bust through your caches (get vs fresh) + Ability to get cached data or bust through your caches (get vs fresh)
+ Ability to listen for any new emissions from network (stream)
+ Ability to be notified and resubscribed when caches are cleared,nice for when you need to do a post request and update another screen (getRefreshing)
We recommend using the above setup of the builder for most Stores. The SourcePersister implementation has a tiny memory footprint as it will stream bytes from network to disk and then from disk to parser. The streaming nature of our stores allows us to download dozens of 1mb+ json responses without worrying about OOM on low-memory devices. As mentioned above, Stores allow us to do things like calling `configStore.get()` a dozen times asynchronously before our Main Activity finishes loading without blocking the main thread or flooding our network. We recommend using the above setup of the builder for most Stores. The SourcePersister implementation has a tiny memory footprint as it will stream bytes from network to disk and then from disk to parser. The streaming nature of our stores allows us to download dozens of 1mb+ json responses without worrying about OOM on low-memory devices. As mentioned above, Stores allow us to do things like calling `configStore.get()` a dozen times asynchronously before our Main Activity finishes loading without blocking the main thread or flooding our network.
### RecordProvider
If you'd like your store to be aware of the staleness of disk data you can have your `Persister` implement `RecordProvider`. After doing so you can configure your Store to work in one of two ways:
store = StoreBuilder.<BufferedSource>barcode()
.fetcher(fetcher)
.persister(persister)
.refreshOnStale()
.open();
refreshOnStale - will backfill the disk cache anytime a record is stale, User will still get the stale record returned to them
Or alternatively
store = StoreBuilder.<BufferedSource>barcode()
.fetcher(fetcher)
.persister(persister)
.networkBeforeStale()
.open();
networkBeforeStale - Store will try to get network source when disk data is stale. if network source throws error or is empty, stale disk data will be returned
### Subclassing a Store ### Subclassing a Store
We can also subclass a Store implementation (RealStore<T>): We can also subclass a Store implementation (RealStore<T>):
```java ```java
public class SampleStore extends RealStore<String> { public class SampleStore extends RealStore<String, BarCode> {
public SampleStore(Fetcher<String> f, Persister<String> p) { public SampleStore(Fetcher<String, BarCode> fetcher, Persister<String, BarCode> persister) {
super(f, p); super(fetcher, persister);
} }
} }
```
Or with a parser:
```java
public class SampleStore extends RealStore<String> {
public SampleStore(Fetcher<BufferedSource> fetcher,
Persister<BufferedSource> persister,
Parser<BufferedSource,String> parser) {
super(fetcher, persister, parser);
}
}
```
Subclassing is useful for when you’d like to inject Store dependencies or add a few helper methods to a store: Subclassing is useful for when you’d like to inject Store dependencies or add a few helper methods to a store:
```java ```java
public class SampleStore extends RealStore<String> { public class SampleStore extends RealStore<String, BarCode> {
@Inject @Inject
public SampleStore(Fetcher<String> f, Persister<String> p) { public SampleStore(Fetcher<String, BarCode> fetcher, Persister<String, BarCode> persister) {
super(f, p); super(fetcher, persister);
} }
} }
``` ```
@ -257,9 +272,9 @@ public class SampleStore extends RealStore<String> {
### Artifacts ### Artifacts
Note: Release is in Sync with current state of master (not develop) branch Note: Release is in Sync with current state of master (not develop) branch
CurrentVersion = **1.0.7** CurrentVersion = **2.0.0**
+ **Cache** Cache extracted from Guava (~200 methods) + **Cache** Cache extracted from Guava (keeps method count to a minimum)
```groovy ```groovy
compile 'com.nytimes.android:cache:CurrentVersion' compile 'com.nytimes.android:cache:CurrentVersion'
@ -269,17 +284,17 @@ CurrentVersion = **1.0.7**
```groovy ```groovy
compile 'com.nytimes.android:store:CurrentVersion' compile 'com.nytimes.android:store:CurrentVersion'
``` ```
+ **Middleware** Sample gson parsers, (feel free to create more and open PRs) + **Middleware** Sample Gson parsers, (feel free to create more and open PRs)
```groovy ```groovy
compile 'com.nytimes.android:middleware:CurrentVersion' compile 'com.nytimes.android:middleware:CurrentVersion'
``` ```
+ **Middleware-Jackson** Sample gson parsers, (feel free to create more and open PRs) + **Middleware-Jackson** Sample Jackon parsers, (feel free to create more and open PRs)
```groovy ```groovy
compile 'com.nytimes.android:middleware:-jackson:CurrentVersion' compile 'com.nytimes.android:middleware:-jackson:CurrentVersion'
``` ```
+ **Middleware-Moshi** Sample gson parsers, (feel free to create more and open PRs) + **Middleware-Moshi** Sample Moshi parsers, (feel free to create more and open PRs)
```groovy ```groovy
compile 'com.nytimes.android:middleware-moshi:CurrentVersion' compile 'com.nytimes.android:middleware-moshi:CurrentVersion'