* Rewrite cache module in Kotlin.
* Migrate RealStore to use new cache.
* Migrate FileSystemImpl to use new cache.
* Migrate StoreTest to new cache.
* Remove Guava Cache.
* Migrate remaining Java file to Kotlin.
* Fix ktlint error.
* Fix package.
* Add support for CacheLoader.
* Remove unnecessary mockito-inline.
* Remove volatile keyword from CacheEntry fields and add docs re. cache entry reuse and thread-safty assumption.
* - Refactor CacheBuilder.
- Update docs.
- Update Key, Value, Input, Output generic types to be bound by Any.
- Replace spy with TestLoader for asserting invocation.
* Support key-based synchronization for get(key: Key, loader: () -> Value): Value.
* Document loader exception propagation.
* Small clean ups.
* Make CacheEntry's Key and Value bound to Any, move CacheEntry to RealCache.kt as a private class.
* Add reference counting key-based synchronization for cache loader.
* Improve KeyedSynchronizer and docs.
* Refactor tests and add a missing test case to reach 100% coverage.
* Add memory barrier after updating access time in cache entry.
* Only process access queue if expiresAfterAccess == true.
* Support configurable concurrency level.
* Increase delay for concurrency test.
* Explicitly set concurrency level for concurrency tests.
* Only add memory barrier after recordRead if expiresAfterAccess is enabled.
* Fix a potential memory leak caused by thread preemption.
* Allow retries (3 times) for concurrency tests.
* Move performMaintenance function to bottom of the class.
* Fix typos, rename performMaintenance() to cleanUpDeadEntries(), improve docs.
* Fix more typos.
* Only remove entries in map but not in queue.
* Replace memory barrier with @Volatile fields in CacheEntry.
* Fix deadlock while iterating sets.
* Remove stale cache/README.md, remove commented out java file.
* Add kotlin plugin to cache module
* Remove custom accessors in StoreDefaults as default values are effectively constant.
* Update JUnit to 4.13 RC2 to enable asserting specific error message on expected exception.
* Remove custom accessors in MemoryPolicy as all values are calculated from constants.
* Remove checkstyle and pmd as the codebase will be 100% Kotlin.
* Fix IDE warning in build.gradle by removing static imports.
* Remove unused guava dependencies.
* Improve StoreDefaults docs.
* Make MemoryPolicy Kotlin friendlier
* Make MemoryPolicy usage more Kotlin like
* Format code in FSReader
* Format code in FSWriter
* Format code in FileSystemRecordPersister
* Format code in FileSystemPersister
* Convert BarCodeReadAllPathResolver to Object instead of Class
* Convert BarCodePathResolver to Object instead of Class
* Fix wrong control flow conversion
* Clean up ChannelManager's interface to make it easier to add more tests (and add them)
Main changes:
* ~`ChannelManager` no longer inherits from `StoreRealActor` but rather
delegates to one.~ - Moved to followup PR
* messages coming from the upstream were placed under `Message.Dispach`
rather than `Message`. These are the only messages accepted from outside
`ChannelManager`
* `ChannelManager` now only exposes `plusAsign`, `minusAsign` for
adding/removing channels, close for closing and send(msg, Message.Dispatch)
for upstream events
* `SharedFlowProducer` no longer has a back dependency on
`ChannelManager`, rather it accepts a `suspend (Message.Dispatch) ->
Unit`
* `ChannelManager` is now built with the upstream flow rather than a
flow factory. Given that a flow is stateless, passing in a flow that can
be re-consumed seems like the simpler API.
* New `ChannelManager` tests
* make changes to existing test minimal
* revert actor delegation to reduce PR size
* comments
* clean up tests
* Bump travis
* lint
* rename `channelManagerInbox` ->``sendUpsteamMessage`
* Fix multiple collections on Multicast
Multicast implementation had a bug where the returned flow could
not be collected multiple times as it was using the same channel
it created when was called.
This PR changes it to create per collection to avoid this issue.
I've also replaced function with a field as there
is no reason to keep creating a new one, it can be just a flow
Test: MultiplexTest#multipleCollections
Fixes: #26
* remove create function
* Cleanup StoreBuilder interface and add some documentation
* Update store/src/main/java/com/dropbox/android/external/store4/StoreBuilder.kt
Co-Authored-By: Mike Nakhimovich <digitalbuddha@users.noreply.github.com>
* A little more docs around persistence
* add dokka for store module
move key parser to tests
* remove parser exception
* move RealStore and SourceOfTruth to internal
* move cache type to internal
* move Clearable to tests
* move fetcher to tests
* move multiplexer to internal
* move map indexed to internal
* random cleanup.
Fixed a bunch of warnings.
Moved SimplePersisterAsFlowable into tests as it is not used anymore but
FlowTest uses it.
Moved MemoryPolicyBuilder test to kotlin to make DEFAULT_POLICY a const
* massive file migration
Rename pipeline tests into impl tests
This PR fixes a bug in StoreRealActor where we could call onClose
while actor is processing some messages. Now instead we send a token
to close and inside the message handler we close the channel so that
no new messages can arrive after close meanwhile messages that arrived
before that close is handled properly.
Also set version to 4.0.0 and cleaned up some code.
Fixes#55
* remove suspend cache
This PR replaces suspend cahce usages w/ direct guava cache usage.
I think we should also get rid of guava cache.
First of all it is in java so needs to be moved to kotlin at least for
future multi-platform support.
Second, it would be nicer to have something that uses kotlin's time so
that people can provide a scope w/ a delay functionality (like the
TestCoroutineScope) and also test time related stuff. An alternative might
be letting people pass a Timer in the builder
Fixes#41
* remove commented code :/
This PR implements piggybacking in fetch controller which means even if
upstream closes, we never close downstream flows in fetcher multiplexing.
Instead, we keep them around and if a new request comes in, they'll receive
the same data as well or even keep the upstream open.
This is implemented as a feature of multiplexer as it already has all of the
machinery to do such re-distribution of data.
It is a bit questionable behavior for multiplexer but it is already very
specific to our use case so I thought it might be fine to do it there
* WIP publish test
* add actor based implementation, seems the most promising
* add notes into channel manager as well
* use unlimited channel on the receiver to avoid launching to send
* carry over remaining subscribers into a new flow
* dispatch errors from upstream to all downstreams
* carry over all leftovers at once to avoid starting producer before all is added to the list
* handle swapping channel managers in the consumer
This CL fixes an issue where we wouldn't unsubscribe from the right channel if
the downstream is moved between channel managers due to not receiving any event
after registering.
I've also cleaned up dispatchError to close the channel with error instead of
passing it down as if it is value and throwing again
* allow live buffering
this adds a live buffering functionality to actor publish where
it only buffers if the upstream is still running
* remove logging
* move into src
* code cleanup, more comments
* first shot at new internal store, tests pass, code ugly
* tmp builder for real internal store, starting pipeline tests
* more wip in fixing pipeline store tests, a lot to cleanup
* all tests pass
* code cleanup
* move multiplexer inside store4
* lots of cleanup of unnecessary code
* release barriers that are not used
* don't use pipeline persister.
also fixed a barrier cleanup code in source of truth with barrier
* revert simple persister as flowable change
cleanup for clearstorememorytest
* close multiplexers in fetcher controller when not used
* code style fixes
This CL avoids having two different codepaths based on whether
we skip disk cache or not. It should eventually make it easier
to implement sharing, if we decide to do it here
* Introduce StoreResponse
This CL changes Pipeline API to dispatch StoreResponse instead of Output.
It will allow us to add more metadata into it and also handle things like
source attribution and/or loading state.
This CL only makes the tests pass, does not do anything for actually dispatching
loading state
* first shot at adding a loading state
This CL adds proper StoreResponse to the built in pipelines such
that they can merge information between network and disk.
It needs some cleanup and PipelinePersister's merging logic is
fairly complicated but i couldn't yet find a simpler way to achieve it
* Add more tests to pipeline
Also fixed a bug in cached where it would emit cached value as
Loading even if no refresh is requested
* fix room sample after rebase
* Update the room sample to support refresh
Also some code cleanup in PipelinePersister
* code cleanup
* explain network + disk flow
* Change StoreResponse for plan D
This CL changes store response to act like plan D described here:
https://paper.dropbox.com/doc/Streaming-Data-How-To--AkwrNu0A7\~BikNOP2m20\~R2_AQ-oLBV39GrNimqGDe1vNQMb
Now StoreResponse has 3 types: Data, Error, Loading and each of these have an field.
Loading and Error does not carry over last data anymore, instead, it is expected downstream
to merge them for UI, if desired.
This change makes Store more verbose but i think it is the right direction to allow
developer to expand on it as desired
* update AGP to 3.6-alpha11
* update pipeline store tests to check finite streams
This CL updates PipelineStoreTest to consume the whole stream
when we expect it to finish rather than limiting the number of
dispatched items
* update RoomActivity to respect new StoreResponse
* add search to room activity
* code cleanup
* Updated coroutines to RC02 and fix Flow invariance
This PR fixes an invariance problem in side collect where it was
creating a scope and emitting from there.
* Fix KeyTracker not to lose emissions
KeyTracker would lose some emissions if they happen in rapid succession
between different keys. This new implementation maintains a separate
channel for each key to avoid that. Since we now create a channel per key,
we also need to track and clean them up to avoid consuming too much memory
This PR fixes an issue in NoOpPersister where it was using a CoroutineScope w/o
really needing one as there wasn't any actual deferred work going on. As a side
effect of that, it was flaking the StoreTest.
After this change, the test does not flake aynmore (tried by setting test to be re-run
until failure
* try delegaitng get calls to steam
* WIP handle null
This CL changes pipeline persister to handle null reader response to fetch.
This is WIP, need another fix before it can be used
* move StoreTest and StoreWithParserTest to coroutines testing lib
The test coroutine scope checks for any remaining jobs at the end of a test
which helped us find two bugs (#17, #18) both of which were discovered when
i tried to move this tests to the TestCoroutineScope after the streamOnly
change.
This change completes the move to avoid further regressions. Eventually,
we should move all tests
* remove unused get method from PipelineStore
* Fix KeyTracker subscription order
This PR fixes a race condition where if the SimplePersisterAsFlowable's write
is called inside a reader, reader wouldn't be notified again
* Fix StreamOneKeyTest
This test was passing by mistake :/. Right now, when we create the backport
Store from a Pipeline, the method skips memory cache but uses
disk cache. This test was not initializing the disk cache w/ empty value,
which meant the initial read would get first value as if it is cached and
then would also get the new value.'
The test didn't fail before because KeyTracker had a concurrency but where
it missed 1 invalidation call
This PR fixes a bug in side-collect where it was not cancelling the side
collection when main flow stops.
It still does not handle the case where side collect fails but that can only be fixed when we have error handling.
Needed to move the method to a non-private place to be able to write tests :/
StoreRequest to encapsulate the get/stream into an object that can be passed down/modified/scaled.
The suspend cache APIs became a bit weird due to supporting both implementations. We can clean it up once we have only 1 implementation.
This PR adds a cache implementation that has all APIs as suspend functions.
It uses guava's cache under the hood and handles query deduplication in a custom
Entry type that we keep in the cache
* AGP 3.5.0-beta05
* Inflight as constructor argument
* New builder class
* Rename addPersister to persister
* Rename addParser to parser
* Rename addCache to cache
* AGP 3.5.0-beta04
* Stream implementation defined in the interface
* New class StoreWrapper
* Delete StoreBuilder
* Rename plus methods
* Move stalePolicy to Persister
* Simplify code
* Move inflight request management to a new class
* Move cache management to a new class
* Move wrappers to new package
* Delete StoreWrapper class
* New test
* Restored old versions
* Restored old versions
* Restored old versions
* Restored old versions
* Restored old versions
* Restored old versions
* Add runCatching to ignore exceptions on persisters
* Backward compatible RealStoreBuilder implementation
* Fix tests
* roomStore
* kt the app class
* refactor
* fix compilation
* Update .travis.yml
* migrate relevant tests to RoomStore, mark new Room API with @Experimental
* pr feedback
* bump minor rev up
* cleanup store
* create factory, clean up example
* pr comments
* Set missing expire-after-access policy to NoopPersister's cache (#312)
* Remove comment about PMD from NoopPersister#create()
* Add regression test for #312
* Deprecate MemoryPolicy#isDefaultPolicy() in favor of isDefaultWritePolicy() and isDefaultAccessPolicy()
