wbrawner/server
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
46874 commits 157 branches 562 tags 714 MiB
Commit graph

74 commits

Author SHA1 Message Date
Robin Appelman
93de63777e
extend storage api to allow directly writing a stream to storage 
this removes the need for temporary storages with some external storage backends.
The new method is added to a separate interface to maintain compatibility with
storage backends implementing the storage interface directly (without inheriting common)

Currently the interface is implemented for objectstorage based storages and local storage
and used by webdav uploads

Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-10-31 21:10:44 +01:00
Robin Appelman
47fd3499a3
Fix listing s3 objects when using google cloud aws compatibility 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-06-11 17:10:57 +02:00
Robin Appelman
ac26175a17
add azure unit tests with azurite 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-06-04 16:18:03 +02:00
Robin Appelman
60b7cbc0f0
add azure object storage backend 
Signed-off-by: Robin Appelman <robin@icewind.nl>
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-06-04 10:57:52 +02:00
Roeland Jago Douma
a0e038f912
generateTokenFromCache is only valid for V3 
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-04-26 20:51:05 +02:00
Roeland Jago Douma
7318882357
json_decode an associative array for the swift cached token 
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-04-26 20:47:47 +02:00
Arthur Schiwon
38a90130ce
move log constants to ILogger 
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
 2018-04-26 10:45:52 +02:00
Roeland Jago Douma
1ceb081c9b
Merge pull request #9280 from nextcloud/bugfix/noid/swift-v3-scope 
Swift v3 requires setting a scope
 2018-04-24 09:49:27 +02:00
Roeland Jago Douma
ba71918ad2
Swift v3 requires setting a scope 
Unscoped auth is not guaranteed to return a catalog. So require the
scope to be set properly.

See https://developer.openstack.org/api-ref/identity/v3/index.html#password-authentication-with-unscoped-authorization

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-04-23 19:34:09 +02:00
Morris Jobke
431ccb6386
Fix undefined variable - found by phan 
Added in https://github.com/nextcloud/server/pull/8314/files#diff-a55c10804586cd797b7f3f16556c1457R157

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-04-23 11:34:41 +02:00
Morris Jobke
2f068bc9d1
Add more logging for the object storage during creation of the buckets 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-04-05 17:26:09 +02:00
Morris Jobke
63bc633d89
Merge pull request #8857 from nextcloud/swift-verify-cached-token 
verify cached swift token
 2018-03-19 10:10:04 +01:00
Robin Appelman
41954d2903
verify cached swift token 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-03-16 15:20:16 +01:00
Roeland Jago Douma
d3e7996e80
Fix bucket and container usage 
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-03-14 13:57:14 +01:00
Roeland Jago Douma
4e8d5504d2
Properly cache swift v3 token 
The V3 token has an export function. Without this the token is
effectively never cached.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-03-12 20:53:50 +01:00
Roeland Jago Douma
d05469959e
Merge pull request #8715 from nextcloud/objectstore-no-part-files 
disable part files for object stores
 2018-03-08 20:52:16 +01:00
Robin Appelman
fc35352122
disable part files for object stores 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-03-07 13:33:35 +01:00
Robin Appelman
9764e70494
don't read existing file when overwriting using object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-03-07 12:05:57 +01:00
Roeland Jago Douma
812ea354d0
User proper username + pass 
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-02-26 14:54:18 +01:00
Robin Appelman
31dd6d6aae
add support for v3 swift auth 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-26 14:54:08 +01:00
Robin Appelman
817f2f8f49
adjust swift external storage to new openstack sdk 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-19 19:41:16 +01:00
Robin Appelman
3192efc415
adjust swift object storage to new openstack sdk 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-19 19:41:16 +01:00
Morris Jobke
d3d045dd5c
Remove unused import statements 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-02-14 16:55:43 +01:00
Roeland Jago Douma
3e0ef7e3f2
Try to silence phan 
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
 2018-02-08 15:18:26 +01:00
Robin Appelman
e34c20af7d
don't swallow exception from touch 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-08 15:18:26 +01:00
Robin Appelman
85d23dd6fc
better error message on catalog error 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-08 15:18:26 +01:00
Morris Jobke
26e0c14c6d
Merge pull request #8033 from nextcloud/s3-uploader 
Use S3Client::upload instead of splitting single/multipart upload ourselves
 2018-02-08 14:30:02 +01:00
Robin Appelman
fd09c06eb4
actually return stream from swift 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-02-07 17:10:59 +01:00
Morris Jobke
a661f043e1
Remove unneeded semicolon and parentheses 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-01-26 23:46:40 +01:00
Morris Jobke
c005fc6755
Cleanup unused code 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-01-26 15:25:19 +01:00
Robin Appelman
3499cbdb76
Use S3Client::upload instead of splitting single/multipart upload ourselves 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2018-01-24 17:22:05 +01:00
Morris Jobke
2a38605545
Properly log the full exception instead of only the message 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2018-01-23 10:57:21 +01:00
Robin Appelman
34ced4dd97
add option to use legacy v2 auth with s3 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-12-19 11:54:55 +01:00
Morris Jobke
d2d73f1ce8
Also replace all other occurences 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2017-12-18 20:57:11 +01:00
Robin Appelman
046eb0c635
add retry wrapper when reading files from swift 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-12-04 16:34:53 +01:00
Bjoern Schiessle
1ade6b0817
only create the file cache entry after the empty file was created successfully, otherwise file_exists() call on the initial file_put_content() will indicate that the file already exists 
Signed-off-by: Bjoern Schiessle <bjoern@schiessle.org>
 2017-11-17 12:45:46 +01:00
Robin Appelman
c566b2854b
Use non empty files for object store touch 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-11-08 17:51:02 +01:00
Robin Appelman
70a4860ccb
Fix contructing headers for s3 download 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-11-08 17:50:46 +01:00
Morris Jobke
31c5c2a592
Change @georgehrke's email 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2017-11-06 20:38:59 +01:00
Morris Jobke
da6c2c9da1
Merge pull request #7083 from nextcloud/update-license-headers 
Update license headers
 2017-11-06 18:48:04 +01:00
Morris Jobke
0eebff152a
Update license headers 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2017-11-06 16:56:19 +01:00
Morris Jobke
8b763ea0f7
Better error message for invalid bucket names 
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2017-11-06 09:48:41 +01:00
Joas Schilling
3e0789aa94
Merge pull request #6958 from nextcloud/improved-mimetype-detection-for-object-storage 
Improve mimetype detection for object storages
 2017-11-01 11:04:15 +01:00
Robin Appelman
f9a7294807
use fopen directly when reading objects from s3 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-10-27 12:58:19 +02:00
Morris Jobke
405bbc1c61
Improve mimetype detection for object storages 
Object storage instances always fall back to the content based mimetype detection, because the file name for object storage was always random due to the fact that it was temporarily storage in a generated temp file. This patch adds a check before that to make sure to use the original file name for this purpose and also remove possible other extensions like the versioning or part file extension.

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
 2017-10-26 16:31:59 +02:00
Robin Appelman
91b3536f45
propagate multipart upload exception when aborting upload 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-10-17 15:10:58 +02:00
Robin Appelman
e393b3553e
set s3 part size to 500mb 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-10-11 15:59:53 +02:00
Robin Appelman
385d6f098c
Add tests for multipart upload 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-09-21 14:47:34 +02:00
Robin Appelman
4ae46d8876
only do multipart upload for large files 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-09-21 14:06:59 +02:00
Robin Appelman
e4e5e735db
multipart upload for s3 object storage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2017-09-20 17:51:58 +02:00