Approach 1: modifying the proxy server<\/h1>\n
The first Approach we considered was to modify the Swift Proxy Server in a way that can distinguish between the different types of data. The path to an object in Swift is The changes in code have been made in The test environment consists of three VMs, all of them are running:<\/p>\n The VMs are meant to simulate three different setups:<\/p>\n With tc qdisc we added the latency between the VMs to have some similar results as what we expect to be the round trip delay between these machines.<\/p>\n The way the tests were done is that from the device VM a HTTP request has been made with curl to first upload files of sizes from 10kb up to 100mb.<\/p>\n First the uploads – using a HTTP PUT with curl – were done without the remote indicator so that all the files are stored on the edge (blue line). Afterwards the uploads with the remote indicator were done, so that swift automatically pushes the file on to the cloud (red line). The object was retrieved using HTTP GET with curl to get each file from the edge and from the cloud alike.<\/p>\n As the graphs can quite clearly show this approach is slow and absolutely not imaginable in a production deployment. The scaling is very bad and nowhere near useful in a daily use. Both the put and the get are very slow, but the way the get scaled is so much worse than the upload that we had to scrape this idea, because we didn’t believe that optimizing our code would help much.<\/p>\n As the results from the modified proxy server show that this is an approach that is not viable we decided to try something more normal in the Swift universe, which is to write a middleware for Swift to handle the redirects. Middlewares in Swift are compareable to Lego pieces that can be plugged into the pipeline, which defines the codeflow of Swift. and can be used to add extra functionalities such as access controls, authentications, logging etc. The default pipeline of swift looks like this (it can be found in the configuration file of the proxy server)<\/p>\n When writing your own middleware you have to add your middleware in the pipeline and add the configurations for it. We advise you to add your The test setup was the same with the only difference being that the middleware is added to this Swift instead of having the proxy server modified. As graphs show the performance is significantly better than the previous approach.<\/p>\n The results for remote uploads scale more parallel to the local uploads than with the first approach which makes this approach more realistic to imagine being used in a daily scenario. Of course the performance could improve and the issue with authentication is not yet solved for this scenario, but we believe it to be the better solution, especially compared to the first approach.<\/p>\n The HTTP GET doesn’t scale as well as the PUT, the time gap between the remote and the local downloads gets bigger by the size of the objects. The download time for bigger files is significantly longer for remote files than for local files, yet still a lot better than the previous approach.<\/p>\n After spending quite some time looking into the inner workings of Swift, I can say we learnt a lot about Swift and how it works and handles requests. Also I believe we can say that the code of Swift should not be touched or changed in a deployed or even a test environment as it’s very intricate and has a lot of components. If you want to try to change some code in Swift we advise you to create a simple VM to test your changes and get comfortable with how Swift works (a tutorial to creating a Swift VM can be found on this blog very soon). As for functionality that you want to introduce in to the Swift code writing a Swift Middleware is highly advised. It will make your life a lot easier and it’s intended to be used in that way.<\/p>\n One thing that could be looked inot is a solution where Swift could take data and independently decide if the object is to be stored locally or remote without the user input or indicator. Another thing that would be good to look into, is the possibility to use metadata to label the objects instead of using indicators in the container names. The last issue that would need to be resolved is the authentication issue that at the moment doesn’t allow the redirect.<\/p>\n As the trend continues to move towards Serverless Computing, Edge Computing and Functions as a Service (FaaS), the need for a storage system that can adapt to these architectures grows ever bigger. In a scenario where smart cars have to make decisions on a whim, there is no chance for that car to ask a […]<\/p>\n","protected":false},"author":151,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[79,889,820,888,240,314,453],"features":[],"class_list":["post-11694","post","type-post","status-publish","format-standard","hentry","category-allgemein","tag-cloud","tag-edge","tag-faas","tag-object-storage","tag-openstack","tag-storage","tag-swift"],"yoast_head":"\nhttp:\/\/localhost:8080\/v1\/AUTH_test\/container\/object<\/code>, we used this to make the data distinguishable. At first we thought about adding metadata to an object when uploading it to identify where the object should be stored. This didn’t work for us as we had trouble parsing the metadata at the point where we implemented the distinction if the object is to be stored remote or local. So the approach was designed with the idea that if the container name contained an indicator that the object is to be pushed remote from the current Swift storage. This indicator was added as a prefix for the container name called 'remote_'<\/code> This was implemented in the proxy-server by parsing the path of the object and looking for the indicator. So a call to the remote Swift would look something like this: http:\/\/localhost:8080\/v1\/AUTH_test\/remote_container\/object<\/code><\/p>\n\/swift\/swift\/proxy\/server.py<\/code> which is the entry point to the whole Swift environment. The get_controller<\/code> method was the method where we implemented our changes, as there the controllers for the account, container and the object get instantiated and this is where the code needs to make the distinction before it pushes the object locally.<\/p>\nResults<\/h3>\n
\n
\n
![\"\"](\"https:\/\/blog.zhaw.ch\/icclab\/files\/2017\/11\/redirect-vm_arch.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/blog.zhaw.ch\/icclab\/files\/2017\/11\/pr_put.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/blog.zhaw.ch\/icclab\/files\/2017\/11\/pr_get.png\")
<\/a><\/p>\nApproach 2: write a middleware<\/h1>\n
[pipeline:main]\r\n# This sample pipeline uses tempauth and is used for SAIO dev work and\r\n# testing.\r\npipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit crossdomain authtoken keystoneauth tempauth formpost staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-serve<\/code><\/pre>\n.py<\/code> file which is the middleware in to the \/swift\/swift\/common\/middleware<\/code> directory. By doing this and adding the entrypoint of your middleware to the following file: \/swift\/swift.egg-info\/entry-points.txt<\/code> like this:<\/p>\nredirectmiddleware = swift.common.middleware.redirectmiddleware:redirect_factory<\/code><\/pre>\nResults<\/h3>\n
![\"\"](\"https:\/\/blog.zhaw.ch\/icclab\/files\/2017\/11\/mw_put.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/blog.zhaw.ch\/icclab\/files\/2017\/11\/mw_get.png\")
<\/a><\/p>\n <\/h1>\n
Lessons learnt<\/h1>\n
Future direction<\/h1>\n
<\/div>","protected":false},"excerpt":{"rendered":"