Use case assessment report M24


This report entails the monitoring and assessment of the various use cases in the project. These evaluations were and will be continuously conducted in line with the methodology agreed by all involved parties in D13.1 “Use case evaluation methodology M24”. The evaluations are being executed by multidisciplinary teams from business and RTD providers from the different use cases. People involved in the development of the use cases and a group of experts external to the consortium participated and will continuously do so in the evaluation activities.

It must be noted that with respect to the status of both the ARTIST methodology and prototypes’ maturity at the moment of conducting the evaluation, the use case providers decided to engage engineers directly involved in the project. Even though this internal evaluation restricted the activities to analysing specific evaluation indicators, namely usability, usefulness and reliability, eventually it can improve the technical feedback produced, since these teams are quite familiarized with the use case, the methodology and the tools.

At the same time, in order to ensure that independent feedback will also be aggregated and no fundamental issues have been left out, the ARTIST consortium formed a structured process, a feedback loop to the consortium with a community outside the project for re-evaluation. This approach on the one hand helps for justification and traceability of the findings and on the other hand for dissemination of the results to an expert community. Thus, the Special Interest Group (SIG) that has been formulated very early in the project and had already participated in several ARTIST-related activities (as reported in D4.3.1 “Dissemination report M12”), was activated in ARTIST evaluations and validations as well. SIG is a group of experts from diverse research groups, organizations and companies that was formed to: a) be aware of what ARTIST is about, b) act as an advisory board that provide ARTIST independent feedback that enriches/improves ARTIST results and achievements, c) help ARTIST in broadcasting more its results and d) continuously evaluate ARTIST work and provide if possible potential solutions to problems and obstacles faced during project’s lifecycle.

In order to ensure actionable feedback, the assessments started early and iterate in line with the major milestones of the project. During the 2nd reporting period, the evaluation team concluded their activities at this phase by focusing on M20 release of methodology and tooling. It must be mentioned that the main focus of the evaluation was on the tools that support the two phases of ARTIST migration methodology, the pre-migration and migration phases. By applying the ARTIST tools classified to these two phases as being provided by the technical team, the use case providers developed a number of artefacts and thus managed to achieve the realization of defined use case scenarios. These artefacts are described in details in D12.3.1 “D12.3.1 Deployed use cases M24”.

With respect to these produced artefacts and to the relevant evaluation activities performed by both group of evaluators (the use case provider’s teams and the group of experts) it is evident that ARTIST methodology and toolset can guide the decision making process towards achieving the organization goals in the context of cloudifying an application. Even though the general conclusion for both of them is positive and their impact to the migration process is high, both assets require further investment in several directions.

ARTIST methodology is quite promising as it already exposes a nice and clear flow between the different migration phases. On the other hand, a more personalized support for the migration of domain – specific applications to the Cloud is needed. The group of experts highlighted the emphasis that the ARTIST project should give on the process customization, from the methodology point of view. This is considered a great advantage the project could provide.

ARTIST toolset is complete and evaluated as very useful in the general context of migration to cloud of non-cloud applications. However, the level of maturity of current prototypes does not allow yet assessing how well the approach would successfully cope with the specificity of the migration of the use cases. Some iteration is definitely required to make the tools more stable in what they do. Even though the toolset as a whole has been used in an experimental environment (Pet Shop example) as a proof of concept demonstration, its realization in real industry scenarios must be proven. At the same time, the group of experts highlighted that significant background is required to use the tools. The ARTIST consortium should find ways to deal with this. For instance training services in the business and technical ecosystem could be offered. Finally, for SW providers it is important to ensure the support and maintenance of the open source code.