Built-in Karton integration

[psrok1]

Steps

• Implement backend Karton integration (Built-in Karton integration #298)
• Implement Web UI part (Built-in Karton integration #298)
• Implement migration (Migrate 'karton' attribute into new KartonAnalysis objects #382)
• Provide documentation for Karton features (Docs: 2.3.0-rc1 #365)
• Implement support for Karton analysis API mwdblib#39
• Move to the native API in https://github.com/CERT-Polska/karton-mwdb-reporter

Introduction

Karton plugin is the most common integration in MWDB ecosystem and crucial part of mwdb.cert.pl service. The only public version is contained in https://github.com/CERT-Polska/karton-playground/ but it's not officially released as redistributable package. The API provided by plugin is undocumented and should be considered non-stable.

Installation of extensions is a bit complicated because there is no simple way to dynamically enable the Web part runtime. Web extensions must be compiled into common JS bundle with the core app.

That's why we decided that Karton integration will be distributed as mwdb-core built-in, disabled by default (opt-in) but ready to use straight out of the box. In addition, more tight integration with mwdb-core enables us to introduce additional features related with exploring data from Karton analyses, that are stored in MWDB.

Although karton-playground integration is non-official, we should still provide a way to easily migrate to the built-in version, without huge breaking changes.

How current mwdb-karton integration works (unofficial Karton plugin)

• All uploaded files and configurations are automatically uploaded to Karton unless they were analyzed before.
• Object <-> Analysis association is based on attribute key "karton". If we want to associate an object with an analysis, we just add analysis UUID as an "karton" attribute value to the object,
• Attribute "karton" is automatically added after the first upload of file/config. When we want to report an object that is already associated with existing analysis, we need to provide "karton" attribute as upload parameter. If "karton" attribute was provided in the same request as object upload, MWDB doesn't spawn new Karton analysis for uploaded object.
• If we have karton_manage capability: we can resubmit any existing object to the Karton to perform new analysis.
• User can find all artifacts coming from single analysis by using query meta.karton:<analysis_uid>

mwdb-karton API endpoints:

• GET /karton/<object sha256> returns status of most recent analysis
• GET /karton/<object sha256>?root_uid=<uid> returns status of older analysis chosen by analysis root_uid
• POST /karton/<object sha256> spawns new analysis for chosen object

Due to inconsistency with other API endpoints and assumption that object-analysis associations are available via attributes: endpoints presented above are a subject of change.

Description of built-in integration (official Karton mwdb-core built-in)

mwdb-core v2.3.0 will introduce built-in, opt-in integration that provides similar feature as mwdb-karton plugin presented before.

In terms of backwards compatiblity concerns, we should focus on:

• spawning Karton tasks in the same manner as plugin does
• not breaking the attributes used by present versions of Karton services
• providing smooth migration from plugin-based implementation to built-in
• keep the API endpoints as similar as possible

Integration can be enabled by user by providing enable_karton=1 option in MWDB settings and setting appropriate Karton configuration file under known path (e.g. /etc/karton/karton.ini or ./karton.ini in MWDB working dir)

Karton analysis will become a separate entity in MWDB model instead of simple attribute value provided by plugin. This will enable us to bind not only the objects to the analysis, but also other things like attributes. It may be especially useful for users to have an association between set of IoCs in Attributes and specific analysis.

In the same time: analyses will be still exposed via attributes to keep the backwards compatibility. Adding an "karton" attribute will add an association between existing analysis and object. Values of "karton" attribute can no longer be arbitrary and they must point at existing analysis spawned by MWDB integration

All features will be exposed via new API elements:

• karton_id option in upload to bind uploaded object with existing analysis
• GET /<object_type>/<object_id>/karton to list all analyses spawned for chosen object and the general analysis status for object
• GET /<object_type>/<object_id>/karton/<analysis_id> to get detailed analysis status
• POST /<object_type>/<object_id>/karton to resubmit object for analysis
• PUT /<object_type>/<object_id>/karton/<analysis_id> to create association between existing object and existing analysis
• (optionally DELETE as well to remove the association)

Legacy interface will be still attribute-based, emulating the previous karton key behavior. New users should use API presented above, attribute-based API should be considered obsolete.