Erik Thuning ff40f17ce3 Documentation tweaks and preparation for pipeline overhaul

2022-09-07 15:15:07 +02:00

15 KiB

Raw Permalink Blame History

DSV play daemon

This application's job is to monitor for presentation uploads, process them and send metadata to a configured remote URL once processing is complete.

Python dependencies

This application is written for python3. The libraries required are specified in requirements.txt.

Workflow

This is an overview of the packaging workflow:

Queue pickup: Queue files are picked up from the directory configured by 'queue' in config.ini.
Origin-specific handling: Create a consistent package from the different incoming formats.
Working directory: Create a temporary working directory and add it to the package.
Subtitles pickup: If a subtitle file was passed, it is copied into the working directory and the package updated accordingly.
Thumb pickup/generation: If a thumbnail image was passed, it is copied to the working directory. Otherwise a thumb is generated based on the package metadata and placed in the working directory. The package is updated accordingly.
Transcoding and poster generation: All transcoding actions are handled here. Incoming video streams are transcoded to the expected resolutions, slide streams are converted to video and any missing poster images are created. The resulting files are placed in the working directory and the package updated.
Update integration: If the package is an update job, integrates the existing files with the updates as appropriate. Otherwise this step is skipped.
Stash originals: If the log level is DEBUG or higher, places all the original files in a subdirectory of the working directory.
Platform notification: Send the package as a notification to the configured endpoint ('notify_url' in config.ini). This notification package will use the format specified below.
Finalize processing: Saves the notification package to the working directory and moves the working directory to the final storage destination ('storage' in config.ini). Deletes the queue file and incoming files.

Package formats

The package format is generally similar across incoming data, processing and notification, but there are differences, especially semantic ones.

Incoming

Incoming formats vary between origins, but the basic format is as follows, except for cattura which has its own format.

Unless noted otherwise, all keys are mandatory. A lot of the values may be empty strings though - what that means in a specific case is documented below.

{
  // The title of the presentation in swedish and english.
  "title": {
    "sv":        "",
    "en":        ""
  },

  // A description of the presentation. No localization support
  // for the time being.
  "description": "",

  // The creation time of the presentation as a unix timestamp.
  "created":     0,

  // The duration of the presentation in seconds, possibly fractional.
  "duration":    0,

  // A possibly empty list of presenter, identified by SU username.
  "presenters":  [],

  // A possibly empty list of courses, identified by course code.
  "courses":     [],

  // A possibly empty list of tags. Tags are arbitrary strings.
  "tags":        [],

  // The path to the presentation thumbnail. May be empty, in which case
  // it will be auto-generated.
  "thumb":       "",

  // The possibly empty path to a subtitle file in .vtt format.
  "subtitles":   "",

  // The list of video sources to be processed. The format of a single
  // source is documented below.
  "sources":     [source1, ...]
}

This is the generic format of an incoming source object:

{
  // The name of the stream. Must be unique within the presentation.
  "name":      "",

  // The path to the video file associated with this stream.
  // Must not be empty.
  "video":     "",

  // The path to a poster fr this stream. May be empty, in which case
  // one will be auto-generated.
  "poster":    "",

  // A boolean value indicating whether to play audio from this stream.
  // Only one stream per presentation should have this set to true.
  "playAudio": bool
}

Mediasite

File paths in a mediasite job must be absolute URL:s to the source files' locations on the mediasite server. The files will be downloaded to local storage in the packaging step.

A mediasite import job will contain an 'id' key as a top-level key in the job object.

{
  ...
  // This is the ID used by the platform to keep track of the job.
  // It is moved to the 'notification_id' field in the packaging step
  // in order to make room for the ID generated by the receiving API.
  "id":    "",
  ...
}

Mediasite jobs can contain a slide stream that needs to be converted to video. This is a top-level key in the job object.

{
  ...
  // A list of slide objects to be merged into a video stream by the
  // transcoding step. The field will be removed entirely after packaging
  // and replaced with a special source object.
  "slides": [
    {
      // The web URL to a slide to be downloaded by the packager and
      // placed in a demux file for later transcoding to video.
      "url":      "",

      // The amount of time that this slide is to be shown,
      // in milliseconds.
      "duration": 0
    },

    // Further slide objects.
    ...
  ],
  ...
}

Manual

Manual uploads must specify a 'base' directory where all associated incoming files are stored. This path should be located under the 'incoming' directory configured in config.ini. All other paths in the job definition must be relative to this path.

{
  ...
  // The base directory for this job's uploaded files. All other paths in the
  // job must be specified relative to this one.
  "base":    "",
  ...
}

Update

Update jobs are special in that they must act on an existing presentation. The existing presentation to be acted upon is specified in the top-level 'id' field in the incoming job definition.

Update jobs also require a 'base' field the same way as the 'manual' job type. Other paths in the job definition must be relative to this path, again just as in the 'manual' job type.

If sources are to be deleted, they should be listed in the top-level field 'delete' as below. The field is mandatory and should be an empty list of no streams are to be deleted. If a source is listed for deletion, it must not occur in the 'sources' element.

{
  ...
  // The ID of the existing presentation to be acted upon. The contents of
  // this field will be moved into the 'orig_id' field in the packaging step,
  // in order to make room for the ID generated by the receiving API.
  "id":              "",

  // Temporary ID used on the notification receiving end to match
  // the job to existing metadata.
  "notification_id": "",

  // The base directory for this job's uploaded files. May be empty if no
  // files were uploaded.
  // All other paths in the job must be specified relative to this one.
  "base":            "",

  // A possibly empty list of streams to be deleted. The streams are
  // identified by their 'name' attribute.
  "delete":          [],
  ...
}

Some fields that are normally mandatory are optional when updating. Please note that all fields not mentioned here are still mandatory.

The presence of an optional top-level field will cause the update process to act on that field as specified below.

{
  ...
  // Replace the existing thumb with the one specified here. If the field is
  // present with an empty value, the thumbnail will be re-generated using the
  // automatic generation function.
  "thumb":     "",

  // Replace the existing subtitle file with the one specified here. If the
  // field is present with an empty value, the subtitles will be deleted.
  "subtitles": "",

  // If present, act on the sources specified here. See below for details
  // on individual sources.
  "sources":   [source1, ...],
  ...
}

A source to be updated works as follows.

{
  ...
  // The name of the stream to be acted upon. If it does not match an existing
  // stream name, a new stream with the given name will be added
  // to the presentation.
  "name":  "",

  // If present, replace the current poster for this source with the one
  // specified here. If the value is blank, a new poster will be generated.
  "poster": "",

  // If present, replace the current video stream for this source with the
  // one specified here. This field may not have an empty value.
  "video":  "",
  ...
}

Cattura

The format of incoming cattura packages is completely determined by the cattura recorder software and hence isn't documented here.

Since this is the package type that is used for capturing lectures in the lecture halls, this package type will read booking data from daisy in order to populate the fields title, presenters, courses and tags.

Processing

This is the format of the package going through the pipeline. Some fields get created partway through the pipeline and others only exist for certain origins.

Unless otherwise noted, a field can be expected to always exist after the origin-specific handling step (which will create the initial package).

Relative paths are relative to the 'base' attribute until they have been processed in the transcoding step. After that the path is relative to the 'workbase' directory.

{
  // The unique ID of the job, assigned by the receiving API. Unless the
  // origin is 'update', this will be the permanent ID of the presentation.
  // It is used to guarantee unique working directories etc.
  "id":              "",

  // The origin of the job; currently one of:
  // 'manual', 'mediasite', 'cattura', 'update'
  "origin":          "",

  // This field only exists for the 'update' origin.
  // The ID of the existing presentation to be updated.
  "orig_id":         "",

  // This field only exists for the 'manual' and 'mediasite' origins.
  // Temporary ID used on the notification receiving end to match
  // the job to existing metadata.
  "notification_id": "",

  // The absolute path to the directory where incoming job files
  // are located. Should be a direct subdirectory of the directory
  // configured by 'incoming' in config.ini.
  "base":            "",

  // The absolute path to the working directory used during processing.
  // Will be a direct subdirectory of the directory configured by
  // 'processing' in config.ini
  "workbase":        "",

  // The presentation creation time as a unix timestamp. Taken from
  // incoming metadata, in no way related to creation time of the
  // job on the server.
  "creation":        0,

  // The duration of the presentation as a number of seconds. Decimals OK.
  "duration":        0,

  // The presentation title.
  "title": {
    "en":            "",
    "sv":            ""
  },

  // A possibly empty list of presenters involved in the presentation.
  "presenters":      [],

  // A possibly empty list of course codes the presentation should
  // be associated with.
  "courses":         [],

  // The relative path to the thumbnail file to be shown when browsing
  // presentations. May be empty before the transcoding step, in which
  // case the transcoder will create it and populate the field.
  "thumb":           "",

  // A possibly empty list of tags. Tags are arbitrary strings.
  "tags":            [],

  // A possibly empty relative path to a subtitle file in VTT format.
  "subtitles":       "",

  // A list of at least one source. The format of a source object is
  // documented below.
  "sources":         [source1, ...],
}

There are two valid source object formats - video and slide objects. Slide objects will be converted to video objects in the transcoding step.

A video object:

{
  // A name for this source. Must be unique within the presentation.
  "name":      "",

  // The possibly empty relative path to the poster image to be shown for
  // this source before playback has started. If it is empty, the
  // transcode step will create a poster image and populate this field.
  "poster":    "",

  // A boolean indicating whether to play audio from this source.
  // Exactly one source in a package should have this set to true.
  "playAudio": bool,

  // The relative path to the source video file. This will be
  // updated in the transcode step to the format specified in
  // the [Notification](#Notification) section.
  "video":     "",
}

A slides object:

{
  // The absolute path to the demux file to be used to produce the video.
  "demux_file": "",

  // The absolute path to the poster file to be shown before playback
  // has started.
  "poster":     "",

  // A boolean indicating whether to play audio from this source.
  // Always false, since a slideshow has no audio.
  "play_audio": false,
}

Notification

This is the format of the package when it is sent as a notification.

All filenames are given as paths relative to the package root.

{
  // The ID of the presentation.
  "id":              "",

  // The origin of the notification; currently one of:
  // cattura, mediasite, manual, update
  "origin":          "",

  // Only exists for the 'manual', 'mediasite' and 'update' origins. Used
  // on the receiving end to match the notification to existing metadata.
  "notification_id": "",

  // Creation time of the presentation, as a unix timestamp.
  "creation":        0,

  // The title of the presentation.
  "title": {
    "en":            "",
    "sv":            ""
  },

  // A possibly empty list of presenters involved in the presentation.
  "presenters":      [],

  // A possibly empty list of course codes the presentation should
  // be associated with.
  "courses":         [],

  // The length of the presentation in seconds. May include decimals.
  "duration":        0,

  // A possibly empty list of tags. Tags are arbitrary strings.
  "tags":            [],

  // The thumbnail image to be shown when browsing presentations.
  "thumb":           "",

  // The filename of the subtitles file. If there is no subtitles file,
  // an empty string is passed.
  "subtitles":       "",

  // A list of one or more source objects. See the source format below.
  "sources":         [source1, ...],
}

This is the format for each source in the list of sources:

{
  // A name for this source. Must be unique within the presentation.
  "name":      "main",

  // The filename of the poster image to be shown for this source
  // before playback has started.
  "poster":    "",

  // A boolean indicating whether to play audio from this source.
  // Exactly one source in a package should have this set to true.
  "playAudio": bool,

  // A dict listing the video filename for each resolution variant
  // of this source. The key should be the variant's vertical
  // resolution as a string, e.g '720' etc.
  // All sources in a presentation will have the same set of resolutions.
  "video":   {
    resolution: "",
    ...
  },
}

15 KiB Raw Permalink Blame History