286 lines
8.6 KiB
Markdown
286 lines
8.6 KiB
Markdown
# DSV play daemon
|
|
|
|
This application's job is to monitor for presentation uploads,
|
|
process them and send metadata to a configured remote URL once
|
|
processing is complete.
|
|
|
|
## Python dependencies
|
|
|
|
- Requests
|
|
- Watchdog
|
|
- ffmpeg-python
|
|
|
|
## Workflow
|
|
|
|
This is an overview of the packaging workflow:
|
|
|
|
1. Queue pickup:
|
|
Queue files are picked up from the directory configured by
|
|
'queue' in config.ini.
|
|
|
|
1. Origin-specific handling:
|
|
Create a consistent package from the different incoming formats.
|
|
|
|
1. Working directory:
|
|
Create a temporary working directory and add it to the package.
|
|
|
|
1. Transcoding and thumb generation:
|
|
All transcoding actions are handled here. Incoming video streams
|
|
are transcoded to the expected resolutions, slide streams are
|
|
converted to video and any missing thumbnails and/or poster images
|
|
are created. The resulting files are placed in the working directory
|
|
and the package updated.
|
|
|
|
1. Update integration:
|
|
If the package is an update job, integrates the existing files with
|
|
the updates as appropriate. Otherwise this step is skipped.
|
|
|
|
1. Stash originals:
|
|
Currently disabled due to difficulties integrating with updates.
|
|
Places all the original files in a subdirectory of the working
|
|
directory for future reference/safekeeping/whatnot.
|
|
|
|
1. Platform notification:
|
|
Send the package as a notification to the configured endpoint
|
|
('notify_url' in config.ini). This notification package will
|
|
use the format specified below.
|
|
|
|
1. Finalize processing:
|
|
Saves the notification package to the working directory and moves
|
|
the working directory to the final storage destination ('storage'
|
|
in config.ini). Deletes the queue file and incoming files.
|
|
|
|
## Package formats
|
|
|
|
The package format is generally similar across incoming data, processing
|
|
and notification, but there are differences, especially semantic ones.
|
|
|
|
### Incoming
|
|
|
|
Incoming formats vary between origins. TODO.
|
|
|
|
#### Mediasite
|
|
|
|
Mediasite jobs can contain a slide stream that needs to be converted to
|
|
video. This is a top-level key in the job file.
|
|
|
|
```javascript
|
|
// A list of slides to be merged into a video stream by the transcoding
|
|
// step. The field will be removed entirely after packaging and replaced
|
|
// with a special source object.
|
|
'slides': [
|
|
{
|
|
// The web URL to a slide to be downloaded by the packager and
|
|
// placed in a demux file for later transcoding to video.
|
|
'url': '',
|
|
|
|
// The amount of time that this slide is to be shown,
|
|
// in milliseconds.
|
|
'duration': 0
|
|
},
|
|
|
|
// Further slide objects.
|
|
...
|
|
]
|
|
```
|
|
|
|
### Processing
|
|
|
|
This is the format of the package going through the pipeline.
|
|
Some fields get created partway through the pipeline and others only
|
|
exist for certain origins.
|
|
|
|
Unless otherwise noted, a field can be expected to always exist after
|
|
the origin-specific handling step (which will create the initial package).
|
|
|
|
Relative paths are relative to the 'base' attribute until they have been
|
|
processed in the transcoding step. After that the path is relative to the
|
|
'workbase' directory.
|
|
|
|
```javascript
|
|
{
|
|
// The unique ID of the job. Assigned by the receiving API and may
|
|
// not match the final ID to be sent in the notification.
|
|
// It is used to guarantee unique working directories etc.
|
|
'id': '',
|
|
|
|
// The origin of the job; currently one of:
|
|
// 'manual', 'mediasite', 'cattura', 'update'
|
|
'origin': '',
|
|
|
|
// This field only exists for the 'update' origin.
|
|
// The ID of the existing presentation to be updated.
|
|
'orig_id': '',
|
|
|
|
// This field only exists for the 'manual' and 'mediasite' origins.
|
|
// Temporary ID used on the notification receiving end to match
|
|
// the job to existing metadata.
|
|
'notification_id': '',
|
|
|
|
// The absolute path to the directory where incoming job files
|
|
// are located. Should be a direct subdirectory of the directory
|
|
// configured by 'incoming' in config.ini.
|
|
'base': '',
|
|
|
|
// The absolute path to the working directory used during processing.
|
|
// Will be a direct subdirectory of the directory configured by
|
|
// 'processing' in config.ini
|
|
'workbase': '',
|
|
|
|
// The presentation creation time as a unix timestamp. Taken from
|
|
// incoming metadata, in no way related to creation time of the
|
|
// job on the server.
|
|
'creation': 0,
|
|
|
|
// The duration of the presentation as a number of seconds. Decimals OK.
|
|
'duration': 0,
|
|
|
|
// The presentation title.
|
|
'title': {
|
|
'en': '',
|
|
'sv': ''
|
|
},
|
|
|
|
// A possibly empty list of presenters involved in the presentation.
|
|
'presenters': [],
|
|
|
|
// A possibly empty list of course codes the presentation should
|
|
// be associated with.
|
|
'courses': [],
|
|
|
|
// The relative path to the thumbnail file to be shown when browsing
|
|
// presentations. May be empty before the transcoding step, in which
|
|
// case the transcoder will create it and populate the field.
|
|
'thumb': '',
|
|
|
|
// A possibly empty list of tags. Tags are arbitrary strings.
|
|
'tags': [],
|
|
|
|
// A possibly empty relative path to a subtitle file in VTT format.
|
|
'subs': '',
|
|
|
|
// A list of at least one source. The format of a source object is
|
|
// documented below.
|
|
'sources': [source1, ...],
|
|
}
|
|
```
|
|
|
|
There are two valid source object formats - video and slide objects. Slide
|
|
objects will be converted to video objects in the transcoding step.
|
|
|
|
A video object:
|
|
```javascript
|
|
{
|
|
// A name for this source. Must be unique within the presentation.
|
|
'name': '',
|
|
|
|
// The possibly empty relative path to the poster image to be shown for
|
|
// this source before playback has started. If it is empty, the
|
|
// transcode step will create a poster image and populate this field.
|
|
'poster': source.get('poster', ''),
|
|
|
|
// A boolean indicating whether to play audio from this source.
|
|
// Exactly one source in a package should have this set to true.
|
|
'playAudio': bool,
|
|
|
|
// The relative path to the source video file. This will be
|
|
// updated in the transcode step to the format specified in
|
|
// the [Notification](#Notification) section.
|
|
'video': '',
|
|
}
|
|
```
|
|
|
|
A slides object:
|
|
```javascript
|
|
{
|
|
// The absolute path to the demux file to be used to produce the video.
|
|
'demux_file': '',
|
|
|
|
// The absolute path to the poster file to be shown before playback
|
|
// has started.
|
|
'poster': '',
|
|
|
|
// A boolean indicating whether to play audio from this source.
|
|
// Always false, since a slideshow has no audio.
|
|
'play_audio': false,
|
|
}
|
|
```
|
|
|
|
### Notification
|
|
|
|
This is the format of the package when it is sent as a notification.
|
|
|
|
All filenames are given as paths relative to the package root.
|
|
|
|
```javascript
|
|
{
|
|
// The ID of the presentation.
|
|
'id': '',
|
|
|
|
// The origin of the notification; currently one of:
|
|
// cattura, mediasite, manual, update
|
|
'origin': '',
|
|
|
|
// Only exists for the 'manual' and 'mediasite' origins. Used
|
|
// on the receiving end to match the notification to existing metadata.
|
|
'notification_id': '',
|
|
|
|
// Creation time of the presentation, as a unix timestamp.
|
|
'creation': 0,
|
|
|
|
// The title of the presentation.
|
|
'title': {
|
|
'en': '',
|
|
'sv': ''
|
|
},
|
|
|
|
// A possibly empty list of presenters involved in the presentation.
|
|
'presenters': [],
|
|
|
|
// A possibly empty list of course codes the presentation should
|
|
// be associated with.
|
|
'courses': [],
|
|
|
|
// The length of the presentation in seconds. May include decimals.
|
|
'duration': 0,
|
|
|
|
// The thumbnail image to be shown when browsing presentations.
|
|
'thumb': '',
|
|
|
|
// A possibly empty list of tags. Tags are arbitrary strings.
|
|
'tags': [],
|
|
|
|
// The filename of the subtitles file. If there is no subtitles file,
|
|
// an empty string is passed.
|
|
'subs': '',
|
|
|
|
// A list of one or more source objects. See the source format below.
|
|
'sources': [source1, ...],
|
|
}
|
|
```
|
|
|
|
This is the format for each source in the list of sources:
|
|
```javascript
|
|
{
|
|
// A name for this source. Must be unique within the presentation.
|
|
'name': 'main',
|
|
|
|
// The filename of the poster image to be shown for this source
|
|
// before playback has started.
|
|
'poster': '',
|
|
|
|
// A boolean indicating whether to play audio from this source.
|
|
// Exactly one source in a package should have this set to true.
|
|
'playAudio': bool,
|
|
|
|
// A dict listing the video filename for each resolution variant
|
|
// of this source. The key should be the variant's vertical
|
|
// resolution as a string, e.g '720' etc.
|
|
// All sources in a presentation should have the same set of resolutions.
|
|
'video': {
|
|
resolution: '',
|
|
...
|
|
},
|
|
}
|
|
```
|