Commit Graph

248 Commits

Author SHA1 Message Date
Erik Thuning
cbf29c4962 Implementing a unified job pool for transcodes and subtitles never panned out,
so the code was just adding unnecessary complexity. The pipeline now uses
mp.pool to manage ffmpeg jobs as before.

This reverts commit f91109fb3e and deletes the
WorkThread class and its associated tests.
2024-10-17 11:34:00 +02:00
Erik Thuning
3da736422d Added a debug log line when the model has finished loading.
Also removed some redundant info from the startup debug line.
2024-10-16 14:03:20 +02:00
Erik Thuning
dd0af2221f Removed GPU code from transcode.py, since it never worked properly.
Also cleaned out irrelevant options from config.ini.example.

 - Removed the encoder setting for transcodes i config.ini.example since only
   software is supported now.
 - Since TranscodeWhisperHandler no longer uses the job pool, removed the
   jobsize setting in config.ini.example.
 - Changed default pool and job size to saner values.

The entire WorkThread class should probably be dropped.
2024-10-16 13:56:17 +02:00
Erik Thuning
27598d4401 Promoted subtitles_whisper_hack from hack status.
- Overwrote subtitles_whisper with subtitles_whisper_hack
 - Moved the comment about it being a hack to the more specific hack spot
 - Updated __init__.py to import from subtitles_whisper

The work invested in _hack is now significant enough that if automatic gpu
detection becomes viable again, the only meaningful starting point is
the _hack implementation. It's probably a good idea to remove _serial as well,
but leaving it around for now.
2024-10-14 16:34:46 +02:00
Erik Thuning
d6206f1e6b Changed instantiation logic for whisper handler:
- Separated GPU and CPU config keywords
 - GPUs are now chosen based on GPU index
 - Configurable number of threads per GPU
2024-10-14 16:26:31 +02:00
Erik Thuning
f1de6a19c4 Whitespace 2024-10-10 12:59:05 +02:00
Erik Thuning
f8ab46bac8 Added support for uploads with name prefix 2024-09-06 11:40:59 +02:00
Erik Thuning
11917608bb Made raise_for_structure provide better feedback on errors
The function now takes an argument corresponding to the key that is
being worked on. Also made the handling of dict validation more explicit
about keys and the reasoning around why it does what it does.
2024-06-18 11:26:20 +02:00
Erik Thuning
8de039a3f3 Fixed some bugs so the preprocessor works:
- start time is now recorded as a timestamp
 - typo fix: match.groups() -> match.group()
 - Subtitle generation is only requested if the presentation has video on channel 1
2024-06-18 11:23:43 +02:00
Erik Thuning
1afa1b93af Added env subdirectory to gitignore 2024-06-13 15:41:09 +02:00
Erik Thuning
4842ed56f1 Fixed several errors in the arec preprocessor
* renamed the 'data' dict to 'arec_data' and made sure all references are consistent
* switched the recorder identifier from hostname to description, because there was a wild \n in the hostname for unknown reasons
* removed timezone awareness from starttime and endtime to make them compatible with times loaded from daisy
2024-06-13 15:37:21 +02:00
Erik Thuning
b7a3d7dec2 Activated the arec preprocessor 2024-06-13 15:36:25 +02:00
Erik Thuning
dec2da4310 Added a preprocessor for arec recorders.
Also moved the code for creating a basic jobspec and pulling information from
the relevant daisy booking into the preprocessor superclass so it can be
called by both the cattura and arec preprocessors.
2024-06-04 15:53:46 +02:00
Erik Thuning
49d1da804b Started seting the running flag immediately on the call to start()
instead of after all the startup tsks are done. This alleviates a potential
problem where start() could be called more than once in quick succession
with unpredictable results.
2024-06-04 15:48:17 +02:00
30361abdd9 VisibilityHandler (#3)
Create a VisibilityHandler

Co-authored-by: nenzen <stefan@nenzen.com>
Reviewed-on: #3
Reviewed-by: erth9960 <thuning@dsv.su.se>
Co-authored-by: Stefan Nenzén <nenzen@dsv.su.se>
Co-committed-by: Stefan Nenzén <nenzen@dsv.su.se>
2024-03-27 14:54:33 +01:00
Erik Thuning
37c2dbaec6 Merge branch 'master' of gitea.dsv.su.se:DMC/play-daemon-threaded 2024-03-27 10:50:33 +01:00
Erik Thuning
3d9075a5e0 Starting on a visibility handler 2024-03-27 10:50:09 +01:00
Erik Thuning
bfe68b9250 Fixed a typo 2024-03-27 10:40:31 +01:00
Erik Thuning
3a50515f7a Merge branch 'master' of gitea.dsv.su.se:DMC/play-daemon-threaded 2024-03-26 15:50:47 +01:00
Erik Thuning
6c11d5578f Added optional support for user-specified notification endpoints in packages 2024-03-26 15:47:39 +01:00
67f8dfc177 Added diagram outlining the pipeline flow 2024-02-21 11:24:54 +01:00
Erik Thuning
19498e64f9 Added logging of contents of queue files from disk and preprocessed jobs 2024-02-16 11:58:05 +01:00
Erik Thuning
02341f0cd8 The original check was a bit hasty, this one should actually work properly. 2024-02-06 14:17:03 +01:00
Erik Thuning
8a0a9fbc33 Set the cattura preprocessor to automatically request subtitling
based on the camera source.
2024-02-06 13:31:11 +01:00
Erik Thuning
9874390a78 Python complains if a dictionary is modified while using an iterator on it, so changing iterator to a list 2024-01-18 10:44:15 +01:00
Erik Thuning
35d7a057e3 Missed another import 2024-01-18 10:37:53 +01:00
Erik Thuning
2cbb501361 Forgot an import 2024-01-18 10:34:18 +01:00
Erik Thuning
bb76807921 Added logging when deleting job files 2024-01-18 10:32:13 +01:00
Erik Thuning
d28be73e2c No longer setting explicit width, so that ffmpeg can take care of odd number oddness 2024-01-15 20:25:51 +01:00
Erik Thuning
7a2736300b Made logging statements use consistent string formats and some minor related fixes 2023-12-21 14:41:11 +01:00
Erik Thuning
96f1e55286 Added a syslogidentifier line 2023-12-21 14:40:55 +01:00
Erik Thuning
18841b44d7 Switched to the hack version of SubtitlesWhisperHandler 2023-12-20 16:19:07 +01:00
Erik Thuning
b4f6ff8bde Formatting 2023-12-20 16:18:43 +01:00
Erik Thuning
e19fb3c9a4 Added a hack version of parallel transcription until triton is fixed 2023-12-20 16:18:03 +01:00
Erik Thuning
a83efc62a5 Logging process information if log level is DEBUG or higher 2023-12-14 16:40:24 +01:00
Erik Thuning
622f7cbb61 Generating two subtitle tracks per job in the generation test 2023-12-14 16:37:16 +01:00
Erik Thuning
4283badaef Saving intermediate subtitle files under the track name instead of the source name 2023-12-14 16:29:30 +01:00
Erik Thuning
784b365f1e Switching to the serial whisper handler 2023-12-14 14:59:48 +01:00
Erik Thuning
51991d327b Testing out a new approach to dispatching subtitle generation jobs.
Hope to solve gpu assignment issues
2023-12-14 14:58:35 +01:00
Erik Thuning
c34ed2cc62 Updated the subtitles generation test to run several generation tasks 2023-12-12 15:39:19 +01:00
Erik Thuning
a5d9eef356 Switched the directory observer in QueueReader from inotify to polling,
in order to properly handle NFS-mounted queue directories.

As a consequence, the default timeout for awaiting job results in the tests
was upped to 10 seconds due to the slightly slower queue pickup.
2023-12-08 11:32:16 +01:00
Erik Thuning
60150c98ab Updated tests to reflect the split subtitles handlers 2023-11-30 16:48:37 +01:00
Erik Thuning
10c126a37c Split subtitles handling into two handlers.
One for generation and one for import.
2023-11-30 16:37:13 +01:00
Erik Thuning
2da06f84a7 Attempting to make sure all log lines get flushed, and fixed a variable name 2023-11-30 16:08:07 +01:00
Erik Thuning
f634fd4b1e Documented the ability to force the language when generating subtitles 2023-11-30 15:51:14 +01:00
Erik Thuning
c7f6207cc6 Pointed out that the user running the daemon must be able to write to its home directory 2023-11-30 12:52:12 +01:00
Erik Thuning
881de4f699 Merged two log lines for better emails 2023-11-30 12:35:11 +01:00
Erik Thuning
a6ccb053d8 Removed an unnecessary try/except.
Handler._process() already catches the exception in a sane way and hence no
such handling is needed in SubtitlesHandler._handle()
2023-11-30 11:53:14 +01:00
Erik Thuning
b5eda4695f Updated logging calls 2023-11-30 09:53:09 +01:00
Erik Thuning
6da6e92d46 Added script to clean up trash files 2023-11-24 16:22:42 +01:00