History

Unreleased

  • Make get_tasks() return a copy of the tasks instead of the internal list. This is arguably a breaking change for some workflows, but returning the internal version was arguably a bug
  • Removed dependency on the six library as we no longer support Python 2.7.

Version 2.1.1

Released 18 March 2021

  • Include requirements files in distribution

Version 2.1.0

Released 18 March 2021

  • Improvements to webstatus (by Robert Denham)
  • Removed Python 2.7 support
  • Fix output encoding for Python 3.8
  • Fix bug mixing mapreduce() & status --cache
  • Make block_access (used in mapreduce()) much faster (20x)
  • Fix important redis bug
  • More precise output in cleanup command

Version 2.0.2

Released Thu Jun 11 2020

Bugfixes

  • Fix command line argument parsing

Version 2.0.1

Released Thu Jun 11 2020

Bugfixes

  • Fix handling of JUG_EXIT_IF_FILE_EXISTS environmental variable
  • Fix passing an argument to jug.main() function
  • Extend --pdb to exceptions raised while importing the jugfile (issue #79)

Version 2.0.0

Released Fri Feb 21 2020

User-visible improvements

  1. Jug can now keep track of failed tasks.

Use jug execute --keep-failed to preserve locks on failing tasks and jug cleanup --failed-only to remove locks from failed tasks. Similarly, jug status and jug graph now display failed tasks

This avoids the situation where a failing task (for whatever reason) blocks the rest of the process. It improves on jug --keep-going (which does the same) as with --keep-going, each new process will attempt to run the task. If the task reproducibly fails after a long period, this results in wasted computation.

  1. The exit environmental variable are now checked by default. Create a file called __jug_please_stop_running.txt in the current working directory and Jug will stop after it finishes the current task.

Bugfixes

  • Fix jug sleep-until in the presence of barrier() (issue #71)

Internal improvements

  • jug.backend.base_store has 1 new method listlocks
  • jug.backend.base_lock has 2 new methods fail and is_failed

Version 1.6.9

Released on Tue Aug 6 2019

  • Fix saving on newer version of numpy

version 1.6.8 (Wed July 10 2019)

  • Add cached_glob() function
  • Fix NoLoad (issue #73)
  • Fix jug shell’s invalidate function with Tasklets (issue #77)

version 1.6.7 (Fri Apr 13 2018)

  • Fix issue with deeply recursive dependency structures and barrier()
  • Allow mapreduce.map() results to be used as dependencies

version 1.6.6 (Sat Apr 7 2018)

  • Fix bug in shell’s invalidate() function
  • Fix wrong dependency handling with mapreduce.map()

version 1.6.5 (Mon Mar 12 2018)

  • Add get_tasks() to ‘jug shell’ and document ‘from jug.task import alltasks’ (patch by Renato Alves)

version 1.6.4 (Thu Nov 2 2017)

  • Fix exit_after_n_tasks. It would previously execute one task too many

version 1.6.3 (Wed Nov 1 2017)

  • Add citation request

version 1.6.2 (Thu Oct 26 2017) - Add return_value argument to jug_execute - Add exit_env_vars

version 1.6.1 (Thu Aug 29 2017) - Fix bug with invalidate() in the shell

version 1.6.0 (Thu Aug 24 2017) - Add ‘graph’ subcommand - Generates a graph of tasks - ‘jug execute –keep-going’ now ends with non-zero exit code in case of failures - Fix bug with cleanup in dict_store not providing the number of removed records - Add ‘jug cleanup –keep-locks’ to remove obsolete results without affecting locks

version 1.5.0 (Sun Jul 16 2017) - Add ‘demo’ subcommand - Add is_jug_running() function - Fix bug in finding config files - Improved –debug mode: check for unsupported recursive task creation - Add invalidate() to shell environment - Use ~/.config/jug/jugrc as configuration file - Add experimental support for extensible commands, use ~/.config/jug/jug_user_commands.py - jugrc: execute_wait_cycle_time_secs is now execute_wait_cycle_time - Expose sync_move in jug.utils

version 1.4.0 (Tue Jan 3 2017)

  • Fix bug with writing very large objects to disk
  • Smarter handling of –aggressive-unload (do not unload what will be immediately necessary)
  • Work around corner case in jug shell command
  • Add test-jug subcommand
  • Add return_tuple decorator

version 1.3.0 (Tue Nov 1 2016) - Update shell subcommand to IPython 5 - Use ~/.config/jugrc as configuration file - Cleanup usage string - Use bottle instead of web.py for webstatus subcommand - Add jug_execute function - Add timing functionality

version 1.2.2 (Sat Jun 25 2016)

  • Fix bugs in shell subcommand and a few corner cases in encoding/decoding results

version 1.2.1 (Mon Feb 15 2016)

  • Changed execution loop to ensure that all tasks are checked (issue #33 on github)
  • Fixed bug that made ‘check’ or ‘sleep-until’ slower than necessary
  • Fixed jug on Windows (which does not support fsync on directories)
  • Made Tasklets use slightly less memory

version 1.2 (Thu Aug 20 2015) - Use HIGHEST_PROTOCOL when pickle()ing - Add compress_numpy option to file_store - Add register_hook_once function - Optimize case when most (or all) tasks are already run - Add –short option to ‘jug status’ and ‘jug execute’ - Fix bug with dictionary order in kwargs (fix by Andreas Sorge) - Fix ipython colors (fix by Andreas Sorge) - Sort tasks in ‘jug status’

version 1.1 (Tue Mar 3 2015) - Python 3 compatibility fixes - fsync(directory) in file backend - Jug hooks (still mostly undocumented, but already enabling internal code simplification)

version 1.0 (Tue May 20 2014) - Adapt status output to terminal width (by Alex Ford) - Add a newline at the end of lockfiles for file backend - Add –cache-file option to specify file for status --cache

version 0.9.7 (Tue Feb 18 2014)

  • Fix use of numpy subclasses
  • Fix redis URL parsing
  • Fix shell for newer versions of IPython
  • Correctly fall back on non-sqlite status
  • Allow user to call set_jugdir() inside jugfile

version 0.9.6 (Tue Aug 6 2013)

  • Faster decoding
  • Add jug-execute script
  • Add describe() function
  • Add write_task_out() function

version 0.9.5 (May 27 2013)

  • Added debug mode
  • Even better map.reduce.map using blocked access
  • Python 3 support
  • Documentation improvements

version 0.9.4 (Apr 15 2013)

  • Add CustomHash wrapper to set __jug_hash__
  • Print traceback on import error
  • Exit when no progress is made even with barrier
  • Use Tasklets for better jug.mapreduce.map
  • Use Ipython debugger if available (patch by Alex Ford)
  • Faster –aggressive-unload
  • Add currymap() function

version 0.9.3 (Dec 2 2012)

  • Fix parsing of ports on redis URL (patch by Alcides Viamontes)
  • Make hashing robust to different orders when using randomized hashing (patch by Alcides Viamontes)
  • Allow regex in invalidate command (patch by Alcides Viamontes)
  • Add --cache --clear suboption to status
  • Allow builtin functions for tasks
  • Fix status –cache`` (a general bug which seems to be triggered mainly by bvalue() usage).
  • Fix CompoundTask (broken by earlier __jug_hash__ hook introduction)
  • Make Tasklets more flexible by allowing slicing with Tasks (previously, slicing with tasks was not allowed)

version 0.9.2 (Nov 4 2012):

  • More flexible mapreduce()/map() functions
  • Make TaskGenerator pickle()able and hash()able
  • Add invalidate() method to Task
  • Add –keep-going option to execute
  • Better help messsage

version 0.9.1 (Jun 11 2012):

  • Add –locks-only option to cleanup subcommand
  • Make cache file (for status subcommand) configurable
  • Add webstatus subcommand
  • Add bvalue() function
  • Fix bug in shell subcommand (value was not in global namespace)
  • Improve identity()
  • Fix bug in using Tasklets and –aggressive-unload
  • Fix bug with Tasklets and sleep-until/check

version 0.9:

  • In the presence of a barrier(), rerun the jugfile. This makes barrier much easier to use.
  • Add set_jugdir to public API
  • Added CompoundTaskGenerator
  • Support subclassing of Task
  • Avoid creating directories in file backend unless it is necessary
  • Add jug.mapreduce.reduce (which mimicks the builtin reduce)

For older version see ChangeLog file.