PyPI v1.16.4 - ClearML

New Features

• Add custom task binary support to clearml-task and CreateAndPopulate (allows bash script execution, requires agent version >=1.9.0)
• Add support for a default extension name when uploading a pandas dataframe artifact (see sdk.development.artifacts.default_pandas_dataframe_extension_name configuration option)
• Add verify field support for downloadable URL instead of a file path (see sdk.aws.s3 configuration section)

Bug Fixes

• Fix valid model URL might be overridden by an invalid one in case the upload failed

PyPI v1.16.3 - ClearML

New Features

• Add --tags option to clearml-task (#1284)
• Add retries parameter to StorageManager.upload_folder() (#1305)
• Add clearml-task and CreateAndPopulate support for bash scripts, ipynb and python modules (requires clearml-agent v1.9+)
• Add support for HTTP file upload progress reporting
• Add CLEARML_MULTI_NODE_SINGLE_TASK (values -1, 0, 1, 2) for easier multi-node single Task workloads
• Add Model.original_task property to models
• Change Model.task property to return connected task
• Update docstring on allowing users to pass packages=False to revert to requirements.txt inside their git repository

Bug Fixes

• Fix Kerastuner framework and examples (#1279)
• Fix scalar logging bug with Fire (#1301, thanks @tvelovraf!)
• Fix support passing folder to Task.get_script_info() to get the git info
• Fix Task.launch_multi_node() to enforce the parent of sub-tasks to be the master node 0 task
• Fix tensorboard numpy 2.0 incompatibility breaks binding
• Fix Task.launch_multi_node() not supported when used via Pytorch Lightning
• Fix Jupyter notebook packages and uncommitted changes are sometimes not fetched
• Fix "can't create new thread at interpreter shutdown" errors (known issue with Python v3.12.0 and other versions)
• Fix injected task import in Task.populate()
• Fix dataset with external links will not reuse downloaded data from parents
• Fix hierarchy for pipeline nodes without args
• Fix when abort callback is set, set task status to stopped only if running locally, otherwise leave it for the Agent to set it
• Fix jsonschema / referencing import to include TypeError protection
• Fix Dataset offline behavior

PyPI v1.16.2 - ClearML

New features

• Make dataset preview optional (#1270, thanks @pedroconceicao!)
• Add api.public_ip_ping (default: 8.8.8.8) and api.public_ip_service_urls (default: ["api.ipify.org";, "ident.me";]) configuration settings to detect public IP and network interface
• Improve Dataset.get_mutable_local_copy() docstring

Bug Fixes

• Fix typo in docs and default SDK config (#1281, thanks @jimdiroffii!)
• Fix python-fire integration (#1275, thanks @tvelovraf!)
• Fix path substitution for file:// URIs (#1251, thanks @nfzd!)
• Fix numpy 2.0 compatibility (np.NINF removed)
• Fix no need to recreate reporter if forking and reporting in subprocess
• Fix forked detection mechanism

PyPI v1.16.1 - ClearML

Bug Fixes

• Fix pipeline breaks when continue_on_abort is set to true
• Fix Pycharm Plugin Windows/Linux interoperability

PyPI v1.16.0 - ClearML

New Features

• Add additional warning instructing on how to install in case we failed detecting a Jupyter notebook with an import error
• Add Task.get_executed_queue() to get the name/ID of the queue a task was executed in
• Move Task.set_resource_monitor_iteration_timeout() to a class method, add wait_for_first_iteration_to_start_sec and max_wait_for_first_iteration_to_start_sec arguments (also add sdk.development.worker.wait_for_first_iteration_to_start_sec and sdk.development.worker.max_wait_for_first_iteration_to_start_sec configuration options)
• Add support for better pipeline continue behavior including control of children using the continue_behaviour argument
• Add Python 3.12 support

Bug Fixes

• Fix issue #1249 pytorch-lightning patches (#1254, thanks @a-gardner1!)
• Fix parameter overrides are converted to string when using HPO (#975)
• Fix FastAI performance (#1234)
• Fix MIG GPU support
• Fix AMD GPU metrics collection
• Fix Jupyter password might not be used in some protected Jupyterlab instances
• Fix URL substitution was not applied to registered uploaded files when reporting an event

PyPI v1.15.1 - ClearML

Bug Fixes

• Fix auto-scaler should recheck that the worker is still IDLE before shutting it down (#1240, thanks @cthorey!)
• Fix resource monitor fails to get GPU stats in some edge cases

PyPI v1.15.0 - ClearML

New Features

• Add draft option to pipeline steps (#1226, thanks @CharlesFrankum!)
• Add support for custom working directory for pipelines (#1194)
• Add Task.get_requirements() method returning the task’s requirements
• Allow controlling the number of threads used by StorageManager.download_folder() using the max_workers argument
• Update examples dependencies
• Improve auto populate in Task.init()
• Documentation:
  • Add docstrings for model properties
  • Improve docstring for force_requirements_env_freeze
  • Add Task.set_packages() docstring notes

Bug Fixes

• Fix UTF-8 script code encoding issue (#1208, thanks @ae-ae!)
• Fix Colab docs (#1220, thanks @tkukurin!)
• Fix metrics reporting with OutputModel while in offline mode results in an error (#1172)
• Fix task running in Google Colab doesn't properly get the notebook diff (#1204)
• Fix hydra binds break in offline mode (#1215)
• Fix HPO crashes when optimizing for single value scalars (#1221)
• Fix GPU info such as gpu_memory and gpu_type is not being collected in some cases
• Fix clearml-data CLI tool will move non-dataset tasks to a .dataset project if the respective task is not a dataset
• Fix pandas DataFrame artifacts with hierarchical indices get mangled by the CSV round-trip
• Fix urllib3 sends deprecation warning when setting ssl_version
• Fix Task.connect(dict) return value is not dict-compatible
• Fix jsonargparse sub-command config parsing
• Fix Lightning integration crashes when a config entry contains . in its name
• Fix Python 3.5 compatibility

PyPI v1.14.4 - ClearML

Bug Fixes

• Fix AWS S3 configuration secure property not forwarded in StorageHelper (#1205, thanks @odulcy-mindee!)
• Fix NVIDIA_VISIBLE_DEVICES=all env var value will result in no GPU metrics being reported
• Fix setting output_uri when an AWS S3 config section with secure=false exists raises an error
• Fix CacheManager attempts to clear cache when file limit has not been reached resulting in slow unnecessary checks

PyPI v1.14.3 - ClearML

New Features and Bug Fixes

• Add hash check for pickle files (resolves CVE-2024-24590)
• Fix clearml-task relative paths calculation when --cwd, --folder and --script are provided (#1161, thanks @dimidagd!)
• Fix regression causing an error when secure is provided in AWS S3 bucket settings

PyPI v1.14.2 - ClearML

New Features and Improvements

• Report upload/download with tqdm if installed
• Add support for a PipelineController.version property
• Add support for specifying an AWS profile when providing bucket storage credentials
• Add an ignore_remote_overrides argument to Task.connect() and Task.connect_configuration() allowing users to disable overriding values when running remotely
• Add support for setting a custom certificate path using the CLEARML_API_HOST_VERIFY_CERT environment variable (instead of just allowing the user to disable verification)
• Add dataset reporting example (#1180)
• Update README GIFs (#1191)

Bug Fixes

• Fix missing component callbacks on multiple step calls (#1195, thanks @materight!)
• Fix unsafe usage of tempfile.mktemp
• Fix potential path traversal on file download (CVE-2024-24591)
• Fix Python 3.12 support by not using distutils
• Fix TaskScheduler does not save local state before running remotely
• Fix pipeline should continue instead of aborting when PipelineController.abort_on_failure is set to False and a step has been aborted
• Fix HPO crashes when its corresponding task is initialized with continue_last_task=True